Jump to content


Photo

Two Node Pool Options with Direct Attached Storage

Started by Adam Ward , 29 September 2016 - 01:25 PM
21 replies to this topic

Adam Ward Members

Adam Ward
  • 95 posts

Posted 29 September 2016 - 01:25 PM

Hi,

 

I need a two node XenServer pool. I have only 4 Virtual Machines so the resource use is light.

 

I need to have a workable way to get the VM's back on quickly in the event of a failure, manual intervention is fine.

 

I did setup a HA-Lizard environment, but it lost my data so I'm never going back to that. I have just spent an evening restoring from backup!

 

For simplicity (and not modifying Dom0) I like the idea of 2 identical XenServer's in a pool, using Direct Attached Storage. However, I am aware with this I will lose HA and XenMotion (but I can still do Storage XenMotion).

 

Has anyone got a nice way to achieve this with two XenServer's in a Pool? We will be running 2 x NetScaler VPX's - these are the appliances I am most worried about, but obviously I don't need XenServer HA for these, I can do HA with the NetScalers themselves. The other two servers are windows boxes.

 

Thanks in advance. 2 node XenServer pools are a bit of a head scratcher!

 

 



Alan Lantz Members

Alan Lantz
  • 6,985 posts

Posted 29 September 2016 - 01:47 PM

If you have a lot of space you could backup each server to each other or backup to a NAS device. To be cost effective use NAUBackup.

 

--Alan--



Adam Ward Members

Adam Ward
  • 95 posts

Posted 29 September 2016 - 02:18 PM

If you have a lot of space you could backup each server to each other or backup to a NAS device. To be cost effective use NAUBackup.

 

--Alan--

 

Can I backup from one xenserver to another? How do I do that? (I have had no sleep for 2 days, please bear with me)



Alan Lantz Members

Alan Lantz
  • 6,985 posts

Posted 29 September 2016 - 02:27 PM

You could do it a couple of ways. You could install a NFS server directly on the XenServer dom0 or more preferable would be to setup a new guest VM that provides NFS Services. The downside is a host failure could cause not only loss of data for the VM's, but also wipe out your other servers backups as well. This is why a separate NAS providing NFS is what I would prefer.

 

--Alan--



Tobias Kreidl CTP Member

Tobias Kreidl
  • 18,287 posts

Posted 29 September 2016 - 02:40 PM

Backing up to an external server is probably still preferred, using most simply an NFS mount. Backups witin your pool are still subject to loss if the physical machines should both be affected. There are a number of inexpensive and free backup/recovery options our there (including the free NAUbackup at https://github.com/NAUbackup/VmBackup).

 

HA-Lizard is a high-availability solution and has nothing to do with storage corruption, so I have no idea why you are trying to put the blame somewhere IMO that it really doesn't belong. Citrix doesn't support officially a two-node pool, so you have to be willing to live with the issues that arise from that if you still try to leverage high availability.

 

-=Tobias



Adam Ward Members

Adam Ward
  • 95 posts

Posted 29 September 2016 - 02:53 PM

Backing up to an external server is probably still preferred, using most simply an NFS mount. Backups witin your pool are still subject to loss if the physical machines should both be affected. There are a number of inexpensive and free backup/recovery options our there (including the free NAUbackup at https://github.com/NAUbackup/VmBackup).

 

HA-Lizard is a high-availability solution and has nothing to do with storage corruption, so I have no idea why you are trying to put the blame somewhere IMO that it really doesn't belong. Citrix doesn't support officially a two-node pool, so you have to be willing to live with the issues that arise from that if you still try to leverage high availability.

 

-=Tobias

 

Tobias,

 

I've been using HA-Lizard for a year or so. Last night I turned off HA, shutdown my VM's and then shutdown the slave and then the host. When the servers were restarted, the master nor slave would connect to the HA-Lizard iSCSI storage.

 

I am not lying the blame on HA-Lizard; I even said in my post on their forum it was probably something I did wrong. However, no matter what I do not believe that cleanly shutting down the servers should result in an unusable SR, which is what happened in my case.

 

For this reason (and being up all night restoring from backups) I am moving away from HA-Lizard. It is a great project and SC always answers my posts and is helpful.

For me though, I am moving away from the software.

 

I have a 3rd server that is Windows 2012R2 - I will setup an NFS Share on this and I have some spare Unitrends licences, so I will use this to backup the VM's to this.



Adam Ward Members

Adam Ward
  • 95 posts

Posted 29 September 2016 - 02:56 PM

I have one more option:

 

1 x HP Server with Windows 2012R2 - NFS Share with LACP - use this for VM Storage (allowing XenMotion)

2 x Xenservers using shared storage on the Windows NFS share

1 x HP Server with Windows 2012R2 - NFS share (multipurpose) - use this as a storage repository for Unitrends VM backups...

 

I am wondering if this is overkill.



Tobias Kreidl CTP Member

Tobias Kreidl
  • 18,287 posts

Posted 29 September 2016 - 04:04 PM

Adam,

The first thing to try to do IMO is take a close look at the logs in order to try to determine the nature of the issue. Otherwise, you're going to be doing a lot of guessing and will keep wondering what happened and hence won;t know perhaps how to avoid it. Two important questions: which version of XenServer was being run and was it fully patched with hotfixes?

 

That said, storage is one area where XenServer is a bit fragile. LVM is not the most robust volume management system IMO, but that's a whole different and long topic, but that's what there is to work with. As to overkill, that's IMO hard to do -- critical systems need to have sufficient redundancy. Going to a 3- or 4-server pool would be certainly a good step. We have issues with crashes on one 2-server pool that is HA-related and parly also network related, but Citrix can't help since that configuration isn't supported. If I turn off the Citrix HA, all seems well (and has been for a month now).

 

HA with only local storage is of course not possible really, anyway, plus there is no agility.Using a simple NFS SR for the heartbeat might be one way to circumvent the iSCSI issue, at least. And the NAUbackup option could push backups to local, non-XenServer-related storage on each of the two servers for backups; it just needs an NFS mount available to write to.

 

-=Tobias



Alan Lantz Members

Alan Lantz
  • 6,985 posts

Posted 29 September 2016 - 05:42 PM

Adam, 

 

I don't think that 4 server setup is overkill at all as long as your NFS shares have plenty of space. Also, if you can isolate that storage traffic that would a plus as well.

 

--Alan--



Tobias Kreidl CTP Member
  • #10

Tobias Kreidl
  • 18,287 posts

Posted 29 September 2016 - 06:29 PM

We've switched with our SDS almost exclusively to NFS and have been really pleased with it -- even the performance, which equals iSCSi under these circumstances and is so much easier to set up and maintain. The NFS storage goes over VLANs so more flexibility to better utilize bandwidth without the timing issues or need to isolate to a totally different physical network.

 

-=Tobias



Niklas Åhden Members
  • #11

Niklas Åhden
  • 258 posts

Posted 30 September 2016 - 06:39 AM

I think that you missunderstood Adam, he only has 2 servers for XenServer. I guess it is due to different hardware configurations or simular.

 

I'd setup a pool of the 2 machines you have and then map  a NFS-share from one of the Windows-servers, which ofcourse has a raid-configuration to securely store your data.

 

Then I'd setup NAUbackup and export the VMs every night to quickly be able to recover your setup incase of a failure.

 

THEN i'd setup proper surveillance to be notified in case of a failure or problem with your pool. That way you can just start the VMs manually if there is a host failure.

//Niklas



Adam Ward Members
  • #12

Adam Ward
  • 95 posts

Posted 30 September 2016 - 10:45 AM

Hi,

 

Its about space, power and a waste of very capable servers...

 

I have a 42u rack in a data centre - this is primarily to host other systems, but we also run 4 VM's from this site.

 

The 4 VM's do not require a lot of processing power (1 x FTP server, 1 x IIS Server and 2 x NetScaler VPX's).

 

Running 3 x DL380 G7 servers for this to get HA just seems like massive overkill. However, in an emergency I need to get this back and running within an hour as these are critical systems.

 

I was thinking 2 x XenServers in a Pool (WITHOUT HA) with Direct Attached Storage in a pool as its such a simple setup. I could then use Windows 2012 R2 server that is available to backup via Unitrends UEB to an NFS share.

 

The downsides to this are obviously no XenMotion and no HA. However, this is mitigated slightly as the NetScalers have their own HA built in that does not require XenServer HA.

 

its a dilemma....



papaf76 Members
  • #13

Fabio Papa
  • 72 posts

Posted 30 September 2016 - 10:56 AM

_If_ you decide to go with NFS as a shared storage, don't use windows. Just go with either a storage dedicated appliance (nexentastore or something to that tune) or just plain linux if you're ok with maintaining stuff via command line.

 

And take advantage of ZFS, it's just sooo good :)



Tobias Kreidl CTP Member
  • #14

Tobias Kreidl
  • 18,287 posts

Posted 30 September 2016 - 02:13 PM

Yes, our SDS (NexentaStor) is based on ZFS and performs extremely well even using NFS. As to overkill, Citrix states that three servers is the minimum for a supported pool. As to a two-sever pool, interestingly Ive been working with SC at HA-Lizard over the last few weeks and we've come up with a better way (I think) to support a two-server pool. I've done a lot of torture testing and think there will be a more robust implementation in the near future. I'm particularly interested in that we now have several two-server pools and there is no need to expand just for the sake of supporting HA. Stay tuned for more news on this! For HA to work, you'd still of course need shared pool storage.

 

-=Tobias



Niklas Åhden Members
  • #15

Niklas Åhden
  • 258 posts

Posted 30 September 2016 - 04:52 PM

I agree, use Nexentastor or FreeNAS for serving NFS-shares.

Both has nice web-gui's so they are easy to manage.

//Niklas



Adam Ward Members
  • #16

Adam Ward
  • 95 posts

Posted 01 October 2016 - 09:26 AM

Hi,

 

Quick update - I'm building a test environment using 2 x DL380 G7 servers with XenServer 6.5

 

I am building a freeNAS box to serve out NFS (I will try as shared storage and check speed, if not OK, i will try it as NFS share for Unitrends Backups only).

 

I will see how that goes and I will see if I can put a 3rd server into the Pool too. I will then see how we go.

 

Thank you all for your helpful suggestions - Tobias and Alan - you always respond to my Xen queries, much appreciated! :)



Tobias Kreidl CTP Member
  • #17

Tobias Kreidl
  • 18,287 posts

Posted 01 October 2016 - 04:42 PM

Thanks, Adam. One forgets that HA from Citrix is only one option. There is no reason for example to leave HA off and rely on alerts to sys admins to bring up any failed VMs on a different server. A lot depends on how critical the service is and how quickly you want it back up.

 

I, for one, am continuing to explore HA-Lizard as the Citrix HA option really is too unreliable for a two-server pool and not supported, anyway. We cannot afford to add a third server to a number of pools just for the sake of making HA work, so I am focusing on helping SC to get HA optimized for a two-host pool. Lots of progress has been made already.

 

-=Tobias



Adam Ward Members
  • #18

Adam Ward
  • 95 posts

Posted 02 October 2016 - 10:08 AM

Thanks, Adam. One forgets that HA from Citrix is only one option. There is no reason for example to leave HA off and rely on alerts to sys admins to bring up any failed VMs on a different server. A lot depends on how critical the service is and how quickly you want it back up.

I, for one, am continuing to explore HA-Lizard as the Citrix HA option really is too unreliable for a two-server pool and not supported, anyway. We cannot afford to add a third server to a number of pools just for the sake of making HA work, so I am focusing on helping SC to get HA optimized for a two-host pool. Lots of progress has been made already.

-=Tobias


Hi,

Yes, two node pools with ha whilst not supported by citrix definately have a production usage case.

If ha-lizard is enhanced, I would certainly look at it again. I am interested to know and am actively looking for alternatives that do the same thing, but without dom0 modifications.

Good luck with the ha-lizard work, I hope it goes well and will watch progress with interest.

Regards,

Adam

Tobias Kreidl CTP Member
  • #19

Tobias Kreidl
  • 18,287 posts

Posted 02 October 2016 - 02:27 PM

Honestly, we're running 2-node pools currently without Citrix HA enabled as it's caused too many issues to be "worth it" for the times it does work. Will hopefully make some headway on HA-Lizard updates within the next few weeks.

Cheers,

-=Tobias



Alan Osborne Members
  • #20

Alan Osborne
  • 241 posts

Posted 11 January 2017 - 12:57 AM

Tobias, any news to share on how things are going with HA-Lizard? It's a very intriguing solution and I'm thinking of implementing a 2-node HA cluster to check it out in a lab. Would love to hear how things are progressing with the ideas you've shared with HA-Lizard...