Jump to content


Photo

XenServer DR scenario (6.2)

Started by Joe Pawlicki , 16 May 2017 - 10:37 PM
21 replies to this topic

Joe Pawlicki Members

Joe Pawlicki
  • 80 posts

Posted 16 May 2017 - 10:37 PM

I confess I don't understand some of the DR limitations (e.g. no NFS SR).  But shouldn't I be able to set up a DR pool for a few key VMs, without the lengthy export/import time?  Since the primary SR is on NFS, and I can regularly snapmirror it (on netapp), then all the VM data is already there at the DR site.  Aren't there some magic commands for the DR pool to read the metadata & resolve all the funky names, just for the VMs I want?  Yes, the DR pool must be separate due to hardware differences, and no I don't want the entire 30+ VMs in the primary pool.  All examples I find documented assume that one wants to restore the entire pool at DR.



Tobias Kreidl CTP Member

Tobias Kreidl
  • 18,477 posts

Posted 16 May 2017 - 11:29 PM

IMHO, you'd be better off with a separate DR backup and recovery solution than what Citrix currently supplies. Doing your own snapshots and combining that with metadata is certainly one option (metadata for the pool as well as SRs can be readily exported to files and stored where you wish).

 

-=Tobias



Jiri Cerny Members

Jiri Cerny
  • 126 posts

Posted 17 May 2017 - 09:35 AM

Hello Joe,

you can pipe vm-export directly to vm-import, so import and export procedures are done in the same time. XenOrchestra does exactly this procedure and also supports NFS storage for DR purpose (https://xen-orchestra.com/docs/disaster_recovery.html).

I never heart about ability to do "magic" with NFS-SR as you wrote. But this shoud be really nice...

Perhaps something with VM Templates?

 

Very inspirative comments here:
https://xenserver.org/blog/entry/creating-backups-with-xenserver.html#comments

 

 

Jiri



Joe Pawlicki Members

Joe Pawlicki
  • 80 posts

Posted 18 May 2017 - 03:49 PM

I appreciate the recommendations.  The obvious problem I see is that vm export/import can only be done with the vm DOWN, and can require a couple of hours.  This is not a workable DR solution.  It's impractical to manually shutdown, export, import multiple vms, even just weekly.

 

Since my NetApp is getting regular snapshots, I can always have a recent set of vhd files at the DR site.  I also see commands such as "vm-export metadata=true" to get just a vm's metadata.  Why isn't that enough, combined with vhd files?  And why can't the xencenter/import function import from vhd, as the documentation continues to insist?  Nowhere have I found an example of taking a set of vhds and a vm's metadata, and restoring it on a DR server.  Presumably this is what these 3rd-party products do, via the xenserver api, so why the mystery?

 

P.S.  Frankly, the poor grammar on XenOrchestra's website is a big red flag to me.



Niklas Åhden Members

Niklas Åhden
  • 258 posts

Posted 18 May 2017 - 05:13 PM

Xen-Orchestra handles this without VM downtime.

It takes a snapshot, then it does a vm-export and at the same time vm-import on the DR site of that snapshot.

We are currently using this and it is working very well.

//Niklas



Joe Pawlicki Members

Joe Pawlicki
  • 80 posts

Posted 18 May 2017 - 05:38 PM

I'm glad to hear the tool works well, as it's one I'm considering.  But not to beat a dead horse here:  We ALREADY PAY Citrix!

 

The lack of functionality on so many fronts, across many years, makes it obvious they simply don't care.  Products like XenOrchestra are just using the provided APIs, so the gimped nature of Citrix's own tools is aggravating.



Jiri Cerny Members

Jiri Cerny
  • 126 posts

Posted 18 May 2017 - 08:59 PM

I believe that SR thin-provissionig would be the killer feature. Because there is real design problem - if you haven't space on SR, you cannot make snapshot => you cannot export running VM. Much worse is something like continuous differential replication/DR => you need 2 snapshots. You just have to have many space ocupied with operational "useless" data. Thick provisioned LVM based SR's are pain and alternative - ext3, hmm so....well....Unfortunately I have to agree with Joe.

 

Many interesting features requested by users (just look at bugs.xenserver.org), and I'm not talking about something hard to implement, but for example just adoption of ext4 or lz4 for vm-export compression -> there are even unofficial projects/tutorials how to DIY. But Citrix doesn't (or doesn't want?) listen.



Tobias Kreidl CTP Member

Tobias Kreidl
  • 18,477 posts

Posted 19 May 2017 - 05:59 AM

We provide our in-house, CLI-based solution for free that also works with live VMs and snapshots: https://github.com/NAUbackup/VmBackup

 

As to Citrix providing its own backup solution (aside from its DR Platinum offering), see https://xenserver.org/blog/entry/creating-backups-with-xenserver.html

 

If something like this will ever come out of Citrix directly for XenServer is of course a big unknown, but it's interesting that it's being discussed.

 

-=Tobias



Jiri Cerny Members

Jiri Cerny
  • 126 posts

Posted 19 May 2017 - 07:03 AM

God bless Tobias and guys from Northern Arizona University;)

VmBackup is great because it's powerfull and easy to use. Also, if you look at XenOrchestra's blog, you see, how many amazing things can be done via XAPI. But there's no frontend implementation in XenServer/Center. 

 

Ok, XS Platinum has own DR, but there's limitation of requirement of iSCSI, so I suppose that's XS's classical LVMonSCSI SR. I have to admint I've never tested it (because of lack of HW equipment), but I think that is solution is also based on creating snapshots and sending it to DR SR in some way delta mode. So if I have insufficient space on source SR I don't have chance to do DR.

Offline/halted VM backup is impossible because of downtime. Also, I also have to pray that VM's VDI's will not grow close to 2TB.

 

But I repeat it again and again.

Tobias, you know that we discussed this here many times and I can't see the light at the end of the tunnel. 

I really appreciate you (and I must not forget Alan, James, and others great guys) for what you are doing here on the forum and all in all for XenServer. Because of it I'm more sorry and angry for seeing your several years old posts on bug.xenserver.org unresolved. The best example of this is work on making vm-export perform faster. Yes, finally we have it in XS 7.1. But is it really so hard to backport patched vhd-util to XS 6.5/7.0?

 

So we'll see, what new features will be introduced on/after Synergy. Have to be optimist, right?

 

Have a nice day;)

 



Joe Pawlicki Members
  • #10

Joe Pawlicki
  • 80 posts

Posted 19 May 2017 - 05:38 PM

Captured here for posterity, and others who may search, is a procedure that is working so far in testing.  It is admittedly inelegant, and perhaps others can point out flaws, but it's the sort of thing I couldn't find documented anywhere.  I haven't found a way to make use of saved metadata.

 

For a DR environment having (a.) NFS SR on NetApp, (b.) snapmirror to DR NetApp, (c.) dissimilar XenServer pool at DR:

  1. Set up the snapmirror schedule to get regular updates of the NFS SR
  2. Keep a list of important VMs, their uuids and vhd files, for reference during recovery
  3. Upon DR event, break snapmirror; make mirror accessible via NFS and CIFS
  4. Create NFS SR on DR pool, pointing to mirror
  5. Immediately detach the SR, and from another system (e.g. Linux nfs) rename the old uuid folder name to the new DR SR uuid, to make all vhds visible in XenCenter
  6. "Repair" the SR to reconnect and see files under Storage; unfortunately they won't have descriptions
  7. For each VM, Import using its OS disk (browse to the CIFS share of the mirror location); this copies & creates new vhd
  8. Attach any additional disks to the vm and boot to verify all is well

I have no idea yet if this will work on complicated chains of vhd files.



Joe Pawlicki Members
  • #11

Joe Pawlicki
  • 80 posts

Posted 19 May 2017 - 09:41 PM

Yeah, scratch that (above).  Worked for a simple test with single-vhd and no snapshots, but then broke.

 

I'm pursuing VMbackup, so thanks for that tip.  But this is insane - every solution seems to require an hours-long and storage-wasting copy (vm-export) of the same data that I already have; it sounds to me like that's what XenOrchestrator is doing as well.

 

There really ought to be something similar to what I outlined above, that can take a vm's metadata and reassemble all the vhd files, etc. from the storage mirror.

 

(P.S.  I've come across many threads with links to xenserver.org, but they're always broken; don't know if that's a recent development.) 



Tobias Kreidl CTP Member
  • #12

Tobias Kreidl
  • 18,477 posts

Posted 21 May 2017 - 05:32 AM

If there were a simple underlying file system such as ZFS, snapshots of entire SRs infinitely many incremental snapshots, as wwell as simple exporting and importing of such snapshots would all be so much easier as a backup and recovery mechanism, but Oracle has put restrictions on ZFS general usage that makes this unlikely to happen because of licensing limitations.

 

I'm off at Citrix Synergy this week, but need to take a look at a number of VMbackup issues that have been posted over the last few weeks. There's just never enough time, it seems, to get to everything that needs attention...

 

-=Tobias



Niklas Åhden Members
  • #13

Niklas Åhden
  • 258 posts

Posted 21 May 2017 - 09:41 AM

Im perfectly fine with the Xen-Orchestra solution, but an advantage that for example Veeam has is that it doesnt need a base-snapshot left at the VM to do delta/incremental snapshots.

If Citrix would provide a way to achieve delta/incremental snapshots without keeping a base-snapshot at the VM the DR-situation would be very much simplified.

 

Im pretty sure Unitrends is able to do this, but with some kind of magic on their own. Not thanks to Citrix.

 

Why do I want/need this? It would dramaticly decrease the time it takes to snapshot and export all my VMs per pool to our DR-site. For now I'll have to do full snapshots/exports which takes about 7-8 hours every night.

//Niklas


Edited by Niklas Åhden, 21 May 2017 - 09:42 AM.


Tobias Kreidl CTP Member
  • #14

Tobias Kreidl
  • 18,477 posts

Posted 21 May 2017 - 01:23 PM

To be able to do incremental snapshots, you must have a base snapshot to leverage. If that base can be thinly-provisioned, then you can save a lot of space. Maybe other products have some trickery to make this work, but AFAIK, not internally within what Citrix can provide.



Niklas Åhden Members
  • #15

Niklas Åhden
  • 258 posts

Posted 21 May 2017 - 01:29 PM

Yes but why cant you use the snapshot you previously exported to disk? I understand that the chain will be broken if you remove one of those files but honestly you shouldnt be removing any backups manually.

In that case, it could just make a full backup and create a new chain.

 

Wouldnt it work to create some kind of hash and then compare it?

I am very interested in how Unitrends (It was named PHDVirtual when I tried it) is handling this, I guess they compare the file you exported previously to the backup you are going to take, but I am not really sure.

 

This is starting to become a real problem in our enviroment as more and more customers wants to buy or DR-service, but we simply dont have enought time during the night to export all the VM's.

 

Luckily we have an ace up our sleve, which is upgrading our network and servers to 10Gb.

//Niklas



Tobias Kreidl CTP Member
  • #16

Tobias Kreidl
  • 18,477 posts

Posted 21 May 2017 - 02:40 PM

In our implementation, we just take a full snapshot, export it, and delete it. But if you want to maintain incrementals, then there has to be a bases snapshot somewhere that you compare against. I have no idea how XenOrchestra, Unitrends, etc. deal with it, but without a base, it can't work. Maybe someone will elaborate on how such mechanisms work.

 

I also encourage taking a look at https://xenserver.org/blog/entry/creating-backups-with-xenserver.html as I think Citrix needs to hear if people think an in-house backup & recovery mechanism is something Citrix should consider.

 

-=Tobias



Niklas Åhden Members
  • #17

Niklas Åhden
  • 258 posts

Posted 21 May 2017 - 03:01 PM

That is what we're doing now as well but it is really time-consuming.

 

I understand that you have to have a reference-point, but why on earth on the SR? It is a waste of expensive 10K SAS/SSD-space since im using ISCSI and I'd rather see it compare to the previously exported VM's .xva-image located on a SATA/7.2K SAS.

 

XOA uses the built in functionality in the XAPI, which requires the above.

Unitrends is doing some magic behind the scenes, or atleast it was.

 

I cannot reply to that blog post, i've tried when it was first posted.

//Niklas



Tobias Kreidl CTP Member
  • #18

Tobias Kreidl
  • 18,477 posts

Posted 21 May 2017 - 03:21 PM

Being able to snapshot to an external SR or other storage device  is on my list of desired features to ask Citrix for while here at the CTP meeting! I have a long list already... :)



Jiri Cerny Members
  • #19

Jiri Cerny
  • 126 posts

Posted 21 May 2017 - 03:49 PM

Having one SR for data (fast but small size) and another for SR is great idea, but I'm not sure, if it's possible to be done with LVM.

I'm affraid that you can't store snapshot on another SR because it should break the chain on original SR. In this case new data should belong to snaphot (on another SR) but not to parent VDI (original SR).

 

Maybe I'm wrong. I hope;)

 

Look here

https://support.citrix.com/article/CTX122978



Niklas Åhden Members
  • #20

Niklas Åhden
  • 258 posts

Posted 21 May 2017 - 04:25 PM

Ive been thinking about the possibility to store snapshots on another storage as well but I think it would involve some complications.

Also, I think the performance would be very much degraded while doing snapshots if you have a very slow SR storing the snapshots and a I/O-heavy machine but I might be wrong :-)

 

The best possible thing they could come up with right now is the possibility for delta/incremental-snapshots without keeping a snapshot on the machine.

 

May I ask what other features you have on ur list Tobias? :-)
//Niklas