I have a serious problem. I'm trying to recover a deleted lvm vdi on XenServer 5.5
After a lot of search work I found a way to create the backup file for the lvm metadata before the delete. Following this topic, I managed to restore the metadata correctly and lvm does see my missing LV:
http://support.citrix.com/article/CTX117508
Before vgcfgrestore, vgdisplay outputs 16 LV's, after the restore and vgchange -ay, it lists 17 LV's. So the last step would be to rescan the SR with 'xe sr-scan uuid'.
But after I ran this, Xen seems to change the lvm config back to the old one, and vgdisply outputs 16 LV's again !
Does anyone know why this happens and how I can reactivate this volume?
Problem with 'xe sr-scan' after vgcfgrestore
Started by Frans van Nispen , 02 January 2011 - 11:59 AM
7 replies to this topic
Members
-
-
#1
Frans van Nispen
-
- 18 posts
Posted 02 January 2011 - 11:59 AM
Members
-
-
#2
Tobias Kreidl
-
- 14,537 posts
Posted 02 January 2011 - 02:47 PM
Did you run sr-probe before the sr-scan? I'm just wondering if that would make a difference.
--Tobias
Members
-
-
#3
Frans van Nispen
-
- 18 posts
Posted 02 January 2011 - 05:16 PM
What should I need to use for 'type' on sr-probe with local lvm storage ?
Frans
Members
-
-
#5
Frans van Nispen
-
- 18 posts
Posted 02 January 2011 - 08:09 PM
Doesn't work :(
Old/current meta:
# vgdisplay
--- Volume group ---
VG Name VG_XenStorage-9e59c965-1368-5cd5-02c7-712559a9e8e2
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 101
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 16
Open LV 9
Max PV 0
Cur PV 1
Act PV 1
VG Size 923.66 GB
PE Size 4.00 MB
Total PE 236458
Alloc PE / Size 77356 / 302.17 GB
Free PE / Size 159102 / 621.49 GB
VG UUID 0iy3T6-FPDg-csLx-4235-1yyg-sFIu-PnM2u8
Then I restore the lvm metadata, first with --test to make sure the backup is ok, then:
# vgcfgrestore VG_XenStorage-9e59c965-1368-5cd5-02c7-712559a9e8e2 -f VG_XenStorage-9e59c965-1368-5cd5-02c7-712559a9e8e2_003.vg
Restored volume group VG_XenStorage-9e59c965-1368-5cd5-02c7-712559a9e8e2
# vgchange -ay VG_XenStorage-9e59c965-1368-5cd5-02c7-712559a9e8e2 17 logical volume(s) in volume group "VG_XenStorage-9e59c965-1368-5cd5-02c7-712559a9e8e2" now active
# vgdisplay --- Volume group ---
VG Name VG_XenStorage-9e59c965-1368-5cd5-02c7-712559a9e8e2
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 100
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 17
Open LV 9
Max PV 0
Cur PV 1
Act PV 1
VG Size 923.66 GB
PE Size 4.00 MB
Total PE 236458
Alloc PE / Size 88379 / 345.23 GB
Free PE / Size 148079 / 578.43 GB
VG UUID 0iy3T6-FPDg-csLx-4235-1yyg-sFIu-PnM2u8
As you can see, the lvm is updated. Also 'lvm' shows the volume with the uuid correctly.
sr-probe with type=lvmoiscsi requires an iSCSI target, so I tried:
# xe sr-probe device-config:device=/dev/sda3 type=lvm
<?xml version="1.0" ?>
<SRlist>
<SR>
<UUID>
9e59c965-1368-5cd5-02c7-712559a9e8e2
</UUID>
<Devlist>
/dev/sda3
</Devlist>
</SR>
</SRlist>
But if I now run:
# xe sr-scan uuid=9e59c965-1368-5cd5-02c7-712559a9e8e2
I'm back to where I started, and vgdisplay shows 16 volumes again in stead of 17. And the volume is indeed missing.
Edited by: Frans van Nispen on 2-jan-2011 15:10
Members
-
-
#6
Tobias Kreidl
-
- 14,537 posts
Posted 02 January 2011 - 08:35 PM
Very strange. Looks like something may have become corrupted at some point so that there's now this discrepancy.
Members
-
-
#7
Alexander Beese
-
- 2 posts
Posted 21 February 2014 - 11:37 PM
I know this is an old thread, but we have exactly the same problem and really have to get these volumes back.
Two volumes where deleted from the volume group VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c.
After following the instructions from http://support.citrix.com/article/CTX117508 we managed to restore the meta data. According to lvscan we have all 7 (including the MGT volumes) volumes:
ACTIVE '/dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/MGT' [4.00 MB] inherit
inactive '/dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/VHD-130c6c15-5620-463a-af83-bd36335dfc0a' [20.05 GB] inheritinactive '/dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/VHD-18c526d2-f52b-4bae-b7a2-d297dad3e661' [20.05 GB] inheritinactive '/dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/VHD-88bd819b-3b8f-4e56-8202-446c58379c00' [80.16 GB] inheritinactive '/dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/VHD-16a4a31a-0306-4994-8cbc-7cea4bc884d1' [150.30 GB] inheritinactive '/dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/VHD-6cce7e42-7f2c-4765-ac4d-0dc2310b1f0b' [8.02 GB] inheritinactive '/dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/VHD-9948ecc3-85bd-4ff7-8424-412e14bef3fc' [20.05 GB] inherit
When we run "xe sr-scan uuid=732df7ea-3009-033b-13ce-6224dc68272c" the two deleted volumes are being removed. We looked at /var/log/SMlog and found out that these two volumes get deleted by SMGC (garbage collection??):
Feb 21 23:37:01 xenserver-ovamed SMGC: [8727] Found 2 VDIs for deletion:
Feb 21 23:37:01 xenserver-ovamed SMGC: [8727] *130c6c15[VHD](20.000G//20.047G|n)Feb 21 23:37:01 xenserver-ovamed SMGC: [8727] *88bd819b[VHD](80.000G//80.164G|n)Feb 21 23:37:01 xenserver-ovamed SMGC: [8727] Deleting unlinked VDI *130c6c15[VHD](20.000G//20.047G|n)Feb 21 23:37:01 xenserver-ovamed SM: [8727] lock: tried lock /var/lock/sm/732df7ea-3009-033b-13ce-6224dc68272c/sr, acquired: True (exists: True)Feb 21 23:37:01 xenserver-ovamed SM: [8727] ['/usr/sbin/lvremove', '-f', '/dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/VHD-130c6c15-5620-463a-af83-bd36335dfc0a']Feb 21 23:37:01 xenserver-ovamed SM: [8727] pread SUCCESSFeb 21 23:37:01 xenserver-ovamed SM: [8727] ['/sbin/dmsetup', 'status', 'VG_XenStorage--732df7ea--3009--033b--13ce--6224dc68272c-VHD--130c6c15--5620--463a--af83--bd36335dfc0a']Feb 21 23:37:01 xenserver-ovamed SM: [8727] pread SUCCESSFeb 21 23:37:01 xenserver-ovamed SM: [8727] Deleting vdi: 130c6c15-5620-463a-af83-bd36335dfc0a
vhd-util tells us, that BEFORE running xe sr-scan, these two partitions have a hidden flag:
# vhd-util scan -f -c -m VHD-* -l VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c:
vhd=VHD-130c6c15-5620-463a-af83-bd36335dfc0a capacity=21474836480 size=21525168128 hidden=1 parent=none
vhd=VHD-18c526d2-f52b-4bae-b7a2-d297dad3e661 capacity=21474836480 size=21525168128 hidden=0 parent=nonevhd=VHD-88bd819b-3b8f-4e56-8202-446c58379c00 capacity=85899345920 size=86075506688 hidden=1 parent=nonevhd=VHD-16a4a31a-0306-4994-8cbc-7cea4bc884d1 capacity=161061273600 size=161384235008 hidden=0 parent=nonevhd=VHD-6cce7e42-7f2c-4765-ac4d-0dc2310b1f0b capacity=8589934592 size=8615100416 hidden=0 parent=nonevhd=VHD-9948ecc3-85bd-4ff7-8424-412e14bef3fc capacity=21474836480 size=21525168128 hidden=0 parent=none
Our assumption is, that the two volumes get deleted because hidden is set to 1.
Might this be correct?
And if so: How can we remove this hidden flag from a VHD?
We already searched until we hit the end of the internet, but without success so far.
Any help would be really appreciated!
Members
-
-
#8
Alexander Beese
-
- 2 posts
Posted 22 February 2014 - 02:13 PM
First of all: My name is not Alexander Beese, I just use his Citrix account to write at the moment.
We did it!
After reaching the end of the internet, we turned back and found another road which took us to the solution.
Our assumption was correct. Executing sr-scan deleted the hidden VHDs. I think this is usually a reasonable behavior to clean up deleted virtual disks.
In case anyone else needs to recover a deleted virtual disk with XenServer 6 (in this case XenServer 6.2.). This is what we did:
We read http://support.citrix.com/article/CTX117508 which told us to use lvm metadata from /etc/lvm/archive. Unfortunately our version of XenServer had disabled this archive feature of lvm (don't know why), so did not have a previous version of the metadata file.
After reading this article however, we knew the UUID of the volume group (could see it in XenCenter), in our case: VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c.
We looked in the current metadata file (/etc/lvm/backup/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c) and found under "physical_device" the device name (sdb):
device = "/dev/sdb" # Hint only
Since everything we read about recovering a deleted virtual disk talked about using metadata from the archive, we needed another way. Fortunately there was one. We created a lvm dump using:
lvmdump -m -d lvm_dump
This created a directory called lvm_dump containing all sorts of lvm data.
Then we used a text editor (vi in this case) to find the last entry in the file lvm_dump/metadata/sdb before the deletion. This was a binary file, which contained several older versions of our metadata configuration file (/etc/lvm/archive/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c_0000).
The first line of every configuration file version was something like this:
# Generated by LVM2 version 2.02.84(2)-RHEL5 (2011-08-26): Fri Feb 21 21:08:51 2014
The last line was the before the next "# Generated by LVM2" line.
The date and time information helped to find the correct version. The file also contains a sequence number (seqno = ...), which was also helpful to identify the right version.
Using "set number" in vi, we could now get the first and last line number of the file we needed.
We extracted this part of the file using sed:
sed -n 9530,9699p sdb > VG-old
(9530 = first line, 9699 = last line)
The extracted configuration file VG-old still held one block of binary data, which we deleted using vi.
Now we had a valid lvm metadata file, which we could use to restore this virtual disks.
From now on we could follow the guide http://support.citrix.com/article/CTX117508 again with one important exception, see below.
Test the extracted metadata file (VG-old):
vgcfgrestore VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c --test -f VG-old
Result was:
Test mode: Metadata will NOT be updated and volumes will not be (de)activated.
Restored volume group VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c
Execute the same command without "--test":
vgcfgrestore VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c -f VG-old
We used vhd-util to check the state of the volume group:
vhd-util scan -f -c -m VHD-* -l VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c
Result:
vhd=VHD-130c6c15-5620-463a-af83-bd36335dfc0a capacity=21474836480 size=21525168128 hidden=1 parent=none
vhd=VHD-18c526d2-f52b-4bae-b7a2-d297dad3e661 capacity=21474836480 size=21525168128 hidden=0 parent=none
vhd=VHD-88bd819b-3b8f-4e56-8202-446c58379c00 capacity=85899345920 size=86075506688 hidden=1 parent=none
vhd=VHD-16a4a31a-0306-4994-8cbc-7cea4bc884d1 capacity=161061273600 size=161384235008 hidden=0 parent=none
vhd=VHD-6cce7e42-7f2c-4765-ac4d-0dc2310b1f0b capacity=8589934592 size=8615100416 hidden=0 parent=none
vhd=VHD-9948ecc3-85bd-4ff7-8424-412e14bef3fc capacity=21474836480 size=21525168128 hidden=0 parent=none
As you can see our two VHDs had a hidden flag and parent=none. We're really not experts in this stuff, but if our understanding is right, only VHDs that are part of a snapshot or deleted VHDs should be hidden. sr-scan removed these hidden VHDs, so we had to unhide them first:
We activated the volume group:
vgchange -ay VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c
Then changed the hidden flag to 0:
vhd-util set -f hidden -v 0 -n /dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/VHD-130c6c15-5620-463a-af83-bd36335dfc0a
vhd-util set -f hidden -v 0 -n /dev/VG_XenStorage-732df7ea-3009-033b-13ce-6224dc68272c/VHD-88bd819b-3b8f-4e56-8202-446c58379c00
After that we used sr-scan so that XenServer would know about the recovered disks:
xe sr-scan uuid=732df7ea-3009-033b-13ce-6224dc68272c
After that we could see the virtual disks in XenCenter again. All we had to do was give them a new label and attach them to the desired virtual machine.
I hope this is helpful to some other unlucky administrators out there.
