Jump to content
Welcome to our new Citrix community!
  • 0

Reclaim free space broken in 7.0


Florin Baca

Question

We have 3 XenServers in our infra, the oldest one running a fresh install of 6.5, newest running a fresh install of 7.0 and one which I have upgraded yesterday from 6.5 to 7.0. There were no upgrade issues, aside from guest tools not installing on Windows hosts. Storage is local on all servers.

 

Cleaning the upgraded server, I tried to 'reclaim free storage' which failed with:

 

blkdiscard: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv: BLKDISCARD ioctl failed: Operation not supported

 

I looked in SMlog and I can see the blkdiscard runs after the lvcreate command:

 

Sep  5 07:20:57 xen2 SM: [11766] do_trim: {'sr_uuid': '54fcd1f8-ef26-d32f-1a18-db9e16169231'}
Sep  5 07:20:57 xen2 SM: [11766] lock: opening lock file /var/lock/sm/54fcd1f8-ef26-d32f-1a18-db9e16169231/sr
Sep  5 07:20:57 xen2 SM: [11766] lock: tried lock /var/lock/sm/54fcd1f8-ef26-d32f-1a18-db9e16169231/sr, acquired: True (exists: True)
Sep  5 07:20:57 xen2 SM: [11766] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:57 xen2 SM: [11766] FAILED in util.pread: (rc 5) stdout: '', stderr: '  Failed to find logical volume "VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv"
Sep  5 07:20:57 xen2 SM: [11766] '
Sep  5 07:20:57 xen2 SM: [11766] Ignoring exception for LV check: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv !
Sep  5 07:20:57 xen2 SM: [11766] ['/sbin/lvcreate', '-n', '54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv', '-l', '100%F', 'VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/usr/sbin/blkdiscard', '-v', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766] FAILED in util.pread: (rc 1) stdout: '', stderr: 'blkdiscard: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv: BLKDISCARD ioctl failed: Operation not supported
Sep  5 07:20:58 xen2 SM: [11766] '
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/lvremove', '-f', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/dmsetup', 'status', 'VG_XenStorage--54fcd1f8--ef26--d32f--1a18--db9e16169231-54fcd1f8--ef26--d32f--1a18--db9e16169231_trim_lv']

 

For testing, I ran the same on the freshly installed XenServer 7.0 but I am seeing the exact same error so it's not caused by the upgrade. I don't know if this can be ignored (since it shows in XenCenter as well) but the virtual allocation reported by XenCenter is larger than the actual size of the disk.

 

As a reference, I also reclaimed the free space on the old 6.5 server. The operation succeeds and there is no blkdiscard command in the SMlog.

 

Has anyone else seen these issues with 7.0?

 

Thanks,

Florin

Link to comment
  • Answers 56
  • Created
  • Last Reply

Top Posters For This Question

Recommended Posts

  • 1
On 11/3/2018 at 5:46 PM, Brian Rummel said:

I too am having this issue with 7.1 CU1.  I only upgraded to 7.1 because I had too, as 6.5 is no longer supported.

I want to add, in my situation, I have a storage EMC storage array that I am connecting to via iSCSI.  Before upgrading to 7.1, when I would migrate or move VMs, the coalesce process would eventually reclaim space from the migrate process.  Now it does not.  I have my storage array set up with multiple LUNS and attach them to my pool as separate SR's.  So even though they show as different SRs, they are on the same array using the some connection and hardware etc.  I have tried the rescan SR to no avail.  I have also tried the reclaim freed space and get the BLKDISCARD error on some of the SRs.  But I can also successfully run it on other SRs.  Keeping in mind they are the same physical hardware and connection.  So the issue cannot be related to some sort of hardware incompatibility, or it would never work on any of the SRs.  I also do not use thin provisioning on my SRs.  So far, no matter what I have tried, I cannot reclaim the lost space.  I am working on migrating VMs off the SR.  But because of the size of some of my VDIs, it is not an easy process.  Is there any word on a long term solution for this?

You might want to try this. In my case this freed up a ton of space. I still plan on retiring XenServer hopefully by year's end. It's too unreliable, too many known issues, and the removed features in new versions is not sustainable. Proceed at your own risk.

 

Remove Orphaned VDIs (Virtual Disk Image)

 

If a live-migration fails, XenServer will often leave behind the VDI image of the VM and it not show up under storage. First list the VMs with 'xe vdi-list'.

 

xe vdi-list

 

...
uuid ( RO)                : e629cbe0-32c9-473e-80cf-9a6426d1111e  <<<This is the UUID of the orphaned VM.
          name-label ( RW): AA-REMOTE-01 0
    name-description ( RW): Created by template provisioner
             sr-uuid ( RO): 5e3bb883-fe1b-e4d0-401c-4006fe1efeb2  <<<NOT THIS!! THIS IS WHERE THE VM IS STORED.
        virtual-size ( RO): 75161927680
            sharable ( RO): false
           read-only ( RO): false
           
           
uuid ( RO)                : 19604cdd-4add-4dce-a671-02bd04ac48c2
          name-label ( RW): BB-REMOTE-01
    name-description ( RW): Created by XenCenter Disk Image Import <<<THIS IS ANOTHER GOOD INDICATOR OF A FAILED XEN EXPORT TAKING UP SPACE.
             sr-uuid ( RO): 5e3bb883-fe1b-e4d0-401c-4006fe1efeb2
        virtual-size ( RO): 84351647744
            sharable ( RO): false
           read-only ( RO): false
...

 

This will list a lot of VMs, and in this case I see the VM 'AA-REMOTE-01 0' is listed as on the system, but I know I am not actively running it on this server. You can also run 'lvscan' to show a list of all logical volumes and show if ACTIVE or inactive:

 

lvscan

...
inactive          '/dev/VG_XenStorage-5e3bb883-fe1b-e4d0-401c-4006fe1efeb2/VHD-e629cbe0-32c9-473e-80cf-9a6426d1111e' [70.14 GiB] inherit
...

 

It will show a longer list, but I can see the UUID matches with the orphaned VM I want to remove, and it is taking up 70gb. Next we delete (destroy) the VDI with 'xe vdi-destroy uuid=UUIDofSystemToDelete'

 

xe vdi-destroy uuid=e629cbe0-32c9-473e-80cf-9a6426d1111e

 

You can run another 'xe vdi-list' to see that the orphaned VDI is gone and also check the storage repository to see if space has been freed.

 

 

  • Like 1
Link to comment
  • 1
On 25/01/2018 at 7:12 AM, Mark Syms said:

No, recovery of space give no error, not the same thing at all. blkdiscard has absolutely no meaning for a local SAS drive. The "reclaim space" operation in XenServer is to allow thinly provisioned remote network storage to deprovisioned no longer used space back to the SAN free pool it is not going to do anything for you on a local SAS drive and arguably should be an option in XenCenter in this case.

 

My definitive test:

 

I installed XenServer 7.4 on a Dell R730 server with RAID 6 SAS disks.
I created a virtual machine of 500GB of disk and I created a snapshot, I removed the snapshot and the space was not recovered. I had an error while performing the space recovery procedure.

 

I formatted, installed XenServer 6.5, recreated a virtual machine with 500GB of disk, created the snapshot, removed the snapshot ... I clicked the button to recover the space and the space was recovered. If the XenServer does not display the Trim error because it has no processing to display it does not matter to me, what matters is that it can rather recover the removed snapshot space, while with version 7.X I have the error and I have no recovery of space.

  • Like 1
Link to comment
  • 0

Are these SSD drives? If so, the TRIM operation doesn't work on them as that's not currently supported on XenServer.

Otherwise, try a coalesce leaf run with:

/opt/xensource/sm/cleanup.py -u <UUID of the SR> -x

 

Note that VMs must be shut down for a coalesce of this nature to work on them, if I recall correctly.

 

-=Tobias

Link to comment
  • 0

They are not SSDs, all servers are using SAS or SATA hard drives. They are also different drive models. The third of our server has also been upgraded to XenServer 7 and it shows the same errors.

 

Unfortunately I can't shutdown the VMs right now, but I should give that a try when we have our next maintenance window.

Link to comment
  • 0

Same issue here. All the drives are 2.5" SAS. HP and IBM drives. Server is HP DL360 Gen8 with HP Smart Array P420i 1GB. This issue also happened on our last server: HP DL360 G5.

 

https://support.citrix.com/article/CTX217613

https://support.citrix.com/article/CTX216146

 

Have found out that the SAS Controller likely doesn't support TRIM.

 

The issue we had last time was that the space filled up and we couldn't reclaim it, even after deleting snapshots. This led to what was a rather critical situation... we were unable to take snapshots as there wasn't enough space. So if we had a VM crash, or an update stopped the websites from running (we run websites from the VM's), we would be unable to do anything. Thankfully when I found the issue again about 2 - 3 days ago, I came back to the issue the next day and there was just enough space to take a snapshot.

 

So now there is just the issue of trying to claim the space back, as I know it will get to the point where it will be critical to take a snapshot, and I can't delete all of them and be left with nothing to restore from if it doesn't let me take one.

Link to comment
  • 0

Hey Guys,

 

we have the same problem on our XenServer 7.1 installation.

 

Our SR is a Synology appliance and connected with ISCSI to XenServer.

The SR is nearly filled up to 100%, but the VMs themselves have just the half of it, around 450-500GB.

 

Trying to reclaim the free space results in the following error according to /var/Log/SMlog:

 

May  2 16:10:15 xen04 SM: [2932] do_trim: {'sr_uuid': '47fbe998-bb64-4b86-6f55-811f8947fb62'}

May  2 16:10:15 xen04 SM: [2932] lock: opening lock file /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr

May  2 16:10:15 xen04 SM: [2932] lock: tried lock /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr, acquired: True (exists: True)

May  2 16:10:15 xen04 SM: [2932] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:16 xen04 SM: [2932] FAILED in util.pread: (rc 5) stdout: '', stderr: '  Failed to find logical volume "VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv"

May  2 16:10:16 xen04 SM: [2932] '

May  2 16:10:16 xen04 SM: [2932] Ignoring exception for LV check: /dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv !

May  2 16:10:16 xen04 SM: [2932] ['/sbin/vgs', '--noheadings', '--nosuffix', '--units', 'b', 'VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62']

May  2 16:10:16 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:16 xen04 SM: [2932] ['/sbin/lvcreate', '-n', '47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv', '-l', '100%F', 'VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] ['/usr/sbin/blkdiscard', '-v', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932] FAILED in util.pread: (rc 1) stdout: '', stderr: 'blkdiscard: /dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv: BLKDISCARD ioctl failed: Operation not supported

May  2 16:10:17 xen04 SM: [2932] '

May  2 16:10:17 xen04 SM: [2932] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] ['/sbin/lvremove', '-f', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] ['/sbin/dmsetup', 'status', 'VG_XenStorage--47fbe998--bb64--4b86--6f55--811f8947fb62-47fbe998--bb64--4b86--6f55--811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] lock: released /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr

May  2 16:10:17 xen04 SM: [2932] lock: closed /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr

 

The disks inside the Synology are not SSD disks.

 

How can we get back our free space?

It is quite annoying as we currently cannot do any snapshots which makes our backup strategy quite unusable for the moment.

 

Any help would be appreciated.

 

Thank you!

Link to comment
  • 0

I have the same problem in XenServer 7.2, when i try to reclaim free space, i get this error:

 

BLKDISCARD ioctl failed: Operation not supported

 

and the disk usage continues high.....

 

I have an SCSI card that does not support TRIM, i know the issue is because of this, but, i need to resolve the problem...

 

Any suggestions would be apreciable! Thanks!

Link to comment
  • 0

Hi All,

 

I am running XenServer 7.1 on one of my servers and tried live migrating some VMs to it which all failed migrating. Now my local storate RAID6 SSD array is showing 91% used. It seems the failed migrations are eating up space and I can't find their ghosts. Trying to reclaim freed space I receive the following error:

 

Reclaiming freed space on SR 'Local storage'
blkdiscard: /dev/VG_XenStorage-5e3bb883-fe1b-e4d0-401c-4006fe1efeb2/5e3bb883-fe1b-e4d0-401c-4006fe1efeb2_trim_lv: BLKDISCARD ioctl failed: Operation not supported

 

Any suggestions how to fix this error?

 

Thank you

Link to comment
  • 0

Hello,

 

I have the same problem here, I have servers with XenServer 7 that the command presents the error, and identical machines running version 6.5 with no problem.

I am migrating my machines and I will try to format a host and put XenServer 6.5 again for testing, if this same hardware that presents the TRIM error running with version 6.5 I will reuse 6.5 on all my servers.

 

You can not use XenServer if you can never recover space when you remove VM when you remove snapshot. My virtual machines have HDDs of 600GB, every time I have this problem I lose 600GB. I am prone to migrating to another virtualization solution.

 

Reading this article I was shocked, maybe because I did not have fluent English.

 

Could you please help me: https://support.citrix.com/article/CTX217613 I understood that Citrix is working to change the displayed error phrase and not to solve the problem.

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...