Jump to content


Photo

Reclaim free space broken in 7.0

Started by Florin Baca , 05 September 2016 - 06:06 AM
13 replies to this topic

Florin Baca Members

Florin Baca
  • 5 posts

Posted 05 September 2016 - 06:06 AM

We have 3 XenServers in our infra, the oldest one running a fresh install of 6.5, newest running a fresh install of 7.0 and one which I have upgraded yesterday from 6.5 to 7.0. There were no upgrade issues, aside from guest tools not installing on Windows hosts. Storage is local on all servers.

 

Cleaning the upgraded server, I tried to 'reclaim free storage' which failed with:

 

blkdiscard: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv: BLKDISCARD ioctl failed: Operation not supported

 

I looked in SMlog and I can see the blkdiscard runs after the lvcreate command:

 

Sep  5 07:20:57 xen2 SM: [11766] do_trim: {'sr_uuid': '54fcd1f8-ef26-d32f-1a18-db9e16169231'}
Sep  5 07:20:57 xen2 SM: [11766] lock: opening lock file /var/lock/sm/54fcd1f8-ef26-d32f-1a18-db9e16169231/sr
Sep  5 07:20:57 xen2 SM: [11766] lock: tried lock /var/lock/sm/54fcd1f8-ef26-d32f-1a18-db9e16169231/sr, acquired: True (exists: True)
Sep  5 07:20:57 xen2 SM: [11766] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:57 xen2 SM: [11766] FAILED in util.pread: (rc 5) stdout: '', stderr: '  Failed to find logical volume "VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv"
Sep  5 07:20:57 xen2 SM: [11766] '
Sep  5 07:20:57 xen2 SM: [11766] Ignoring exception for LV check: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv !
Sep  5 07:20:57 xen2 SM: [11766] ['/sbin/lvcreate', '-n', '54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv', '-l', '100%F', 'VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/usr/sbin/blkdiscard', '-v', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766] FAILED in util.pread: (rc 1) stdout: '', stderr: 'blkdiscard: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv: BLKDISCARD ioctl failed: Operation not supported
Sep  5 07:20:58 xen2 SM: [11766] '
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/lvremove', '-f', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/dmsetup', 'status', 'VG_XenStorage--54fcd1f8--ef26--d32f--1a18--db9e16169231-54fcd1f8--ef26--d32f--1a18--db9e16169231_trim_lv']

 

For testing, I ran the same on the freshly installed XenServer 7.0 but I am seeing the exact same error so it's not caused by the upgrade. I don't know if this can be ignored (since it shows in XenCenter as well) but the virtual allocation reported by XenCenter is larger than the actual size of the disk.

 

As a reference, I also reclaimed the free space on the old 6.5 server. The operation succeeds and there is no blkdiscard command in the SMlog.

 

Has anyone else seen these issues with 7.0?

 

Thanks,

Florin



Net Admin Members

Net Admin
  • 2 posts

Posted 11 September 2016 - 09:12 AM

We have the same problem and we are getting below error on XenCenter:

Bulkdiscard ioctl failed: operation not supported

 

have you got any update on this?

 

Thanks,



Florin Baca Members

Florin Baca
  • 5 posts

Posted 12 September 2016 - 08:52 AM

Unfortunately this is still unresolved as I haven't had too much time to look into it.



tahir abbas Members

tahir abbas
  • 2 posts

Posted 31 October 2016 - 10:37 AM

I have same problem too...



Francisco Rafael de Assis E. Fonseca Members

Francisco Rafael de Assis E. Fonseca
  • 2 posts

Posted 17 January 2017 - 01:28 PM

I have the same problem, any solution? Thanks!



Sysadmins Linux Members

Sysadmins Linux
  • 2 posts

Posted 28 January 2017 - 05:22 PM

Same here!



Tobias Kreidl Members

Tobias Kreidl
  • 18,867 posts

Posted 28 January 2017 - 07:22 PM

Are these SSD drives? If so, the TRIM operation doesn't work on them as that's not currently supported on XenServer.

Otherwise, try a coalesce leaf run with:

/opt/xensource/sm/cleanup.py -u <UUID of the SR> -x

 

Note that VMs must be shut down for a coalesce of this nature to work on them, if I recall correctly.

 

-=Tobias



Florin Baca Members

Florin Baca
  • 5 posts

Posted 30 January 2017 - 07:03 AM

They are not SSDs, all servers are using SAS or SATA hard drives. They are also different drive models. The third of our server has also been upgraded to XenServer 7 and it shows the same errors.

 

Unfortunately I can't shutdown the VMs right now, but I should give that a try when we have our next maintenance window.



Tyson Moore Members

Tyson Moore
  • 5 posts

Posted 18 March 2017 - 12:00 AM

Same issue here. All the drives are 2.5" SAS. HP and IBM drives. Server is HP DL360 Gen8 with HP Smart Array P420i 1GB. This issue also happened on our last server: HP DL360 G5.

 

https://support.citrix.com/article/CTX217613

https://support.citrix.com/article/CTX216146

 

Have found out that the SAS Controller likely doesn't support TRIM.

 

The issue we had last time was that the space filled up and we couldn't reclaim it, even after deleting snapshots. This led to what was a rather critical situation... we were unable to take snapshots as there wasn't enough space. So if we had a VM crash, or an update stopped the websites from running (we run websites from the VM's), we would be unable to do anything. Thankfully when I found the issue again about 2 - 3 days ago, I came back to the issue the next day and there was just enough space to take a snapshot.

 

So now there is just the issue of trying to claim the space back, as I know it will get to the point where it will be critical to take a snapshot, and I can't delete all of them and be left with nothing to restore from if it doesn't let me take one.



Tobias Kreidl Members
  • #10

Tobias Kreidl
  • 18,867 posts

Posted 20 March 2017 - 12:30 AM

Lack of TRIM would likely cause this condition, regardless whether involving an SSD or spinning drive.

 

-=Tobias



Jerome Eichler Members
  • #11

Jerome Eichler
  • 11 posts

Posted 02 May 2017 - 02:13 PM

Hey Guys,

 

we have the same problem on our XenServer 7.1 installation.

 

Our SR is a Synology appliance and connected with ISCSI to XenServer.

The SR is nearly filled up to 100%, but the VMs themselves have just the half of it, around 450-500GB.

 

Trying to reclaim the free space results in the following error according to /var/Log/SMlog:

 

May  2 16:10:15 xen04 SM: [2932] do_trim: {'sr_uuid': '47fbe998-bb64-4b86-6f55-811f8947fb62'}

May  2 16:10:15 xen04 SM: [2932] lock: opening lock file /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr

May  2 16:10:15 xen04 SM: [2932] lock: tried lock /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr, acquired: True (exists: True)

May  2 16:10:15 xen04 SM: [2932] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:16 xen04 SM: [2932] FAILED in util.pread: (rc 5) stdout: '', stderr: '  Failed to find logical volume "VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv"

May  2 16:10:16 xen04 SM: [2932] '

May  2 16:10:16 xen04 SM: [2932] Ignoring exception for LV check: /dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv !

May  2 16:10:16 xen04 SM: [2932] ['/sbin/vgs', '--noheadings', '--nosuffix', '--units', 'b', 'VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62']

May  2 16:10:16 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:16 xen04 SM: [2932] ['/sbin/lvcreate', '-n', '47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv', '-l', '100%F', 'VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] ['/usr/sbin/blkdiscard', '-v', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932] FAILED in util.pread: (rc 1) stdout: '', stderr: 'blkdiscard: /dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv: BLKDISCARD ioctl failed: Operation not supported

May  2 16:10:17 xen04 SM: [2932] '

May  2 16:10:17 xen04 SM: [2932] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] ['/sbin/lvremove', '-f', '/dev/VG_XenStorage-47fbe998-bb64-4b86-6f55-811f8947fb62/47fbe998-bb64-4b86-6f55-811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] ['/sbin/dmsetup', 'status', 'VG_XenStorage--47fbe998--bb64--4b86--6f55--811f8947fb62-47fbe998--bb64--4b86--6f55--811f8947fb62_trim_lv']

May  2 16:10:17 xen04 SM: [2932]   pread SUCCESS

May  2 16:10:17 xen04 SM: [2932] lock: released /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr

May  2 16:10:17 xen04 SM: [2932] lock: closed /var/lock/sm/47fbe998-bb64-4b86-6f55-811f8947fb62/sr

 

The disks inside the Synology are not SSD disks.

 

How can we get back our free space?

It is quite annoying as we currently cannot do any snapshots which makes our backup strategy quite unusable for the moment.

 

Any help would be appreciated.

 

Thank you!



Chandrika Srinivasan Citrix Employees
  • #12

Chandrika Srinivasan
  • 38 posts

Posted 22 June 2017 - 08:00 AM

Hi Florin, 

 

We made the change in XenServer 7.0 because TRIM was failing silently on 6.5 when it wasn't supported on the disk/array. On your 6.5 setup, when TRIM "succeeds", can you verify if it actually freed up space on the backend?

 

Thanks,

Chandrika



Florin Baca Members
  • #13

Florin Baca
  • 5 posts

Posted 22 June 2017 - 08:26 AM

Hi Srinivasan,

 

Thanks for the reply! Unfortunately our environment is now all upgraded to 7.0 so I am unable to test if this actually worked in the past but I do not remember to have had issues.

 

-Florin



Chandrika Srinivasan Citrix Employees
  • #14

Chandrika Srinivasan
  • 38 posts

Posted 22 June 2017 - 11:52 AM

Thanks Florin!