Jump to content


Photo

Reclaim free space broken in 7.0

Started by Florin Baca , 05 September 2016 - 06:06 AM
9 replies to this topic

Florin Baca Members

Florin Baca
  • 4 posts

Posted 05 September 2016 - 06:06 AM

We have 3 XenServers in our infra, the oldest one running a fresh install of 6.5, newest running a fresh install of 7.0 and one which I have upgraded yesterday from 6.5 to 7.0. There were no upgrade issues, aside from guest tools not installing on Windows hosts. Storage is local on all servers.

 

Cleaning the upgraded server, I tried to 'reclaim free storage' which failed with:

 

blkdiscard: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv: BLKDISCARD ioctl failed: Operation not supported

 

I looked in SMlog and I can see the blkdiscard runs after the lvcreate command:

 

Sep  5 07:20:57 xen2 SM: [11766] do_trim: {'sr_uuid': '54fcd1f8-ef26-d32f-1a18-db9e16169231'}
Sep  5 07:20:57 xen2 SM: [11766] lock: opening lock file /var/lock/sm/54fcd1f8-ef26-d32f-1a18-db9e16169231/sr
Sep  5 07:20:57 xen2 SM: [11766] lock: tried lock /var/lock/sm/54fcd1f8-ef26-d32f-1a18-db9e16169231/sr, acquired: True (exists: True)
Sep  5 07:20:57 xen2 SM: [11766] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:57 xen2 SM: [11766] FAILED in util.pread: (rc 5) stdout: '', stderr: '  Failed to find logical volume "VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv"
Sep  5 07:20:57 xen2 SM: [11766] '
Sep  5 07:20:57 xen2 SM: [11766] Ignoring exception for LV check: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv !
Sep  5 07:20:57 xen2 SM: [11766] ['/sbin/lvcreate', '-n', '54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv', '-l', '100%F', 'VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/usr/sbin/blkdiscard', '-v', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766] FAILED in util.pread: (rc 1) stdout: '', stderr: 'blkdiscard: /dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv: BLKDISCARD ioctl failed: Operation not supported
Sep  5 07:20:58 xen2 SM: [11766] '
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/lvs', '--noheadings', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/lvremove', '-f', '/dev/VG_XenStorage-54fcd1f8-ef26-d32f-1a18-db9e16169231/54fcd1f8-ef26-d32f-1a18-db9e16169231_trim_lv']
Sep  5 07:20:58 xen2 SM: [11766]   pread SUCCESS
Sep  5 07:20:58 xen2 SM: [11766] ['/sbin/dmsetup', 'status', 'VG_XenStorage--54fcd1f8--ef26--d32f--1a18--db9e16169231-54fcd1f8--ef26--d32f--1a18--db9e16169231_trim_lv']

 

For testing, I ran the same on the freshly installed XenServer 7.0 but I am seeing the exact same error so it's not caused by the upgrade. I don't know if this can be ignored (since it shows in XenCenter as well) but the virtual allocation reported by XenCenter is larger than the actual size of the disk.

 

As a reference, I also reclaimed the free space on the old 6.5 server. The operation succeeds and there is no blkdiscard command in the SMlog.

 

Has anyone else seen these issues with 7.0?

 

Thanks,

Florin



Net Admin Members

Net Admin
  • 2 posts

Posted 11 September 2016 - 09:12 AM

We have the same problem and we are getting below error on XenCenter:

Bulkdiscard ioctl failed: operation not supported

 

have you got any update on this?

 

Thanks,



Florin Baca Members

Florin Baca
  • 4 posts

Posted 12 September 2016 - 08:52 AM

Unfortunately this is still unresolved as I haven't had too much time to look into it.



tahir abbas Members

tahir abbas
  • 2 posts

Posted 31 October 2016 - 10:37 AM

I have same problem too...



Francisco Rafael de Assis E. Fonseca Members

Francisco Rafael de Assis E. Fonseca
  • 2 posts

Posted 17 January 2017 - 01:28 PM

I have the same problem, any solution? Thanks!



Sysadmins Linux Members

Sysadmins Linux
  • 2 posts

Posted 28 January 2017 - 05:22 PM

Same here!



Tobias Kreidl CTP Member

Tobias Kreidl
  • 17,935 posts

Posted 28 January 2017 - 07:22 PM

Are these SSD drives? If so, the TRIM operation doesn't work on them as that's not currently supported on XenServer.

Otherwise, try a coalesce leaf run with:

/opt/xensource/sm/cleanup.py -u <UUID of the SR> -x

 

Note that VMs must be shut down for a coalesce of this nature to work on them, if I recall correctly.

 

-=Tobias



Florin Baca Members

Florin Baca
  • 4 posts

Posted 30 January 2017 - 07:03 AM

They are not SSDs, all servers are using SAS or SATA hard drives. They are also different drive models. The third of our server has also been upgraded to XenServer 7 and it shows the same errors.

 

Unfortunately I can't shutdown the VMs right now, but I should give that a try when we have our next maintenance window.



Tyson Moore Members

Tyson Moore
  • 5 posts

Posted 18 March 2017 - 12:00 AM

Same issue here. All the drives are 2.5" SAS. HP and IBM drives. Server is HP DL360 Gen8 with HP Smart Array P420i 1GB. This issue also happened on our last server: HP DL360 G5.

 

https://support.citrix.com/article/CTX217613

https://support.citrix.com/article/CTX216146

 

Have found out that the SAS Controller likely doesn't support TRIM.

 

The issue we had last time was that the space filled up and we couldn't reclaim it, even after deleting snapshots. This led to what was a rather critical situation... we were unable to take snapshots as there wasn't enough space. So if we had a VM crash, or an update stopped the websites from running (we run websites from the VM's), we would be unable to do anything. Thankfully when I found the issue again about 2 - 3 days ago, I came back to the issue the next day and there was just enough space to take a snapshot.

 

So now there is just the issue of trying to claim the space back, as I know it will get to the point where it will be critical to take a snapshot, and I can't delete all of them and be left with nothing to restore from if it doesn't let me take one.



Tobias Kreidl CTP Member
  • #10

Tobias Kreidl
  • 17,935 posts

Posted 20 March 2017 - 12:30 AM

Lack of TRIM would likely cause this condition, regardless whether involving an SSD or spinning drive.

 

-=Tobias