Failed to Perform Space Reclamation on Machine in Horizon View

  • Post category:Horizon View / VMware
  • Post last modified:August 2, 2024

While working with Vmware I came across this issue where a Customer was not able run the Space reclamation task on one of his Desktop Pools. He was getting the below Error in the Event Logs:

Failed to perform space reclamation on machine XX for Pool XXXX

Failed to Perform Space Reclamation

While looking into the Configuration of the Pool I can see that the Space Reclamation settings are configured correctly:

Failed to Perform Space Reclamation

 

I reviewed the rights of the User which is used to configure Horizon and I can see that the Admin Rights are assigned to him.

Failed to Perform Space Reclamation

Later I started reviewing the logs from the Connection Server to get the Names of the VMs where the Space Reclamation is getting failed.

I Found the Below Errors in the Debug Logs under vdm-logs Folder

2020-05-01T01:38:11.104+08:00 DEBUG (16CC-24C8) <PendingOperation-vm-40974-ReclaimSpace> [EventLogger] Error_Event:[BROKER_MACHINE_SESPARSE_RECLAMATION_FAILED] "Failed to perform space reclamation on machine A in Pool XXX": MachineName=A, PoolId=XXX, Node=XXX.
2020-05-01T02:40:28.646+08:00 DEBUG (16CC-24D8) <PendingOperation-vm-40833-ReclaimSpace> [EventLogger] Error_Event:[BROKER_MACHINE_SESPARSE_RECLAMATION_FAILED] "Failed to perform space reclamation on machine B in Pool XXX": MachineName=B, PoolId=XXX, Node=XXX.
2020-05-01T13:43:18.239+08:00 DEBUG (16CC-2610) <PendingOperation-vm-40953-ReclaimSpace> [EventLogger] Error_Event:[BROKER_MACHINE_SESPARSE_RECLAMATION_FAILED] "Failed to perform space reclamation on machine C in Pool XXX": MachineName=C, PoolId= XXX, Node=XXX.,

 

  • Tracking Events for Machine: A

Time in SGT: 2020-05-01T11:42

Time in GMT: 2020-05-01T03:42

 

Connection Server Debug Logs:

2020-05-01T11:42:31.835+08:00 DEBUG (16CC-2074) <PendingOperation-vm-40974-ReclaimSpace> [ServiceConnection25] Failed, reason: 'Error encountered while shrinking disk.', fault (com.vmware.vim25.ShrinkDiskFault): com.vmware.vim25.ShrinkDiskFault@13aa907
2020-05-01T11:42:31.835+08:00 DEBUG (16CC-2074) <PendingOperation-vm-40974-ReclaimSpace> [ServiceConnection25] Progress: finished (error)
2020-05-01T11:42:31.838+08:00 ERROR (16CC-2074) <PendingOperation-vm-40974-ReclaimSpace> [PendingOperation] Error reclaiming space for VM /CBJ Datacenter/vm/<Location of VM>/A:com.vmware.vdi.vcsupport25.VmException25: Error encountered while shrinking disk.
2020-05-01T11:42:31.838+08:00 DEBUG (16CC-2074) <PendingOperation-vm-40974-ReclaimSpace> [EventLogger] Error_Event:[BROKER_MACHINE_SESPARSE_RECLAMATION_FAILED] "Failed to perform space reclamation on machine A in Pool XXX": MachineName=A, PoolId=XXX,  Severity=ERROR, Time=Fri May 01 11:42:31 SGT 2020, Module=Broker, Source=com.vmware.vdi.desktoptracker.VmInformation, Acknowledged=true

Vmware Logs:

2020-05-01T03:42:31.555Z| vmx| I125:  VAAI-NAS :: SvaNasPlugin: FAILURES: RsrvSpace [0] Cln-Full [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [0] 
2020-05-01T03:42:31.555Z| vmx| I125:  VAAI-NAS :: nutanix_nfs_plugin: FAILURES: RsrvSpace [0] Cln-Full [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [0] 
2020-05-01T03:42:31.555Z| vmx| I125:  VAAI-NAS :: vmfsNasPlugin: FAILURES: RsrvSpace [0] Cln-Full [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [0] 
2020-05-01T03:42:31.890Z| vmx| I125: DISKLIB-SESPARSE :seSparseExtentShrinkCB: Shrink operation on disk failed Inappropriate ioctl for device (1638409)

 

  • Taking reference of VM : C

Time in SGT: 2020-05-01T13:43:18

Time in GMT: 2020-05-01T05:43

 

VMware Logs

2020-05-01T05:43:18.031Z| vmx| I125: DISKLIB-LIB   : Shrink (Asynchronous) chain A97033F6A0.
2020-05-01T05:43:18.031Z| vmx| I125: DISKLIB-SESPARSE :seSparseExtentShrinkCB: Shrink operation on disk failed Inappropriate ioctl for device (1638409)
2020-05-01T06:43:02.539Z| vmx| I125: VigorTransportProcessClientPayload: opID=726ac6ad-23-d5df seq=6492790: Receiving Disk.Wipe request.
2020-05-01T06:43:41.437Z| vmx| I125: nutanix_nfs_plugin: Established VAAI session with NFS server  datastore: <Location of VM> remoteshare: /<Remote Share Name> timeout: 30 seconds crossvol: 1 snap: 1 xfer_size: -1
2020-05-01T06:43:41.437Z| vmx| I125: nutanix_nfs_plugin: ExtentdedStat <Location of VM>/C/C1-internal-flat.vmdk
2020-05-01T06:43:41.438Z| vmx| I125: nutanix_nfs_plugin: Extended stat on <Location of VM>/C/C1-internal-flat.vmdk completed with status: 0
2020-05-01T06:43:41.438Z| vmx| I125:  VAAI-NAS :: SvaNasPlugin: SUCCESSES: RsrvSpace [0] Cln-Full [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [0]
2020-05-01T06:43:41.438Z| vmx| I125:  VAAI-NAS :: SvaNasPlugin: FAILURES: RsrvSpace [0] Cln-Full [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [0]
2020-05-01T06:43:41.438Z| vmx| I125:  VAAI-NAS :: nutanix_nfs_plugin: SUCCESSES: RsrvSpace [0] Cln-Full [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [110]
2020-05-01T06:43:41.438Z| vmx| I125:  VAAI-NAS :: nutanix_nfs_plugin: FAILURES: RsrvSpace [0] Cln-Full [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [0]
2020-05-01T06:43:41.438Z| vmx| I125:  VAAI-NAS :: vmfsNasPlugin: SUCCESSES: RsrvSpace [0] Cln-Full [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [0]
2020-05-01T06:43:41.438Z| vmx| I125:  VAAI-NAS :: vmfsNasPlugin: FAILURES: RsrvSpace [0] Cln-Full [0] Cln-Lazy [0] cln-DRun [0], Ext-stats [0]
2020-05-01T06:43:52.050Z| vmx| I125: DISKLIB-LIB   : Shrink (Asynchronous) chain A97033F6A0.
2020-05-01T06:43:52.050Z| vmx| I125: DISKLIB-SESPARSE :seSparseExtentShrinkCB: Shrink operation on disk failed Inappropriate ioctl for device (1638409)

 

ESXi Host where the VM C was residing.

VMKernel Logs:

2020-05-01T05:43:18.031Z cpu32:55807231)WARNING: CBT: 1107: Unsupported ioctl 46
2020-05-01T06:43:52.050Z cpu32:55807232)WARNING: CBT: 1107: Unsupported ioctl 46

Conclusion:

  • Based on the logs we can conclude that the shrink operation is getting failed from the ESXi Host. While reviewing the logs we found that the Virtual Machines have CBT Enabled on them.
     

ctkEnabled = “TRUE”
scsi1:0.ctkEnabled = “TRUE”
scsi1:1.ctkEnabled = “TRUE”
scsi0:0.ctkEnabled = “TRUE“

As per the Article: https://kb.vmware.com/s/article/2039907, Space Reclamation will not work if CBT is enabled on an SE sparse disk

 

Action Plan:

  • To work around this issue, disable CBT on the parent virtual machine and recompose the pool.
  • Disable CBT for the OS disk (typically scsi0:0 ). For more information, see Enabling Changed Block Tracking (CBT) on virtual machines (1031873).
    Note: Space reclamation only affects the OS disk, so disable the CBT on the OS disk only.

 

Steps To disable CBT:

  • Power off the virtual machine.
  • Right-click the virtual machine and click Edit Settings.
  • Click the Options tab.
  • Click General under the Advanced section and then click Configuration Parameters. The Configuration Parameters dialog opens.
  • Set the ctkEnabled parameter to false for the desired SCSI disk(s).
  • Power on the virtual machine.

Ashutosh Dixit

I am currently working as a Senior Technical Support Engineer with VMware Premier Services for Telco. Before this, I worked as a Technical Lead with Microsoft Enterprise Platform Support for Production and Premier Support. I am an expert in High-Availability, Deployments, and VMware Core technology along with Tanzu and Horizon.

Leave a Reply