RCA 31: ESXi Host went Unresponsive

Hostname: abcdefcesx07.jkt.supporting.corp.abcdef.co.id

 

ESXi Version: ESXi 6.7 U1

 

Time of Issue: 5 PM GMT +7 9th Oct.

Time in GMT: 10:00 AM 9th OCT.

 

 

Hostd Logs:

 

  • Reviewed the Hostd logs and was not able to find much details about the unresponsiveness:

 

2020-10-09T09:47:30.073Z error hostd[2100704] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled
2020-10-09T09:47:34.424Z info hostd[2100704] [Originator@6876 sub=Libs opID=5b629096-43-f4c7 user=vpxuser:SUPPORTING\svc-vrops-mpsd] FILE: FileCreateDirectoryEx: Failed to create /var/run/vmware/tickets/. Error = 17
2020-10-09T09:47:34.425Z info hostd[2100704] [Originator@6876 sub=Libs opID=5b629096-43-f4c7 user=vpxuser:SUPPORTING\svc-vrops-mpsd] FILE: FileCreateDirectoryEx: Failed to create /var/run/vmware/tickets/. Error = 17
2020-10-09T09:47:34.520Z info hostd[2101178] [Originator@6876 sub=Libs opID=5c95cec7-fd-f4cd user=vpxuser:SUPPORTING\svc-vrops-mpsd] FILE: FileCreateDirectoryEx: Failed to create /var/run/vmware/tickets/. Error = 17
2020-10-09T09:47:34.520Z info hostd[2101178] [Originator@6876 sub=Libs opID=5c95cec7-fd-f4cd user=vpxuser:SUPPORTING\svc-vrops-mpsd] FILE: FileCreateDirectoryEx: Failed to create /var/run/vmware/tickets/. Error = 17
2020-10-09T09:48:00.074Z error hostd[2101460] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled
2020-10-09T09:48:30.075Z error hostd[2100783] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled
2020-10-09T09:49:00.075Z error hostd[2101460] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled
2020-10-09T09:49:30.076Z error hostd[2184891] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled
2020-10-09T09:50:00.077Z error hostd[2101227] [Originator@6876 sub=Hostsvc.NsxSpecTracker] Object not found/hostspec disabled
2020-10-09T11:42:48.923Z warning -[2100755] [Originator@6876 sub=Default] Estimated fds limit 5120 > 4096 max supported by setrlimit. Setting fds limit to 4096
2020-10-09T11:42:54.920Z error hostd[2100755] [Originator@6876 sub=Hostsvc] Dvs manager not found

 

2020-10-09T11:43:00.332Z info hostd[2100755] [Originator@6876 sub=Libs] FILE: FileCreateDirectoryEx: Failed to create /var/lib/vmware. Error = 17
2020-10-09T11:43:00.332Z info hostd[2100755] [Originator@6876 sub=Libs] FILE: FileCreateDirectoryEx: Failed to create /var/lib/vmware/hostd. Error = 17
–>    fullName = “VMware ESX build-10302608”,
–>    version = “6.7.0”,
–>    build = “10302608”,

 

VMKernel Logs:

 

  • Logs are filled with the SIOC Errors:

 

2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 256/256, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 33/33, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 256/256, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 33/33, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 256/256, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 33/33, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 256/256, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 33/33, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 256/256, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 33/33, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 256/256, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 477: uw.2590616 (3452510) extraMin/extraFromParent: 33/33, sioc (800) childEmin/eMinLimit: 14052/14080
2020-10-09T09:44:34.182Z cpu83:2590616)MemSchedAdmit: 470: Admission failure in path: sioc/storageRM.2590616/uw.2590616

  • Later machine was rebooted:

 

2020-10-09T09:44:39.426Z cpu64:2592428)ScsiVsi: 2964: Can’t set the maxPathQueueDepth value to more than device advertised maxPathQueueDepth 30
VMB: 66: Reserved 4 MPNs starting @ 0x4a0
VMB: 113: mbMagic: 1badb005, mbInfo 0x600000

 

 

 

Conclusion:

 

 

 

  • Please check if its possible to update your ESXi Host to the U3 and Also Incase if you experience this issue next time, i will recommend you to try the below workaround:

 

1. Check the status of storageRM and sdrsInjector

/etc/init.d/storageRM status

/etc/init.d/sdrsInjector status

 

2. Stop the service

/etc/init.d/storageRM stop

/etc/init.d/sdrsInjector stop

 

3. Start the service

/etc/init.d/storageRM start

/etc/init.d/sdrsInjector start

 

Ashutosh Dixit

I am currently working as a Senior Technical Support Engineer with VMware Premier Services for Telco. Before this, I worked as a Technical Lead with Microsoft Enterprise Platform Support for Production and Premier Support. I am an expert in High-Availability, Deployments, and VMware Core technology along with Tanzu and Horizon.

Leave a Reply