Host Name: abc03esxapp03

Esxi Version: VMware ESXi 6.0.0 build-5050593

Issue Description:

The host Disconnected from VCenter but all the VMs inside the host are running normally.

Time of Issue: 1/5/2019 8:58 AM Indonesian Timezone

Time in GMT: 1/5/2019 1:59 AM GMT

Hostd Logs:

• • •  From the Hostd logs, we can see that the issue started around 2:00 AM when the Processes started failing due to an esxcli process having a lock on the Esx.conf.

2019-05-01T02:00:28.357Z error
 hostd[382C1B70] [Originator@6876 sub=Hostsvc.NetworkProvider opID=f2545e5e] An
 error occurred while fetching stack instance configuration: Error interacting
 with configuration file /etc/vmware/esx.conf: Timout while waiting for lock,
 /etc/vmware/esx.conf.LOCK, to be released. Another process has kept this file
 locked for more than 30 seconds. The process currently holding the lock is
 localcli(24589226).  This is likely a
 temporary condition.  Please try your
 operation again..
2019-05-01T02:00:28.357Z
 warning hostd[382C1B70] [Originator@6876 sub=PropertyProvider opID=f2545e5e]
 It took 197791287 microseconds to get property runtime for
 vim.HostSystem:ha-host
  [LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND
 (2/0):
2019-05-01T02:01:00.557Z
 warning hostd[36EC2B70] [Originator@6876 sub=Hostsvc.NetworkProvider
 opID=f2545a8a user=vpxuser] Error getting dvs a2 24 00 50 33 95 e3 ad-3c ed a9
 4b f0 84 3c 7c : Error interacting with configuration file
 /etc/vmware/esx.conf: Timout while waiting for lock,
 /etc/vmware/esx.conf.LOCK, to be released. Another process has kept this file
 locked for more than 30 seconds. The process currently holding the lock is
 localcli(24589226).  This is likely a
 temporary condition.  Please try your
 operation again.

2019-05-01T02:01:00.557Z
 error hostd[36EC2B70] [Originator@6876 sub=Default opID=f2545a8a user=vpxuser]
 FetchDVPortgroupInfo:Error fetching dvPortgroup info for moid DVPG-a2 24 00 50
 33 95 e3 ad-3c ed a9 4b f0 84 3c 7c-dvportgroup-71346 : vim.fault.PlatformConfigFault,
 ignoring
2019-05-01T02:01:00.560Z
 warning hostd[382C1B70] [Originator@6876 sub=PropertyProvider opID=f2545e5e]
 It took 32200808 microseconds to get property summary for
 vim.VirtualMachine:285
  [LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND
 (2/0):

Syslog:

• Syslogs shows the same error after which the Hostd Service does not respond due to an “Out of Memory Issue”

2019-05-01T01:32:55Z ImageConfigManager:
 2019-05-01 01:32:55,674 [MainProcess ERROR 'HostImage' f81b615e-6bb0-11e9]
 Illegal acceptance level 'Error interacting with configuration file
 /etc/vmware/esx.conf: Timout while waiting for lock, /etc/vmware/esx.conf.LOCK,
 to be released. Another process has kept this file locked for more than 30
 seconds. The process currently holding the lock is localcli(24589226).  This is likely a temporary condition.  Please try your operation again.' obtained
 from configuration 
2019-05-01T16:38:28Z DCUI:
 DCUI aborted by exception: Error interacting with configuration file
 /etc/vmware/esx.conf: Timout while waiting for lock,
 /etc/vmware/esx.conf.LOCK, to be released. Another process has kept this file
 locked for more than 30 seconds. The process currently holding the lock is
 localcli(24589226).  This is likely a
 temporary condition.  Please try your
 operation again.
2019-05-02T08:05:01Z syslog[24651815]:
 starting hostd probing.
  2019-05-02T08:07:31Z Unknown: out of memory [24651858]
  2019-05-02T08:07:31Z Unknown: out of memory [24651858]

Conclusion:

• Based on the Error I can see that the issue happened because an Esxcli Command has a lock over the esx.conf which is not being released properly and due to which the Hostd Terminated with the ” out of memory” Error .
• While researching found few articles:
  • https://kb.vmware.com/kb/2146548
• • https://kb.vmware.com/kb/2145106

• Both Articles talks about the same issue but on different processes.

Recommendation:

• This is a known issue in the Esxi server and is resolve in ESXi 6.0 Update 3a (ESXi 6.0 Patch 5), Currently we are running at ESXi 6.0 Update 3. I would recommend you to update the Esxi Server to the below mentioned release:
• https://my.vmware.com/en/web/vmware/info/slug/datacenter_cloud_infrastructure/vmware_vsphere/6_0.