vCenter Environment

Hostname: ABCD26p4128.svr.us.ABCDE.net
vCenter Server 6.5 U2c

• From the vCenter logs we can see that the Task was started, However after sometime we can see that it was waiting for the script to complete.

2019-08-28T09:04:58.174Z info vpxd[7F5E2F7EF700] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f] [VpxLRO] — BEGIN task-690280 — HostProfileManager — vim.profile.host.ProfileManager.applyEntitiesConfiguration — 5292ce86-f778-41ed-8d2d-9f61d89a7d93(52334b71-ac99-3485-62f1-2fe799b0da31)
2019-08-28T09:04:58.174Z info vpxd[7F5E50F6B700] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01] [VpxLRO] — BEGIN lro-18284123 —  — HostProfileApply —
2019-08-28T09:04:58.174Z info vpxd[7F5E50F6B700] [Originator@6876 sub=moHostProfileMgr opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01] [ApplyHostProfileToHost]: Remediate Started: abcddc200924.svr.us.ABCDE.net

2019-08-28T09:04:59.227Z error vpxd[7F5E50F6B700] [Originator@6876 sub=moHostProfileMgr opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01] [ApplyHostConfig] Could not get host profile engine state from host abcddc200924.svr.us.ABCDE.net

2019-08-28T09:14:58.174Z warning vpxd[7F5E2F7EF700] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f] [VpxLRO] Still waiting

2019-08-28T09:24:58.175Z warning vpxd[7F5E2F7EF700] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f] [VpxLRO] Still waiting
2019-08-28T09:26:08.057Z warning vpxd[7F5BF4E9D700] [Originator@6876 sub=MoHost] [HandleAsyncQueryPerfResultsDoFinally] The RPC for host host-263 (abcddc200924.svr.us.ABCDE.net) did not return in time
2019-08-28T09:26:08.125Z warning vpxd[7F5E2C6BA700] [Originator@6876 sub=VpxProfiler opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02] DoHostSync:host-263 (abcddc200924.svr.us.ABCDE.net) [GetChangesTime] took 1197397 ms
2019-08-28T09:26:08.140Z info vpxd[7F5E2F5EB700] [Originator@6876 sub=vpxLro opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02-01] [VpxLRO] — BEGIN lro-18337817 —  — VsanHostConfigIssuesLRO —
2019-08-28T09:26:08.142Z info vpxd[7F5E2F5EB700] [Originator@6876 sub=vpxLro opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02-01] [VpxLRO] — FINISH lro-18337817
2019-08-28T09:26:08.143Z warning vpxd[7F5E2C6BA700] [Originator@6876 sub=VpxProfiler opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02] DoHostSync:host-263 (abcddc200924.svr.us.ABCDE.net) [DoHostSyncTime] took 1197415 ms
2019-08-28T09:26:08.143Z warning vpxd[7F5E2C6BA700] [Originator@6876 sub=VpxProfiler opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02] InvtHostSyncLRO::StartWork [HostSyncTime] took 1197415 ms
2019-08-28T09:26:08.143Z warning vpxd[7F5E2C6BA700] [Originator@6876 sub=VpxProfiler opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02] VpxLro::LroMain [TotalTime] took 1197415 ms

2019-08-28T09:36:05.447Z error vpxd[7F5BEF7EF700] [Originator@6876 sub=MoHost] Response from host host-263 (abcddc200924.svr.us.ABCDE.net) is null, even though no error received

2019-08-28T09:36:05.447Z error vpxd[7F5BEF7EF700] [Originator@6876 sub=MoHost] [HandleAsyncQueryPerfResultsDoFinally] The RPC for host host-263 (abcddc200924.svr.us.ABCDE.net) completed with status code 3

2019-08-28T09:45:45.543Z warning vpxd[7F5E508DE700] [Originator@6876 sub=VpxProfiler opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-SWI-4a4f8303] InvokeWithOpId [TotalTime] took 581467 ms

2019-08-28T09:45:45.579Z info vpxd[7F5E2F7EF700] [Originator@6876 sub=moHostProfile opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f] [CheckCompliance]: Host abcddc200924.svr.us.ABCDE.net is not

compliant with profile ICP_VSI_CDC2_039-PROFILE-ICP_20_HP_XL170R_6V5U2C_R1: (vim.profile.ComplianceResult.ComplianceFailure) [

–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “ActiveDirectory.HostNotJoinedInDomain”,
–>          key = “com.vmware.vim.profile.ComplianceFailure.ActiveDirectory.HostNotJoinedInDomain.label”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “Option.Mismatch”,
–>          key = “com.vmware.vim.profile.ComplianceFailure.Option.Mismatch.label”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–> WARNING:
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “Option.Mismatch”,
–>          key = “com.vmware.vim.profile.ComplianceFailure.Option.Mismatch.label”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “Option.Mismatch”,
–>          key = “com.vmware.vim.profile.ComplianceFailure.Option.Mismatch.label”,
–>                value = “Security.AccountLockFailures”
–>          message = “Option Security.AccountLockFailures doesn’t match the specified criteria”
–>       expressionName = “option[“key-vim-profile-host-OptionProfile-Security_AccountLockFailures”]-optionFixed”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>             comparisonIdentifier = “Value-[Security.AccountLockFailures]”,
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “com.vmware.vim.profile.Profile.security.ADPermissionProfile.ActiveDirectoryPermissionProfile.NotFound”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “com.vmware.profile.serviceConfig.startupPolicyMismatch”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “com.vmware.profile.serviceConfig.startupPolicyMismatch”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “com.vmware.profile.passwdPAM.PAMSectionMismatch”,
–>       failureValues = <unset>
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “com.vmware.vim.profile.Profile.coredumpPartitionConfig.coredumpPartition.CoredumpPartitionProfile.Different”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “com.vmware.vim.profile.Profile.coredumpFileConfig.coredumpFile.CoredumpFileProfile.Different”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “com.vmware.vim.profile.Profile.sfcbConfig.sfcbConfigProfile.SfcbConfigProfile.Different”,
–>                value = “warning”
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “com.vmware.vim.profile.caching.stateless.caching.esx.unmatch.image”,
–>       failureValues = <unset>
–>    (vim.profile.ComplianceResult.ComplianceFailure) {
–>       failureType = “com.vmware.profile.powerSystem.ChangedFromStatic”,
–>       failureValues = (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>          (vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) {

2019-08-28T09:45:45.584Z info vpxd[7F5E2F7EF700] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f] [VpxLRO] — FINISH task-690280

2019-08-28T09:45:45.584Z info vpxd[7F5E2F7EF700] [Originator@6876 sub=Default opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f] [VpxLRO] — ERROR task-690280 — HostProfileManager — vim.profile.host.ProfileManager.applyEntitiesConfiguration: vim.fault.HostConfigFailed:

–> (vim.fault.HostConfigFailed) {
–>    failure = (vmodl.MethodFault) [
–>       (vmodl.fault.SystemError) {
–>          reason = “Batch host remediation failed.”
–>                            failureCriteria = (vim.host.NetworkPolicy.NicFailureCriteria) {
–>                               checkErrorPercent = false,

2019-08-28T09:46:57.544Z warning vpxd[7F5E504D6700] [Originator@6876 sub=VpxProfiler opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-EventManagerProcessJobs-72d5b943] EventManagerProcessJobs [TotalTime] took 71968 ms

Hostname: abcddc200924.svr.us.ABCDE.net

VMware ESXi 6.5.0 build-9298722

ESXi 6.5 U2C

 vCenter: 169.84.163.33

Syslog:

• From the Syslog we can see that the Issue is initiated by Domain Join operation which is getting failed:

2019-08-28T09:04:17Z Host Profiles[104110 opID=ReadyToCompletePageMediator-validate-745780-ngc: 70058476-f3-01-86-0b01]:WARNING: Failed to find property snmp_AgentProfiles_AgentConfigProfile
2019-08-28T09:04:17Z Host Profiles[104110 opID=ReadyToCompletePageMediator-validate-745780-ngc: 70058476-f3-01-86-0b01]:INFO: Calling GatherData() for profile type PowerSystemProfile
2019-08-28T09:04:17Z Host Profiles[104110 opID=ReadyToCompletePageMediator-validate-745780-ngc: 70058476-f3-01-86-0b01]:INFO: Calling GatherData() for profile type CachingProfile
2019-08-28T09:04:17Z HostProfileManager: 2019-08-28 09:04:17,713 [MainProcess INFO ‘root’ ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01] b’\n[\n]\n’
2019-08-28T09:04:18Z HostProfileManager: 2019-08-28 09:04:18,037 [MainProcess INFO ‘root’ ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01] b’gpt\n1069397 255 63 17179869184\n1 2048 17179869150 AA31E02A400F11DB9590000C2911D1B8 vmfs 0\n’
2019-08-28T09:04:18Z HostProfileManager: 2019-08-28 09:04:18,231 [MainProcess INFO ‘root’ ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01] b”

2019-08-28T09:04:21Z lwsmd: [lsass] Failed to run provider specific request (request code = 12, provider = ‘lsa-activedirectory-provider’) -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 104193
2019-08-28T09:04:31Z HostProfileManager: 2019-08-28 09:04:31,910 [MainProcess INFO ‘root’ ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01] Discovered lun — eui.6594a516643418a9a5a97ee90000000c (console /vmfs/devices/disks/eui.6594a516643418a9a5a97ee90000000c) — EMC ScaleIO (8388608 MiB, scini)
2019-08-28T09:04:31Z HostProfileManager: 2019-08-28 09:04:31,912 [MainProcess ERROR ‘root’ ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01] Exception: /dev/disks/eui.6594a516643418a9441a005400000003: unrecognised disk label

2019-08-28T09:04:41Z Host Profiles[104110 opID=ReadyToCompletePageMediator-validate-745780-ngc: 70058476-f3-01-86-0b01]:INFO: Image compliance passed: False; Failures: [(LocalizableMessageWithPath) {    dynamicType = <unset>,    dynamicProperty = (vmodl.DynamicProperty) [],    key = ‘com.vmware.vim.profile.caching.stateless.caching.esx.unmatch.image’,    arg = (vmodl.KeyAnyValue) [],    message = “The host’s current cache is not equivalent to the booted image.” }]
2019-08-28T09:05:00Z lwsmd: [lsass] Failed to run provider specific request (request code = 12, provider = ‘lsa-activedirectory-provider’) -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 104289
2019-08-28T09:05:01Z crond[67686]: crond: USER root pid 104293 cmd /bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
2019-08-28T09:05:01Z syslog[104296]: starting hostd probing.
2019-08-28T09:05:05Z lwsmd: [lsass] Failed to run provider specific request (request code = 12, provider = ‘lsa-activedirectory-provider’) -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 104306
2019-08-28T09:05:05Z lwsmd: [lsass] Failed to run provider specific request (request code = 12, provider = ‘lsa-activedirectory-provider’) -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 104307

Vpxa Logs:

• From the logs we can see that the Task initiated however it failed with the Error “Security.AccountLockFailures”

• 2019-08-28T09:04:58.143Z info vpxa[93B4B70] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-79] [VpxLRO] — BEGIN lro-2387 — HostdHostProfileManager — vim.profile.host.profileEngine.HostProfileManager.queryState — 52f2c147-cf32-ceb8-55ce-bc3dcc25a599
  2019-08-28T09:04:58.143Z verbose vpxa[93B4B70] [Originator@6876 sub=proxyManagedObjectImpl opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-79] [ProxyManagedObjectImpl::DispatchAsync] Invoking vim.profile.host.profileEngine.HostProfileManager::queryState with 0 params
  2019-08-28T09:04:59.188Z info vpxa[93B4B70] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-79] [VpxLRO] — FINISH lro-2387
  2019-08-28T09:04:59.218Z info vpxa[9438B70] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-d7] [VpxLRO] — BEGIN lro-2388 — HostdHostProfileManager — vim.profile.host.profileEngine.HostProfileManager.updateTaskConfigSpec — 52f2c147-cf32-ceb8-55ce-bc3dcc25a599
  2019-08-28T09:04:59.218Z verbose vpxa[9438B70] [Originator@6876 sub=proxyManagedObjectImpl opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-d7] [ProxyManagedObjectImpl::DispatchAsync] Invoking vim.profile.host.profileEngine.HostProfileManager::updateTaskConfigSpec with 2 params

2019-08-28T09:06:04.802Z error vpxa[93B4B70] [Originator@6876 sub=PropertyCache opID=WFU-31a2d80f] Failed to diff ha-host:network, had ManagedObjectReference[], got ManagedObjectReference[]

2019-08-28T09:26:08.009Z warning vpxa[93F6B70] [Originator@6876 sub=vpxLro] [VpxLRO] Taskinfo polling sees no change
2019-08-28T09:26:08.116Z error vpxa[94BCB70] [Originator@6876 sub=PropertyCache opID=WFU-3cbf2f27] Failed to diff ha-host:network, had ManagedObjectReference[], got ManagedObjectReference[]
2019-08-28T09:26:08.116Z info vpxa[920EB70] [Originator@6876 sub=VpxaHalResourcePool opID=SWI-51015f90] GetOverrideFailureCount() = 0 MAX_CONSECUTIVE_OVERRIDE_FAILURES = 3

• 2019-08-28T09:36:03.988Z verbose vpxa[9393B70] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df] [VpxLRO] LRO task-2 has been resumed

2019-08-28T09:36:03.989Z info vpxa[9393B70] [Originator@6876 sub=Default opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df] [VpxLRO] — ERROR task-2 — HostdHostProfileManager — vim.profile.host.profileEngine.HostProfileManager.applyHostConfig: vmodl.fault.SystemError:
–> Result:
–> (vmodl.fault.SystemError) {
–>    msg = “A general system error occurred: “

–>          key = “Security.AccountLockFailures”,

HostD Logs:

• From the Hostd Logs we can see that the Domain join Operation failed

[LikewisePerformDomainAction:185] DJRunJoinProcess(): ERROR_ACCESS_DENIED (5/0): Access is denied
2019-08-28T09:35:18.273Z info hostd[10F81B70] [Originator@6876 sub=Vmsvc.ProductLockerWatcher] N5boost10filesystem16filesystem_errorE(boost::filesystem::directory_iterator::construct: No such file or directory: “/usr/lib/vmware/isoimages”)
[LikewisePerformDomainAction:185] DJRunJoinProcess(): ERROR_ACCESS_DENIED (5/0): Access is denied
2019-08-28T09:36:03.982Z error hostd[BCC2B70] [Originator@6876 sub=SOAP command stub adapter /usr/bin/sh opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb user=vpxuser] Error: timeout waiting for command
2019-08-28T09:36:03.982Z error hostd[BCC2B70] [Originator@6876 sub=Default opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb user=vpxuser] AdapterServer caught exception: Operation timed out
2019-08-28T09:36:03.982Z error hostd[BCC2B70] [Originator@6876 sub=Default opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb user=vpxuser] Backtrace:
–> [backtrace begin] product: VMware ESX, version: 6.5.0, build: build-9298722, tag: hostd, cpu: x86, os: esx, buildType: release
–> backtrace[00] libvmacore.so[0x002DBD9F]: Vmacore::System::Stacktrace::CaptureWork(unsigned int)
–> backtrace[01] libvmacore.so[0x001269F2]: Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
–> backtrace[02] libvmacore.so[0x000DD63F]: Vmacore::Throwable::Throwable(std::string const&)
–> backtrace[03] libvmomi.so[0x0015BF6F]
–> backtrace[04] libvmomi.so[0x0015A3FB]
–> backtrace[05] libvmomi.so[0x000F8DE0]: Vmomi::StubImpl::_Invoke_Task(Vmomi::ManagedMethod*, std::vector<Vmacore::Ref<Vmomi::Any>, std::allocator<Vmacore::Ref<Vmomi::Any> > >&, Vmacore::Ref<Vmomi::Any>&)
–> backtrace[06] libvmomi.so[0x00161FE9]: Vmomi::StubBackedManagedObjectImpl::_Invoke(Vmomi::ManagedMethod*, std::vector<Vmacore::Ref<Vmomi::Any>, std::allocator<Vmacore::Ref<Vmomi::Any> > >&, Vmacore::Ref<Vmomi::Any>&)
–> backtrace[07] libvmomi.so[0x000C4E15]
–> backtrace[08] hostd[0x008E221A]
–> backtrace[09] hostd[0x008E29B7]
–> backtrace[10] hostd[0x008E4BD2]
–> backtrace[11] hostd[0x008F82C6]
–> backtrace[12] libvmacore.so[0x002009E9]
–> backtrace[13] libvmacore.so[0x0012B972]: std::function<void ()>::operator()() const
–> backtrace[14] libvmacore.so[0x001F98C9]
–> backtrace[15] libvmacore.so[0x001FB199]
–> backtrace[16] libvmacore.so[0x001FB36B]
–> backtrace[17] libvmacore.so[0x00200466]
–> backtrace[18] libvmacore.so[0x0020137B]: std::_Function_handler<void (), std::_Bind<std::function<void (int)> (int)> >::_M_invoke(std::_Any_data const&)
–> backtrace[19] libvmacore.so[0x0012B972]: std::function<void ()>::operator()() const
–> backtrace[20] libvmacore.so[0x001FE8C6]
–> backtrace[21] libvmacore.so[0x001385FF]
–> backtrace[22] libvmacore.so[0x001FA14A]
–> backtrace[23] libvmacore.so[0x001FDEE5]
–> backtrace[24] libvmacore.so[0x002E3BE4]
–> backtrace[25] libpthread.so.0[0x00006CCA]
–> backtrace[26] libc.so.6[0x000D4C3E]
–> [backtrace end]

Screen clipping taken: 05-09-2019 07:01 AM

2019-08-28T09:36:03.985Z info hostd[BCC2B70] [Originator@6876 sub=Vimsvc.TaskManager opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb user=vpxuser] Task Completed : haTask–vim.profile.host.profileEngine.HostProfileManager.applyHostConfig-162473602 Status error

2019-08-28T09:45:00.640Z verbose hostd[10DC2B70] [Originator@6876 sub=Solo.Vmomi opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb-0de9] Arg error:
–>          key = “com.vmware.vim.profile.applyHostConfigError.LikewiseDomainJoinFailed”,
–>                value = “failed”
–>          message = “Likewise (ActiveDirectory) Domain Join operation failed while joining new domain via username and password..”

pam_unix(vmware-authd:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=  user=root

VMKernel Logs:

• From the Vmkernel Logs I can see that there is some issues with the Vmk0 as well,  as we are getting the errors of Leaking Routes.

2019-08-28T09:06:04.087Z cpu70:104416)Config: 866: “ManagementIface” = “”, Old value: “vmk0” (Status: 0x0)
2019-08-28T09:06:05.019Z cpu70:104416)WARNING: NetDVS: 681: portAlias is NULL
2019-08-28T09:06:05.019Z cpu70:104416)Net: 2324: connected vmk0  to Management Network, portID 0x200000a
2019-08-28T09:06:05.019Z cpu70:104416)Tcpip_Vmk: 124: in6_domifattach: for ifname=vmk0
2019-08-28T09:06:05.019Z cpu70:104416)Tcpip_Vmk: 124: vmk0:
2019-08-28T09:06:05.021Z cpu70:104416)Config: 866: “ManagementIface” = “vmk0”, Old value: “” (Status: 0x0)
2019-08-28T09:06:05.021Z cpu70:104416)Config: 866: “HostIPAddr” = “0.0.0.0”, Old value: “10.19.122.96” (Status: 0x0)
2019-08-28T09:06:05.021Z cpu70:104416)Config: 866: “ManagementAddr” = “0.0.0.0”, Old value: “10.19.122.96” (Status: 0x0)
2019-08-28T09:06:05.039Z cpu70:104416)Config: 866: “HostIPAddr” = “10.19.122.96”, Old value: “0.0.0.0” (Status: 0x0)
2019-08-28T09:06:05.039Z cpu70:104416)Config: 866: “ManagementAddr” = “10.19.122.96”, Old value: “0.0.0.0” (Status: 0x0)

2019-08-28T09:06:05.738Z cpu70:104416)Net: 478: Opening Netdump Client config: vmknic vmk0  NIC:”vmnic2″, PortGroup:”Management Network”
2019-08-28T09:06:14.098Z cpu33:66661)Tcpip: 1446: Cleaning 5 Leaked routes on ‘vmk0’
2019-08-28T09:06:14.098Z cpu33:66661)Tcpip: 1450: Freeing interface ‘vmk0’
2019-08-28T09:06:17.078Z cpu33:66661)Tcpip_Vmk: 124: nd6_rs_timer: no routers on ‘vmk0’. Giving up

0:00:00:12.368 cpu0:65536)WARNING: Serial: 274: logPort initialization failed
0:00:00:12.368 cpu0:65536)WARNING: Serial: 283: gdbPort initialization failed
0:00:00:12.368 cpu0:65536)WARNING: Serial: 300: shellPort initialization failed
0:00:00:12.368 cpu0:65536)WARNING: Serial: 317: consolePort initialization failed
0:00:00:12.473 cpu71:65607)WARNING: NUMALatency: 1244: Measured local memory access latency(=277) for node 1 deviates by more than 10 percent from the minimum observed=208
0:00:00:12.482 cpu0:65536)Init: 893: Vmkernel initialization done.
0:00:00:12.482 cpu0:65536)VMKernel loaded successfully.
2019-08-27T16:15:31.465Z cpu0:65536)Loading module user …
2019-08-27T16:15:31.480Z cpu0:65536)user loaded successfully.

2019-08-27T16:16:01.297Z cpu69:66151)Jumpstart plugin driver-status-check activation failed: driver-status-check->start() failed: exited with code 1
2019-08-27T16:16:03.416Z cpu43:66372)ScaleIO R2_5:mosCpu_CpuID Failed, can’t query CPU
2019-08-27T16:16:26.093Z cpu15:66784)Net: 2324: connected vmk0  to Management Network, portID 0x2000006
2019-08-27T16:16:26.093Z cpu15:66784)Tcpip_Vmk: 124: in6_domifattach: for ifname=vmk0
2019-08-27T16:16:26.093Z cpu15:66784)Tcpip_Vmk: 124: vmk0:
2019-08-27T16:16:26.096Z cpu15:66784)Config: 866: “ManagementIface” = “vmk0”, Old value: “” (Status: 0x0)
2019-08-27T16:16:26.096Z cpu15:66784)Config: 866: “HostIPAddr” = “0.0.0.0”, Old value: “127.0.0.1” (Status: 0x0)
2019-08-27T16:16:26.096Z cpu15:66784)Config: 866: “ManagementAddr” = “0.0.0.0”, Old value: “127.0.0.1” (Status: 0x0)
2019-08-27T16:16:26.113Z cpu15:66784)Config: 866: “HostIPAddr” = “10.19.122.96”, Old value: “0.0.0.0” (Status: 0x0)
2019-08-27T16:16:26.113Z cpu15:66784)Config: 866: “ManagementAddr” = “10.19.122.96”, Old value: “0.0.0.0” (Status: 0x0)

2019-08-27T16:16:26.207Z cpu15:66784)Tcpip: 1153: gateway = 0x0
2019-08-27T16:16:26.265Z cpu15:66784)Net: 478: Opening Netdump Client config: vmknic vmk0  NIC:”vmnic2″, PortGroup:”Management Network”
2019-08-27T16:16:26.265Z cpu15:66784)Net: 499: GatewayIPv4: 10.19.122.1, vmkIPv4: 10.19.122.96

6:38.151Z cpu4:66661)Tcpip_Vmk: 124: nd6_rs_timer: no routers on ‘vmk0’. Giving up

2019-08-27T16:59:59.877Z cpu11:72806)Boot Successful

Conclusion:

• From the logs we can see that the Compliance is failed due to which there is some issues when the Esxi Host is not able to get the IP address and so the Domain Join operations failed.
• We will have to review the Host Profile which are being used and check if there is any correction that we need to make.