vCenter Environment
Hostname: ABCD26p4128.svr.us.ABCDE.net
vCenter Server 6.5 U2c
- From the
vCenter logs we can see that the Task was started, However after sometime
we can see that it was waiting for the script to complete.
2019-08-28T09:04:58.174Z info vpxd[7F5E2F7EF700]
[Originator@6876 sub=vpxLro
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f]
[VpxLRO] — BEGIN task-690280 — HostProfileManager —
vim.profile.host.ProfileManager.applyEntitiesConfiguration — 5292ce86-f778-41ed-8d2d-9f61d89a7d93(52334b71-ac99-3485-62f1-2fe799b0da31)
2019-08-28T09:04:58.174Z info vpxd[7F5E50F6B700] [Originator@6876 sub=vpxLro
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01]
[VpxLRO] — BEGIN lro-18284123 — — HostProfileApply
—
2019-08-28T09:04:58.174Z info vpxd[7F5E50F6B700] [Originator@6876
sub=moHostProfileMgr
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01]
[ApplyHostProfileToHost]: Remediate Started: abcddc200924.svr.us.ABCDE.net
2019-08-28T09:04:59.227Z error vpxd[7F5E50F6B700]
[Originator@6876 sub=moHostProfileMgr
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01]
[ApplyHostConfig] Could not get host profile engine state from host abcddc200924.svr.us.ABCDE.net
2019-08-28T09:14:58.174Z warning vpxd[7F5E2F7EF700]
[Originator@6876 sub=vpxLro
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f]
[VpxLRO] Still waiting
2019-08-28T09:24:58.175Z warning vpxd[7F5E2F7EF700]
[Originator@6876 sub=vpxLro
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f]
[VpxLRO] Still waiting
2019-08-28T09:26:08.057Z warning vpxd[7F5BF4E9D700] [Originator@6876
sub=MoHost] [HandleAsyncQueryPerfResultsDoFinally] The RPC for host host-263 (abcddc200924.svr.us.ABCDE.net)
did not return in time
2019-08-28T09:26:08.125Z warning vpxd[7F5E2C6BA700] [Originator@6876
sub=VpxProfiler opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02]
DoHostSync:host-263 (abcddc200924.svr.us.ABCDE.net) [GetChangesTime] took
1197397 ms
2019-08-28T09:26:08.140Z info vpxd[7F5E2F5EB700] [Originator@6876 sub=vpxLro
opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02-01] [VpxLRO] —
BEGIN lro-18337817 — —
VsanHostConfigIssuesLRO —
2019-08-28T09:26:08.142Z info vpxd[7F5E2F5EB700] [Originator@6876 sub=vpxLro
opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02-01] [VpxLRO] —
FINISH lro-18337817
2019-08-28T09:26:08.143Z warning vpxd[7F5E2C6BA700] [Originator@6876
sub=VpxProfiler opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02]
DoHostSync:host-263 (abcddc200924.svr.us.ABCDE.net) [DoHostSyncTime] took
1197415 ms
2019-08-28T09:26:08.143Z warning vpxd[7F5E2C6BA700] [Originator@6876
sub=VpxProfiler opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02]
InvtHostSyncLRO::StartWork [HostSyncTime] took 1197415 ms
2019-08-28T09:26:08.143Z warning vpxd[7F5E2C6BA700] [Originator@6876
sub=VpxProfiler opID=HB-host-263 (abcddc200924.svr.us.ABCDE.net)@539-524daa02]
VpxLro::LroMain [TotalTime] took 1197415 ms
2019-08-28T09:36:05.447Z error vpxd[7F5BEF7EF700]
[Originator@6876 sub=MoHost] Response from host host-263 (abcddc200924.svr.us.ABCDE.net)
is null, even though no error received
2019-08-28T09:36:05.447Z error vpxd[7F5BEF7EF700]
[Originator@6876 sub=MoHost] [HandleAsyncQueryPerfResultsDoFinally] The RPC for
host host-263 (abcddc200924.svr.us.ABCDE.net) completed with status code 3
2019-08-28T09:45:45.543Z warning vpxd[7F5E508DE700]
[Originator@6876 sub=VpxProfiler
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-SWI-4a4f8303]
InvokeWithOpId [TotalTime] took 581467 ms
2019-08-28T09:45:45.579Z info
vpxd[7F5E2F7EF700] [Originator@6876 sub=moHostProfile
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f]
[CheckCompliance]: Host abcddc200924.svr.us.ABCDE.net is not
compliant with profile
ICP_VSI_CDC2_039-PROFILE-ICP_20_HP_XL170R_6V5U2C_R1:
(vim.profile.ComplianceResult.ComplianceFailure) [
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType = “ActiveDirectory.HostNotJoinedInDomain”,
–>
key = “com.vmware.vim.profile.ComplianceFailure.ActiveDirectory.HostNotJoinedInDomain.label”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType = “Option.Mismatch”,
–>
key =
“com.vmware.vim.profile.ComplianceFailure.Option.Mismatch.label”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
WARNING:
–> (vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType = “Option.Mismatch”,
–>
key =
“com.vmware.vim.profile.ComplianceFailure.Option.Mismatch.label”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType = “Option.Mismatch”,
–>
key =
“com.vmware.vim.profile.ComplianceFailure.Option.Mismatch.label”,
–> value =
“Security.AccountLockFailures”
–>
message = “Option Security.AccountLockFailures doesn’t match the
specified criteria”
–>
expressionName =
“option[“key-vim-profile-host-OptionProfile-Security_AccountLockFailures”]-optionFixed”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
comparisonIdentifier = “Value-[Security.AccountLockFailures]”,
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType =
“com.vmware.vim.profile.Profile.security.ADPermissionProfile.ActiveDirectoryPermissionProfile.NotFound”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType =
“com.vmware.profile.serviceConfig.startupPolicyMismatch”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType =
“com.vmware.profile.serviceConfig.startupPolicyMismatch”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType =
“com.vmware.profile.passwdPAM.PAMSectionMismatch”,
–>
failureValues = <unset>
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType =
“com.vmware.vim.profile.Profile.coredumpPartitionConfig.coredumpPartition.CoredumpPartitionProfile.Different”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType =
“com.vmware.vim.profile.Profile.coredumpFileConfig.coredumpFile.CoredumpFileProfile.Different”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType =
“com.vmware.vim.profile.Profile.sfcbConfig.sfcbConfigProfile.SfcbConfigProfile.Different”,
–> value = “warning”
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType =
“com.vmware.vim.profile.caching.stateless.caching.esx.unmatch.image”,
–>
failureValues = <unset>
–>
(vim.profile.ComplianceResult.ComplianceFailure) {
–>
failureType =
“com.vmware.profile.powerSystem.ChangedFromStatic”,
–>
failureValues =
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues) [
–>
(vim.profile.ComplianceResult.ComplianceFailure.ComplianceFailureValues)
{
2019-08-28T09:45:45.584Z info vpxd[7F5E2F7EF700] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f] [VpxLRO] — FINISH task-690280
2019-08-28T09:45:45.584Z info vpxd[7F5E2F7EF700] [Originator@6876 sub=Default opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f] [VpxLRO] — ERROR task-690280 — HostProfileManager — vim.profile.host.ProfileManager.applyEntitiesConfiguration: vim.fault.HostConfigFailed:
–> (vim.fault.HostConfigFailed) {
–> failure
= (vmodl.MethodFault) [
–>
(vmodl.fault.SystemError) {
–>
reason = “Batch host remediation failed.”
–> failureCriteria =
(vim.host.NetworkPolicy.NicFailureCriteria) {
–>
checkErrorPercent = false,
2019-08-28T09:46:57.544Z warning vpxd[7F5E504D6700] [Originator@6876 sub=VpxProfiler opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-EventManagerProcessJobs-72d5b943] EventManagerProcessJobs [TotalTime] took 71968 ms
Hostname: abcddc200924.svr.us.ABCDE.net
VMware ESXi 6.5.0 build-9298722
ESXi 6.5 U2C
vCenter:
169.84.163.33
vmnic |
PCI bus address |
link |
speed |
duplex |
MTU |
driver |
driver version |
firmware version |
MAC address |
VID |
DID |
SVID |
SDID |
name |
vmnic0 |
0000:02:00.0 |
Down |
0 |
Half |
1500 |
igbn |
1.04.01 |
1.363888889 |
d0:67:26:cb:ae:4e |
8086 |
1521 |
103c |
337f |
Intel Corporation I350 Gigabit Network Connection |
vmnic1 |
0000:02:00.1 |
Down |
0 |
Half |
1500 |
igbn |
1.04.01 |
1.363888889 |
d0:67:26:cb:ae:4f |
8086 |
1521 |
103c |
337f |
Intel Corporation I350 Gigabit Network Connection |
vmnic2 |
0000:81:00.0 |
Up |
10000 |
Full |
9000 |
ixgbe |
4.5.2-iov |
0x80000835, 1.1904.0 |
48:df:37:22:eb:38 |
8086 |
10fb |
103c |
17d3 |
Intel(R) 82599 10 Gigabit Dual Port Network
Connection |
vmnic3 |
0000:81:00.1 |
Up |
10000 |
Full |
9000 |
ixgbe |
4.5.2-iov |
0x80000835, 1.1904.0 |
48:df:37:22:eb:39 |
8086 |
10fb |
103c |
17d3 |
Intel(R) 82599 10 Gigabit Dual Port Network
Connection |
Syslog:
- From the
Syslog we can see that the Issue is initiated by Domain Join operation
which is getting failed:
2019-08-28T09:04:17Z Host Profiles[104110
opID=ReadyToCompletePageMediator-validate-745780-ngc:
70058476-f3-01-86-0b01]:WARNING: Failed to find property
snmp_AgentProfiles_AgentConfigProfile
2019-08-28T09:04:17Z Host Profiles[104110
opID=ReadyToCompletePageMediator-validate-745780-ngc:
70058476-f3-01-86-0b01]:INFO: Calling GatherData() for profile type
PowerSystemProfile
2019-08-28T09:04:17Z Host Profiles[104110
opID=ReadyToCompletePageMediator-validate-745780-ngc:
70058476-f3-01-86-0b01]:INFO: Calling GatherData() for profile type
CachingProfile
2019-08-28T09:04:17Z HostProfileManager: 2019-08-28 09:04:17,713 [MainProcess
INFO ‘root’ ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01]
b’\n[\n]\n’
2019-08-28T09:04:18Z HostProfileManager: 2019-08-28 09:04:18,037 [MainProcess
INFO ‘root’
ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01]
b’gpt\n1069397 255 63 17179869184\n1 2048 17179869150
AA31E02A400F11DB9590000C2911D1B8 vmfs 0\n’
2019-08-28T09:04:18Z HostProfileManager: 2019-08-28 09:04:18,231 [MainProcess
INFO ‘root’
ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01] b”
2019-08-28T09:04:21Z lwsmd: [lsass] Failed to run
provider specific request (request code = 12, provider =
‘lsa-activedirectory-provider’) -> error = 2692, symbol =
NERR_SetupNotJoined, client pid = 104193
2019-08-28T09:04:31Z HostProfileManager: 2019-08-28 09:04:31,910 [MainProcess
INFO ‘root’
ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01]
Discovered lun — eui.6594a516643418a9a5a97ee90000000c (console
/vmfs/devices/disks/eui.6594a516643418a9a5a97ee90000000c) — EMC ScaleIO
(8388608 MiB, scini)
2019-08-28T09:04:31Z
HostProfileManager: 2019-08-28 09:04:31,912 [MainProcess ERROR ‘root’
ReadyToCompletePageMediator-validate-745780-ngc:70058476-f3-01-86-0b01]
Exception: /dev/disks/eui.6594a516643418a9441a005400000003: unrecognised disk
label
2019-08-28T09:04:41Z Host Profiles[104110
opID=ReadyToCompletePageMediator-validate-745780-ngc:
70058476-f3-01-86-0b01]:INFO: Image compliance passed: False; Failures:
[(LocalizableMessageWithPath) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [], key =
‘com.vmware.vim.profile.caching.stateless.caching.esx.unmatch.image’, arg = (vmodl.KeyAnyValue) [], message = “The host’s current cache is
not equivalent to the booted image.” }]
2019-08-28T09:05:00Z
lwsmd: [lsass] Failed to run provider specific request (request code = 12,
provider = ‘lsa-activedirectory-provider’) -> error = 2692, symbol =
NERR_SetupNotJoined, client pid = 104289
2019-08-28T09:05:01Z crond[67686]: crond: USER root pid 104293 cmd
/bin/hostd-probe.sh ++group=host/vim/vmvisor/hostd-probe/stats/sh
2019-08-28T09:05:01Z syslog[104296]: starting hostd probing.
2019-08-28T09:05:05Z
lwsmd: [lsass] Failed to run provider specific request (request code = 12,
provider = ‘lsa-activedirectory-provider’) -> error = 2692, symbol =
NERR_SetupNotJoined, client pid = 104306
2019-08-28T09:05:05Z
lwsmd: [lsass] Failed to run provider specific request (request code = 12,
provider = ‘lsa-activedirectory-provider’) -> error = 2692, symbol =
NERR_SetupNotJoined, client pid = 104307
Vpxa Logs:
- From the
logs we can see that the Task initiated however it failed with the Error “Security.AccountLockFailures”
- 2019-08-28T09:04:58.143Z info
vpxa[93B4B70] [Originator@6876 sub=vpxLro
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-79]
[VpxLRO] — BEGIN lro-2387 — HostdHostProfileManager —
vim.profile.host.profileEngine.HostProfileManager.queryState —
52f2c147-cf32-ceb8-55ce-bc3dcc25a599
2019-08-28T09:04:58.143Z verbose vpxa[93B4B70] [Originator@6876 sub=proxyManagedObjectImpl opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-79] [ProxyManagedObjectImpl::DispatchAsync] Invoking vim.profile.host.profileEngine.HostProfileManager::queryState with 0 params
2019-08-28T09:04:59.188Z info vpxa[93B4B70] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-79] [VpxLRO] — FINISH lro-2387
2019-08-28T09:04:59.218Z info vpxa[9438B70] [Originator@6876 sub=vpxLro opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-d7] [VpxLRO] — BEGIN lro-2388 — HostdHostProfileManager — vim.profile.host.profileEngine.HostProfileManager.updateTaskConfigSpec — 52f2c147-cf32-ceb8-55ce-bc3dcc25a599
2019-08-28T09:04:59.218Z verbose vpxa[9438B70] [Originator@6876 sub=proxyManagedObjectImpl opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-d7] [ProxyManagedObjectImpl::DispatchAsync] Invoking vim.profile.host.profileEngine.HostProfileManager::updateTaskConfigSpec with 2 params
2019-08-28T09:06:04.802Z error
vpxa[93B4B70] [Originator@6876 sub=PropertyCache opID=WFU-31a2d80f] Failed to
diff ha-host:network, had ManagedObjectReference[], got
ManagedObjectReference[]
2019-08-28T09:26:08.009Z warning
vpxa[93F6B70] [Originator@6876 sub=vpxLro] [VpxLRO] Taskinfo polling sees no
change
2019-08-28T09:26:08.116Z error vpxa[94BCB70] [Originator@6876 sub=PropertyCache
opID=WFU-3cbf2f27] Failed to diff ha-host:network, had
ManagedObjectReference[], got ManagedObjectReference[]
2019-08-28T09:26:08.116Z info vpxa[920EB70] [Originator@6876
sub=VpxaHalResourcePool opID=SWI-51015f90] GetOverrideFailureCount() = 0
MAX_CONSECUTIVE_OVERRIDE_FAILURES = 3
- 2019-08-28T09:36:03.988Z
verbose vpxa[9393B70] [Originator@6876 sub=vpxLro
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df]
[VpxLRO] LRO task-2 has been resumed
2019-08-28T09:36:03.989Z info
vpxa[9393B70] [Originator@6876 sub=Default opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df]
[VpxLRO] — ERROR task-2 — HostdHostProfileManager —
vim.profile.host.profileEngine.HostProfileManager.applyHostConfig:
vmodl.fault.SystemError:
–> Result:
–> (vmodl.fault.SystemError) {
–> msg = “A general
system error occurred: “
–>
key = “Security.AccountLockFailures”,
HostD Logs:
- From the
Hostd Logs we can see that the Domain join Operation failed
[LikewisePerformDomainAction:185]
DJRunJoinProcess(): ERROR_ACCESS_DENIED (5/0): Access is denied
2019-08-28T09:35:18.273Z info hostd[10F81B70] [Originator@6876
sub=Vmsvc.ProductLockerWatcher]
N5boost10filesystem16filesystem_errorE(boost::filesystem::directory_iterator::construct:
No such file or directory: “/usr/lib/vmware/isoimages”)
[LikewisePerformDomainAction:185] DJRunJoinProcess():
ERROR_ACCESS_DENIED (5/0): Access is denied
2019-08-28T09:36:03.982Z error hostd[BCC2B70] [Originator@6876 sub=SOAP
command stub adapter /usr/bin/sh
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb
user=vpxuser] Error: timeout waiting for command
2019-08-28T09:36:03.982Z error hostd[BCC2B70] [Originator@6876 sub=Default
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb
user=vpxuser] AdapterServer caught exception: Operation timed out
2019-08-28T09:36:03.982Z error hostd[BCC2B70] [Originator@6876 sub=Default
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb
user=vpxuser] Backtrace:
–>
[backtrace begin] product: VMware ESX, version: 6.5.0, build: build-9298722,
tag: hostd, cpu: x86, os: esx, buildType: release
–> backtrace[00] libvmacore.so[0x002DBD9F]:
Vmacore::System::Stacktrace::CaptureWork(unsigned int)
–> backtrace[01] libvmacore.so[0x001269F2]:
Vmacore::System::SystemFactoryImpl::CreateQuickBacktrace(Vmacore::Ref<Vmacore::System::Backtrace>&)
–> backtrace[02] libvmacore.so[0x000DD63F]:
Vmacore::Throwable::Throwable(std::string const&)
–> backtrace[03] libvmomi.so[0x0015BF6F]
–> backtrace[04] libvmomi.so[0x0015A3FB]
–> backtrace[05] libvmomi.so[0x000F8DE0]:
Vmomi::StubImpl::_Invoke_Task(Vmomi::ManagedMethod*,
std::vector<Vmacore::Ref<Vmomi::Any>,
std::allocator<Vmacore::Ref<Vmomi::Any> > >&,
Vmacore::Ref<Vmomi::Any>&)
–> backtrace[06] libvmomi.so[0x00161FE9]:
Vmomi::StubBackedManagedObjectImpl::_Invoke(Vmomi::ManagedMethod*,
std::vector<Vmacore::Ref<Vmomi::Any>,
std::allocator<Vmacore::Ref<Vmomi::Any> > >&,
Vmacore::Ref<Vmomi::Any>&)
–> backtrace[07] libvmomi.so[0x000C4E15]
–> backtrace[08] hostd[0x008E221A]
–> backtrace[09] hostd[0x008E29B7]
–> backtrace[10] hostd[0x008E4BD2]
–> backtrace[11] hostd[0x008F82C6]
–> backtrace[12] libvmacore.so[0x002009E9]
–> backtrace[13] libvmacore.so[0x0012B972]: std::function<void
()>::operator()() const
–> backtrace[14] libvmacore.so[0x001F98C9]
–> backtrace[15] libvmacore.so[0x001FB199]
–> backtrace[16] libvmacore.so[0x001FB36B]
–> backtrace[17] libvmacore.so[0x00200466]
–> backtrace[18] libvmacore.so[0x0020137B]:
std::_Function_handler<void (), std::_Bind<std::function<void
(int)> (int)> >::_M_invoke(std::_Any_data const&)
–> backtrace[19] libvmacore.so[0x0012B972]: std::function<void
()>::operator()() const
–> backtrace[20] libvmacore.so[0x001FE8C6]
–> backtrace[21] libvmacore.so[0x001385FF]
–> backtrace[22] libvmacore.so[0x001FA14A]
–> backtrace[23] libvmacore.so[0x001FDEE5]
–> backtrace[24] libvmacore.so[0x002E3BE4]
–> backtrace[25] libpthread.so.0[0x00006CCA]
–> backtrace[26] libc.so.6[0x000D4C3E]
–>
[backtrace end]
Screen
clipping taken: 05-09-2019 07:01 AM
2019-08-28T09:36:03.985Z info hostd[BCC2B70]
[Originator@6876 sub=Vimsvc.TaskManager
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb
user=vpxuser] Task Completed :
haTask–vim.profile.host.profileEngine.HostProfileManager.applyHostConfig-162473602
Status error
2019-08-28T09:45:00.640Z verbose hostd[10DC2B70]
[Originator@6876 sub=Solo.Vmomi
opID=RemediateHostInBatchWizard-applyOnMultiEntity-746234-ngc:70058499-2f-01-df-0beb-0de9]
Arg error:
–> key =
“com.vmware.vim.profile.applyHostConfigError.LikewiseDomainJoinFailed”,
–> value =
“failed”
–> message = “Likewise
(ActiveDirectory) Domain Join operation failed while joining new domain via
username and password..”
pam_unix(vmware-authd:auth): authentication failure;
logname= uid=0 euid=0 tty= ruser= rhost=
user=root
VMKernel Logs:
- From the
Vmkernel Logs I can see that there is some issues with the Vmk0 as
well, as we are getting the errors
of Leaking Routes.
2019-08-28T09:06:04.087Z cpu70:104416)Config: 866:
“ManagementIface” = “”, Old value: “vmk0”
(Status: 0x0)
2019-08-28T09:06:05.019Z cpu70:104416)WARNING: NetDVS: 681: portAlias is NULL
2019-08-28T09:06:05.019Z cpu70:104416)Net: 2324: connected vmk0 to Management Network, portID 0x200000a
2019-08-28T09:06:05.019Z cpu70:104416)Tcpip_Vmk: 124: in6_domifattach: for
ifname=vmk0
2019-08-28T09:06:05.019Z cpu70:104416)Tcpip_Vmk: 124: vmk0:
2019-08-28T09:06:05.021Z cpu70:104416)Config: 866: “ManagementIface”
= “vmk0”, Old value: “” (Status: 0x0)
2019-08-28T09:06:05.021Z cpu70:104416)Config: 866: “HostIPAddr” =
“0.0.0.0”, Old value: “10.19.122.96” (Status: 0x0)
2019-08-28T09:06:05.021Z cpu70:104416)Config: 866: “ManagementAddr” =
“0.0.0.0”, Old value: “10.19.122.96” (Status: 0x0)
2019-08-28T09:06:05.039Z cpu70:104416)Config: 866: “HostIPAddr” =
“10.19.122.96”, Old value: “0.0.0.0” (Status: 0x0)
2019-08-28T09:06:05.039Z cpu70:104416)Config: 866: “ManagementAddr” =
“10.19.122.96”, Old value: “0.0.0.0” (Status: 0x0)
2019-08-28T09:06:05.738Z cpu70:104416)Net: 478:
Opening Netdump Client config: vmknic vmk0
NIC:”vmnic2″, PortGroup:”Management Network”
2019-08-28T09:06:14.098Z cpu33:66661)Tcpip: 1446: Cleaning 5 Leaked routes on
‘vmk0’
2019-08-28T09:06:14.098Z cpu33:66661)Tcpip: 1450: Freeing interface ‘vmk0’
2019-08-28T09:06:17.078Z cpu33:66661)Tcpip_Vmk: 124: nd6_rs_timer: no routers
on ‘vmk0’. Giving up
0:00:00:12.368 cpu0:65536)WARNING: Serial: 274:
logPort initialization failed
0:00:00:12.368 cpu0:65536)WARNING: Serial: 283: gdbPort initialization failed
0:00:00:12.368 cpu0:65536)WARNING: Serial: 300: shellPort initialization failed
0:00:00:12.368 cpu0:65536)WARNING: Serial: 317: consolePort initialization
failed
0:00:00:12.473 cpu71:65607)WARNING: NUMALatency: 1244: Measured local memory
access latency(=277) for node 1 deviates by more than 10 percent from the
minimum observed=208
0:00:00:12.482
cpu0:65536)Init: 893: Vmkernel initialization done.
0:00:00:12.482
cpu0:65536)VMKernel loaded successfully.
2019-08-27T16:15:31.465Z
cpu0:65536)Loading module user …
2019-08-27T16:15:31.480Z
cpu0:65536)user loaded successfully.
2019-08-27T16:16:01.297Z cpu69:66151)Jumpstart plugin
driver-status-check activation failed: driver-status-check->start() failed:
exited with code 1
2019-08-27T16:16:03.416Z cpu43:66372)ScaleIO R2_5:mosCpu_CpuID Failed, can’t
query CPU
2019-08-27T16:16:26.093Z cpu15:66784)Net: 2324: connected vmk0 to Management Network, portID 0x2000006
2019-08-27T16:16:26.093Z cpu15:66784)Tcpip_Vmk: 124: in6_domifattach: for
ifname=vmk0
2019-08-27T16:16:26.093Z cpu15:66784)Tcpip_Vmk: 124: vmk0:
2019-08-27T16:16:26.096Z cpu15:66784)Config: 866: “ManagementIface” =
“vmk0”, Old value: “” (Status: 0x0)
2019-08-27T16:16:26.096Z cpu15:66784)Config: 866: “HostIPAddr” =
“0.0.0.0”, Old value: “127.0.0.1” (Status: 0x0)
2019-08-27T16:16:26.096Z cpu15:66784)Config: 866: “ManagementAddr” =
“0.0.0.0”, Old value: “127.0.0.1” (Status: 0x0)
2019-08-27T16:16:26.113Z cpu15:66784)Config: 866: “HostIPAddr” =
“10.19.122.96”, Old value: “0.0.0.0” (Status: 0x0)
2019-08-27T16:16:26.113Z cpu15:66784)Config: 866: “ManagementAddr” =
“10.19.122.96”, Old value: “0.0.0.0” (Status: 0x0)
2019-08-27T16:16:26.207Z
cpu15:66784)Tcpip: 1153: gateway = 0x0
2019-08-27T16:16:26.265Z cpu15:66784)Net: 478: Opening Netdump Client config:
vmknic vmk0 NIC:”vmnic2″,
PortGroup:”Management Network”
2019-08-27T16:16:26.265Z cpu15:66784)Net: 499: GatewayIPv4: 10.19.122.1,
vmkIPv4: 10.19.122.96
6:38.151Z cpu4:66661)Tcpip_Vmk: 124: nd6_rs_timer: no
routers on ‘vmk0’. Giving up
2019-08-27T16:59:59.877Z cpu11:72806)Boot Successful
Conclusion:
- From the
logs we can see that the Compliance is failed due to which there is some
issues when the Esxi Host is not able to get the IP address and so the
Domain Join operations failed.
- We will
have to review the Host Profile which are being used and check if there is
any correction that we need to make.