RCA 41: Unable to see AD Users in the vCenter Server

PSC Name: abcvappsc001.gdc.abc.com-

PSC 6.5.0 on Linux, build-17590285

 

  # checked ’17’ signatures against ‘._var_log_vmware_sso_vmware-sts-idmd.log’, file type category = ‘vmware-sts-idmd.log’

    # sig-701, 1 of 1 related to ‘KB50111790’, weight [GSS:100%|TriggerHappy:1.1545%], had 28 matches on 0 of 0 unique days (no time stamp)

    # KB title or comment: ‘Unable to add AD Domain as Identity Source as Active Directory as an LDAP Server’

    # signature regex (what we ‘grep’ for): ‘com.vmware.identity.idm.IDMException: Failed to establish server connection’

    

 com.vmware.identity.idm.IDMException: Failed to establish server connection

      [2021-06-02T03:21:19.576Z vsphere.local        a5347672-0729-4a76-a3d6-2e143ae4eef0 ERROR] [ServerUtils] Exception ‘com.vmware.identity.idm.IDMException: Failed to establish server connection’

      com.vmware.identity.idm.IDMException: Failed to establish server connection

 

 

*************************

**** SSO configuration *****

*************************

– there are ‘3’ identity sources

  IdentitySourceName:  vsphere.local  DomainType:  SYSTEM_DOMAIN

  IdentitySourceName:  ABCVAPPSC001   DomainType:  LOCAL_OS_DOMAIN

  IdentitySourceName:  abc.net        DomainType:  EXTERNAL_DOMAIN

 

 

 

 

Hostname:abcvapvct001.gdc.abc.com

Version: VMware vCenter Server Appliance 6.5.0 (Build Number: 15259038)

 

 

vmware-sts-idmd  Logs:

 

 

[2021-05-30T13:22:05.092Z vsphere.local        444f4315-0508-49de-bc66-35f16e161934 INFO ] [ActiveDirectoryProvider] Failed to find user g.tideway.001@abc.netFailed to establish server connection via ldap search 
[2021-05-30T13:22:05.129Z vsphere.local        0bb76499-709a-4fe7-96ec-e9bba9938409 WARN ] [ServerUtils] cannot bind connection: [ldap://UKARKDOMR01.abc.net, null]
[2021-05-30T13:22:05.129Z vsphere.local        0bb76499-709a-4fe7-96ec-e9bba9938409 ERROR] [ServerUtils] cannot establish connection with uri: ldap://UKARKDOMR01.abc.net
[2021-05-30T13:22:05.129Z vsphere.local        0bb76499-709a-4fe7-96ec-e9bba9938409 INFO ] [ActiveDirectoryProvider] removeDcInfo – domain [abc.net], domainFQDN [UKARKDOMR01.abc.net], domainIpAddress [10.4.142.41]
[2021-05-30T13:22:05.129Z vsphere.local        0bb76499-709a-4fe7-96ec-e9bba9938409 ERROR] [ActiveDirectoryProvider] Failed to get non-GC connection to domain abc.net – domain controller might be offline
com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: 40287][LW_ERROR_LDAP_LOCAL_ERROR][]
    at com.vmware.identity.interop.idm.LinuxIdmNativeAdapter.LdapSaslBind(LinuxIdmNativeAdapter.java:353) ~[vmware-identity-platform.jar:?]
    at com.vmware.identity.interop.ldap.OpenLdapClientLibrary.ldap_sasl_bind_s(OpenLdapClientLibrary.java:735) ~[vmware-identity-platform.jar:?]
    at com.vmware.identity.interop.ldap.LdapConnection.bindSaslConnection(LdapConnection.java:148) ~[vmware-identity-platform.jar:?]
    at com.vmware.identity.idm.server.ServerUtils.getLdapConnection(ServerUtils.java:376) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.ServerUtils.getLdapConnectionByURIs(ServerUtils.java:253) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.provider.BaseLdapProvider.getConnection(BaseLdapProvider.java:379) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.provider.BaseLdapProvider.getConnection(BaseLdapProvider.java:165) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.getLdapConnection(ActiveDirectoryProvider.java:1943) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.getAdConnection(ActiveDirectoryProvider.java:1971) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.getNonGcConnToDomain(ActiveDirectoryProvider.java:1961) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.findUserByLdap(ActiveDirectoryProvider.java:1734) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.findUser(ActiveDirectoryProvider.java:390) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.IdentityManager.findPersonUser(IdentityManager.java:4175) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.IdentityManager.findPersonUser(IdentityManager.java:10082) ~[vmware-identity-idm-server.jar:?]
    at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) ~[?:1.8.0_251]
    at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[?:1.8.0_251]
    at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[?:1.8.0_251]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
    at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) ~[?:1.8.0_251]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) [?:1.8.0_251]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]

 

 

[2021-05-30T00:55:28.759Z vsphere.local        37ba59f3-715e-4c1e-bec2-9f57c1019928 INFO ] [VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[vsphere.local], eventid=[USER_NAME_PWD_AUTH_FAILED], level=[ERROR], category=[VMEVENT_CATEGORY_IDM], text=[org.apache.logging.log4j.core.impl.MutableLogEvent@1dd03b8a], detailText=[Native platform error [code: -1765328360][null][null]], corelationId=[37ba59f3-715e-4c1e-bec2-9f57c1019928], timestamp=[1622336128759]
[2021-05-30T00:55:28.759Z vsphere.local        37ba59f3-715e-4c1e-bec2-9f57c1019928 ERROR] [IdentityManager] Failed to authenticate principal [a.suppro.1508081@zone1.abc.net]. Native platform error [code: -1765328360][null][null]
com.vmware.identity.interop.idm.IdmNativeException: Native platform error [code: -1765328360][null][null]
    at com.vmware.identity.interop.idm.LinuxIdmNativeAdapter.AuthenticateByPassword(LinuxIdmNativeAdapter.java:188) ~[vmware-identity-platform.jar:?]
    at com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider.authenticate(ActiveDirectoryProvider.java:283) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:3018) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:9821) ~[vmware-identity-idm-server.jar:?]
    at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) ~[?:1.8.0_251]
    at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[?:1.8.0_251]
    at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[?:1.8.0_251]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
    at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) ~[?:1.8.0_251]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) [?:1.8.0_251]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
[2021-05-30T00:55:28.760Z vsphere.local        37ba59f3-715e-4c1e-bec2-9f57c1019928 INFO ] [IdentityManager] Authentication failed for user [a.suppro.1508081@zone1.abc.net] in tenant [vsphere.local] in [383] milliseconds with provider [abc.net] of type [com.vmware.identity.idm.server.provider.activedirectory.ActiveDirectoryProvider]
[2021-05-30T00:55:28.760Z vsphere.local        37ba59f3-715e-4c1e-bec2-9f57c1019928 ERROR] [ServerUtils] Exception ‘com.vmware.identity.idm.IDMLoginException: Native platform error [code: -1765328360][null][null]’
com.vmware.identity.idm.IDMLoginException: Native platform error [code: -1765328360][null][null]
    at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:3128) ~[vmware-identity-idm-server.jar:?]
    at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:9821) ~[vmware-identity-idm-server.jar:?]
    at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
    at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357) ~[?:1.8.0_251]
    at sun.rmi.transport.Transport$1.run(Transport.java:200) ~[?:1.8.0_251]
    at sun.rmi.transport.Transport$1.run(Transport.java:197) ~[?:1.8.0_251]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
    at sun.rmi.transport.Transport.serviceCall(Transport.java:196) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688) ~[?:1.8.0_251]
    at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_251]
    at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687) [?:1.8.0_251]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]

 

Conclusion:

 

  • Based on the logs the issue seems to be with the Platform Service Controller unable to access the Domain Controller.

 

  • From the below log, we can see that the PSC is not able to connect to DC: UKARKDOMR01.abc.net

[2021-06-02T03:17:55.141Z vsphere.local        c8c47186-4b99-44d5-b7e7-d73ed8c8654b ERROR] [ServerUtils] cannot establish connection with uri: ldap://UKARKDOMR01.abc.net

 

Action Plan:

 

  • Please make sure that the Domain Controller is online and able to communicate with the Platform Service Controller.

 

 

  • Please review if the Forward and Reverse lookup is configured correctly by running the nslookup command from the PSC for the Domain Controller UKARKDOMR01.

 

  • Please share with me the output for the below files:

/etc/resolv.conf

/var/lib/likewise/krb5-affinity.conf

 

Ashutosh Dixit

I am currently working as a Senior Technical Support Engineer with VMware Premier Services for Telco. Before this, I worked as a Technical Lead with Microsoft Enterprise Platform Support for Production and Premier Support. I am an expert in High-Availability, Deployments, and VMware Core technology along with Tanzu and Horizon.

Leave a Reply