Unable to create vCLs VM on vCenter Server

  • Post category:VMware / Vmware vSphere
  • Post last modified:July 25, 2024
Unable to create vCLs VM on vCenter Server

Sometimes you might see an issue with your vSphere DRS where the DRS functionality stopped working for a cluster. This can generally happen after you have performed an upgrade on your vCenter server to 7.0.

In These scenarios, you will notice that the cluster is having issues deploying the vCLS VMs.

In the logs, you will notice the following errors:

EAM Logs:

  • In the EAM Logs we can see the below Error while trying to Deploy the Agent VMs.
 

2021-02-03T12:10:35.312Z | ERROR | cluster-agent-2 | AuditedJob.java | 106 | JOB FAILED: [#924491140] DeployVmJob(ClusterAgent(ID: ‘Agent:12345678-270d-4a82-8fb8-971f570c31ae:null’))
com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can’t provision VM for ClusterAgent(ID: ‘Agent:12345678-270d-4a82-8fb8-971f570c31ae:null’) due to lack of suitable datastore.
    at com.vmware.eam.job.DeployVmJob.doJob(DeployVmJob.java:317) ~[eam-server.jar:?]
    at com.vmware.eam.job.DeployVmJob.call(DeployVmJob.java:227) ~[eam-server.jar:?]
    at com.vmware.eam.job.DeployVmJob.call(DeployVmJob.java:118) ~[eam-server.jar:?]
    at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) ~[eam-server.jar:?]
    at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) ~[eam-server.jar:?]
    at com.vmware.eam.async.impl.SameThreadJ2SEExecutor.execute(SameThreadJ2SEExecutor.java:18) ~[eam-server.jar:?]
    at com.vmware.eam.async.impl.Executor.submit(Executor.java:40) ~[eam-server.jar:?]
    at com.vmware.eam.job.AgentWorkflowJob.execute(AgentWorkflowJob.java:167) ~[eam-server.jar:?]
    at com.vmware.eam.job.InstallClusterAgentJob.executeNestedJob(InstallClusterAgentJob.java:115) ~[eam-server.jar:?]
    at com.vmware.eam.job.InstallAgentJob.installAgentVM(InstallAgentJob.java:244) ~[eam-server.jar:?]
    at com.vmware.eam.job.InstallAgentJob.lambda$runWorkflow$0(InstallAgentJob.java:159) ~[eam-server.jar:?]
    at com.vmware.eam.agency.impl.ClusterVmDeploymentLock.lambda$withLock$0(ClusterVmDeploymentLock.java:31) ~[eam-server.jar:?]
    at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:89) [eam-server.jar:?]
    at com.vmware.eam.lock.RaceProtection.exec(RaceProtection.java:59) [eam-server.jar:?]
    at com.vmware.eam.agency.impl.ClusterVmDeploymentLock.withLock(ClusterVmDeploymentLock.java:29) [eam-server.jar:?]
    at com.vmware.eam.job.InstallAgentJob.runWorkflow(InstallAgentJob.java:157) [eam-server.jar:?]
    at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:94) [eam-server.jar:?]
    at com.vmware.eam.job.AgentWorkflowJob.call(AgentWorkflowJob.java:48) [eam-server.jar:?]
    at com.vmware.eam.async.impl.AuditedJob.call(AuditedJob.java:58) [eam-server.jar:?]
    at com.vmware.eam.async.impl.FutureRunnable.run(FutureRunnable.java:55) [eam-server.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_261]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_261]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_261]

 

Identity STS Logs:

  • In the Identity STS we can see issues related to Trust and exception were being raised.
 

2021-02-03T11:13:45.603Z
WARN sts[5025:tomcat-http–1099] [CorId=83a454c2-6db6-4301-88dc-2447ac2486b0]
[com.vmware.identity.sts.util.VapiClientConnection] Caught exception invoking
stub type interface com.vmware.vcenter.trust.VcTrusts. Marking connection
invalid so that it can be re-established. Exception was: Unauthenticated
(com.vmware.vapi.std.errors.unauthenticated) => {
    messages = [LocalizableMessage
(com.vmware.vapi.std.localizable_message) => {
    id = vapi.method.authentication.required,
    defaultMessage = This method requires
authentication.,
    args = []
    [dynamic fields]: {
        localized = <unset>,
        params = <unset>
    }
}],
    data = <null>
    [dynamic fields]: {
error_type = UNAUTHENTICATED
    }
}
2021-02-03T11:13:45.693Z ERROR
sts[5025:tomcat-http–1099] [CorId=83a454c2-6db6-4301-88dc-2447ac2486b0]
[com.vmware.identity.sts.util.VcTrustUtil] Not able to read VC-TRUST objects
due to
com.vmware.vapi.std.errors.Unauthenticated:
Unauthenticated (com.vmware.vapi.std.errors.unauthenticated) => {
    messages = [LocalizableMessage
(com.vmware.vapi.std.localizable_message) => {
    id = vapi.method.authentication.required,
    defaultMessage = This method requires
authentication.,
    args = []
    [dynamic fields]: {
        localized = <unset>,
        params = <unset>
    }
}],
    data = <null>
    [dynamic fields]: {

error_type = UNAUTHENTICATED
    }
}
at
com.vmware.vapi.std.errors.Unauthenticated._newInstance(Unauthenticated.java:164)
~[vapi-runtime-7.0.0.jar:?]
    at
sun.reflect.GeneratedMethodAccessor303.invoke(Unknown Source) ~[?:?]
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_261]
    at
java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_261]
    at
com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.createStructBinding(JavaClassStructConverter.java:184)
~[vapi-runtime-7.0.0.jar:?]
    at
com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:77)
~[vapi-runtime-7.0.0.jar:?]
    at
com.vmware.vapi.internal.bindings.convert.impl.JavaClassStructConverter.fromValue(JavaClassStructConverter.java:33)
~[vapi-runtime-7.0.0.jar:?]
    at
com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:346)
~[vapi-runtime-7.0.0.jar:?]

at com.vmware.vapi.bindings.type.ErrorType.accept(ErrorType.java:31)
~[vapi-runtime-7.0.0.jar:?]
    at
com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:688)
~[vapi-runtime-7.0.0.jar:?]
at
com.vmware.vapi.internal.bindings.convert.impl.StaticErrorAnyErrorConverter.fromValue(StaticErrorAnyErrorConverter.java:44)
~[vapi-runtime-7.0.0.jar:?]
at
com.vmware.vapi.internal.bindings.convert.impl.StaticErrorAnyErrorConverter.fromValue(StaticErrorAnyErrorConverter.java:23)
~[vapi-runtime-7.0.0.jar:?]
    at
com.vmware.vapi.internal.bindings.TypeConverterImpl$ValueToJavaVisitor.visit(TypeConverterImpl.java:364)
~[vapi-runtime-7.0.0.jar:?]
at
com.vmware.vapi.bindings.type.AnyErrorType.accept(AnyErrorType.java:13)
~[vapi-runtime-7.0.0.jar:?]
    at
com.vmware.vapi.internal.bindings.TypeConverterImpl.convertToJava(TypeConverterImpl.java:688)
~[vapi-runtime-7.0.0.jar:?]
    at
com.vmware.vapi.internal.bindings.Stub.convert(Stub.java:424)
~[vapi-runtime-7.0.0.jar:?]
at com.vmware.vapi.internal.bindings.Stub.convertError(Stub.java:441)
~[vapi-runtime-7.0.0.jar:?]
    at
com.vmware.vapi.internal.bindings.Stub.access$300(Stub.java:59)
~[vapi-runtime-7.0.0.jar:?]
    at
com.vmware.vapi.internal.bindings.Stub$2.setResult(Stub.java:241)
~[vapi-runtime-7.0.0.jar:?]

Solution:

  • Take offline Snapshot of all the vCenter server and PSCs in the environment.
  • In order to fix this issue you can ran the FixSTS Script:
root@SW730VC123456789 [ /tmp ]# ./fixsts.sh
NOTE: This works on external and embedded PSCs
This script will do the following
1: Regenerate STS certificate
What is needed?
1: Offline snapshots of VCs/PSCs
2: SSO Admin Password
IMPORTANT: This script should only be run on a single PSC per SSO domain
==================================

Resetting STS certificate for SW730VC123456789.abcd.com started on Thu Feb 4 07:48:55 UTC 2021
Detected DN: cn=sw730vcntrpa027.abcd.com,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: SW730VC123456789.abcd.com
Detected PSC: SW730VC123456789.abcd.com
Detected SSO domain name: vsphere.local
Detected Machine ID: 6e615e30-5041-11e6-9eda-00501234567
Detected IP Address: 10.123.11.21
Domain CN: dc=vsphere,dc=local
==================================

==================================Detected Root’s certificate expiration date: 2030 Feb 2
Detected today’s date: 2021 Feb 4

==================================Exporting and generating STS certificateStatus : Success
Using config file : /tmp/vmware-fixsts/certool.cfg
Status : Success
Enter password for administrator@vsphere.local:
Amount of tenant credentials: 1
Exporting tenant 1 to /tmp/vmware-fixstsDeleting tenant 1Amount of trustedcertchains: 1
Exporting trustedcertchain 1 to /tmp/vmware-fixstsDeleting trustedcertchain 1
Applying newly generated STS certificate to SSO domain
adding new entry “cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local”adding new entry “cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local”
Replacement finished – Please restart services on all vCenters and PSCs in your SSO domain

==================================
IMPORTANT: In case you’re using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
==================================

  • Post that restart the Services on vCenter Servers.
  • In case you still see any issues you can restart the vCenter servers.
  • Tried rebooting the vCenter Servers and they came online.

 

Ashutosh Dixit

I am currently working as a Senior Technical Support Engineer with VMware Premier Services for Telco. Before this, I worked as a Technical Lead with Microsoft Enterprise Platform Support for Production and Premier Support. I am an expert in High-Availability, Deployments, and VMware Core technology along with Tanzu and Horizon.

Leave a Reply