Platform Service Controller Upgrade Failure
The Platform Services Controller (PSC) is a component of VMware’s vSphere suite, which provides a set of common infrastructure services for the data center. It was introduced to centralize and streamline various vSphere services, making them easier to manage and ensuring better scalability and consistency across the environment.
While performing an upgrade of a PSC You might come across an issue where the Upgrade operation is failing during the Precheck and you get the Error: Certificate has expired
While reviewing the logs you can see below:
bootstrap.log
2021-04-02T19:07:02.294Z INFO upgrade_commands Reporting source preupgrade result errors and warnings.
2021-04-02T19:07:02.295Z ERROR upgrade_commands Pre-upgrade checks errors:
2021-04-02T19:07:02.295Z ERROR upgrade_commands {‘problemId’: None, ‘text’: {‘id’: ‘upgrade.sso.precheck.error.text’, ‘localized’: ‘Certificate validation failed during pre-upgrade check.’, ‘translatable’: ‘Certificate validation failed during pre-upgrade check.’}, ‘description’: {‘id’: ‘upgrade.sso.precheck.error.description’, ‘localized’: ‘Certificate has expired’, ‘translatable’: ‘Certificate has expired’}, ‘resolution’: {‘id’: ‘upgrade.sso.precheck.error.resolution’, ‘localized’: ‘Regenerate certificates for sso and try again’, ‘translatable’: ‘Regenerate certificates for sso and try again’}}
2021-04-02T19:07:02.296Z ERROR upgrade_commands Pre-upgrade checks failed on source host. Check upgrade-source-requirements.log log for details.
Upgrade Runner Logs:
“name”: “com.vmware.vmafd”
},
{
“optional”: false,
“installedOn”: “127.0.0.1”,
“requirements”: {
“requirementMismatchSpecs”: [
{
“resolution”: {
“localized”: “Regenerate certificates for sso and try again”,
“id”: “upgrade.sso.precheck.error.resolution”,
“translatable”: “Regenerate certificates for sso and try again”
},
“problemId”: null,
“description”: {
“localized”: “Certificate has expired”,
“id”: “upgrade.sso.precheck.error.description”,
“translatable”: “Certificate has expired”
},
“severity”: “ERROR”,
“text”: {
“localized”: “Certificate validation failed during pre-upgrade check.”,
“id”: “upgrade.sso.precheck.error.text”,
“translatable”: “Certificate validation failed during pre-upgrade check.”
}
}
],
Action Plan:
Take an Offline Snapshot of All the PSCs and vCenter that are linked together so that Incase if anything goes wrong we can fall back.
Download the FixSTS Tool:
https://kb.vmware.com/sfc/servlet.shepherd/version/download/068f400000JAn50AAD
Copy the tool to the PSC using WinSCP and give an execute permission to the tool using the below command:
Chmod +x /<Location of Tool>/fixsts.shOnce done you can run the below command and when asked enter your SSO Admin password:
root@yo2-abcpsc [ ~ ]# ./fixsts.sh
NOTE: This works on external and embedded PSCs This script will do the following
1: Regenerate STS certificate
What is needed?
1: Offline snapshots of VCs/PSCs
2: SSO Admin Password
IMPORTANT: This script should only be run on a single PSC per SSO domain ==================================
Resetting STS certificate for yo2-abcpsc.ema.abcd.com started on Sun Apr 4 01:28:19 UTC 2021
Detected DN: cn=yo2-abcpsc.ema.abcd.com,ou=Domain Controllers,dc=vsphere,dc=local Detected PNID: yo2-abcpsc.ema.abcd.com Detected PSC: yo2-abcpsc.ema.abcd.com Detected SSO domain name: vsphere.local Detected Machine ID: 0905d9e2-4aa0-11e6-a0ba-005056abcdef
Detected IP Address: 40.111.1.11
Domain CN: dc=vsphere,dc=local
==================================
==================================
Detected Root’s certificate expiration date: 2026 Jul 12 Detected today’s date: 2021 Apr 4 ==================================
Exporting and generating STS certificate
Status : Success
Using config file : /tmp/vmware-fixsts/certool.cfg Status : Success
Enter password for administrator@vsphere.local
Amount of tenant credentials: 1
Exporting tenant 1 to /tmp/vmware-fixsts
Deleting tenant 1
Amount of trustedcertchains: 2
Exporting trustedcertchain 1 to /tmp/vmware-fixsts
Deleting trustedcertchain 1
Exporting trustedcertchain 2 to /tmp/vmware-fixsts
Deleting trustedcertchain 2
Applying newly generated STS certificate to SSO domain adding new entry “cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local”
adding new entry “cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local”
Replacement finished – Please restart services on all vCenters and PSCs in your SSO domain
==================================
IMPORTANT: In case you’re using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure
=========================================================
