RCA-4: Unable to Bring vCenter online after Destroying vCenter HA

Sometimes you might come across this issue where you have tried to destroy a vCenter HA and post that you are not able to start the vCenter services on the passive Node.

While investigating on this issue you can see the below error messages in the logs:

Lookup Service Logs:

[2023-01-16T08:32:19.755Z localhost-startStop-1 ERROR com.vmware.sync.subscribe.SusbscribeDbStore] Failed to create table lookupsvc_subscriptions
[2023-01-16T08:32:19.761Z localhost-startStop-1 WARN org.springframework.web.context.support.XmlWebApplicationContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'firstMobStep' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'mobAuthenticator' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mobAuthenticator' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'ssoServerAdapter' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAdapter' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot create inner bean 'com.vmware.vim.lookup.impl.SsoServerAdapterImpl#6b7f5274' of type [com.vmware.vim.lookup.impl.SsoServerAdapterImpl] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.vmware.vim.lookup.impl.SsoServerAdapterImpl#6b7f5274' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'serviceRegistration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceRegistration' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'syncAdapter' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'syncAdapter' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'syncAdapterImpl' while setting bean property 'syncAdapterImpl'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'syncAdapterImpl' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'lsSubscribableImpl' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'lsSubscribableImpl' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Invocation of init method failed; nested exception is org.postgresql.util.PSQLException: ERROR: cannot execute CREATE TABLE in a read-only transaction
[2023-01-16T08:32:19.767Z localhost-startStop-1 ERROR org.springframework.web.context.ContextLoader] Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'firstMobStep' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'mobAuthenticator' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mobAuthenticator' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'ssoServerAdapter' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAdapter' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot create inner bean 'com.vmware.vim.lookup.impl.SsoServerAdapterImpl#6b7f5274' of type [com.vmware.vim.lookup.impl.SsoServerAdapterImpl] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.vmware.vim.lookup.impl.SsoServerAdapterImpl#6b7f5274' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'serviceRegistration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceRegistration' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'syncAdapter' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'syncAdapter' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'syncAdapterImpl' while setting bean property 'syncAdapterImpl'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'syncAdapterImpl' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'lsSubscribableImpl' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'lsSubscribableImpl' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Invocation of init method failed; nested exception is org.postgresql.util.PSQLException: ERROR: cannot execute CREATE TABLE in a read-only transaction
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mobAuthenticator' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'ssoServerAdapter' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ssoServerAdapter' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot create inner bean 'com.vmware.vim.lookup.impl.SsoServerAdapterImpl#6b7f5274' of type [com.vmware.vim.lookup.impl.SsoServerAdapterImpl] while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.vmware.vim.lookup.impl.SsoServerAdapterImpl#6b7f5274' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'serviceRegistration' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'serviceRegistration' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'syncAdapter' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'syncAdapter' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'syncAdapterImpl' while setting bean property 'syncAdapterImpl'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'syncAdapterImpl' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Cannot resolve reference to bean 'lsSubscribableImpl' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'lsSubscribableImpl' defined in ServletContext resource [/WEB-INF/tomcat-ls.xml]: Invocation of init method failed; nested exception is org.postgresql.util.PSQLException: ERROR: cannot execute CREATE TABLE in a read-only transaction

Caused by: org.postgresql.util.PSQLException: ERROR: cannot execute CREATE TABLE in a read-only transaction
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468) ~[postgresql-42.2.6.jre7.jar:42.2.6.jre7]

Postgres Logs:

2023-01-16 08:32:10.591 UTC 63c50b8a.3070 0 archiver archiver FATAL: password authentication failed for user "archiver"
2023-01-16 08:32:10.591 UTC 63c50b8a.3070 0 archiver archiver DETAIL: User "archiver" has no password assigned.
Connection matched pg_hba.conf line 7: "local all all md5"
2023-01-16 08:32:10.591 UTC 63c50b8a.3070 0 archiver archiver LOG: could not send data to client: Broken pipe
2023-01-16 08:32:12.554 UTC 63c50b89.305f 0 FATAL: could not connect to the primary server: could not connect to server: No route to host
Is the server running on host "192.168.11.5" and accepting
TCP/IP connections on port 5432?
2023-01-16 08:32:15.626 UTC 63c50b8c.30ab 0 FATAL: could not connect to the primary server: could not connect to server: No route to host
Is the server running on host "192.168.11.5" and accepting
TCP/IP connections on port 5432?
2023-01-16 08:32:18.701 UTC 63c50b91.30c3 0 FATAL: could not connect to the primary server: could not connect to server: No route to host
Is the server running on host "192.168.11.5" and accepting
TCP/IP connections on port 5432?
2023-01-16 08:32:19.754 UTC 63c50b93.310d 0 VCDB vc ERROR: cannot execute CREATE TABLE in a read-only transaction
2023-01-16 08:32:19.754 UTC 63c50b93.310d 0 VCDB vc STATEMENT: CREATE TABLE IF NOT EXISTS lookupsvc_subscriptions ( subscription_id text PRIMARY KEY,data jsonb)
2023-01-16 08:32:25.609 UTC 63c50b96.314d 0 FATAL: could not connect to the primary server: could not connect to server: No route to host

In order to fix this issue please follow the below steps:

Action Plan.

Please ensure there is a valid backup or snapshot before proceeding with any of the steps.

  • Log into appliance as root.
  • Manually promote the database with the command:
sudo -u vpostgres /opt/vmware/vpostgres/current/bin/pg_ctl promote -D /storage/db/vpostgres
  • Restart all the services.
service-control --stop --all
service-control --start --all
  • If that fails to resolve the issue, try moving the standby files:
    cd /storage/db/vpostgres
    mv standby.conf /tmp
    mv standby.signal /tmp
    Restart all the services.
    service-control --stop --all
    service-control --start --all

This should bring your vCenter Server online with all the services healthy.

Ashutosh Dixit

I am currently working as a Senior Technical Support Engineer with VMware Premier Services for Telco. Before this, I worked as a Technical Lead with Microsoft Enterprise Platform Support for Production and Premier Support. I am an expert in High-Availability, Deployments, and VMware Core technology along with Tanzu and Horizon.

Leave a Reply