Symtoms:

  • The SDDC Manager UI does may not display any domain information
  • UI displays “Failed to Poll” and “Failed to Fetch” messages 

Failed to poll domain info. Could not reach the server., possible reasons could be network connectivity issue or server going down.
Failed too fetch allocation. Could not reach the server., possible reasons could be network connectivity issue or server going down

  • In the Common Services log, located at /var/log/vmware/vcf/commonsvcs, there are references to 502 Bad Gateway:

2023-08-11 11:41:21.132 ERROR [pool-2-thread-5] [com.vmware.evo.sddc.taskaggregator.services.TaskCollector] collect() failedorg.springframework.web.client.HttpServerErrorException: 502 Bad Gateway

Snippets from logs:

2023-08-11T00:00:09.007+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2023-08-11T00:00:19.017+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2023-08-11T00:00:29.019+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2023-08-11T00:00:39.021+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2023-08-11T00:00:49.022+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2023-08-11T00:00:59.023+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2023-08-11T00:01:09.025+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2023-08-11T00:01:19.026+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2023-08-11T00:01:29.029+0000 INFO [0000000000000000,0000] [liquibase,main] Waiting for changelog lock....
2023-08-11T00:01:39.033+0000 WARN [0000000000000000,0000] [o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext,main] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: Invocation of init method failed; nested exception is liquibase.exception.LockException: Could not acquire change log lock. Currently locked by 10.x.x.x (10.x.x.x) since 7/22/19 8:02 PM
2023-08-11T00:01:39.036+0000 INFO [0000000000000000,0000] [o.a.catalina.core.StandardService,main] Stopping service [Tomcat]

Cause:

This issues occurs when a service acquires a database change log lock during initialization.

Resolution/Workaround:

  1. Take a backup/snapshot of the SDDC Manager VM before proceeding
  2. Open an SSH to the SDDC Manager VM, change to root:

su –

  1. Run the following command to drop the databasechangeloglock table for the failing service:

For VMware Cloud Foundation 3.8.x and onward:

psql -h localhost -U postgres -d {service-name} -c ‘drop table databasechangeloglock;’

  1. Reboot the SDDC Manager VM to ensure all services come up successfully