Troubleshooting Edge Gateway Deployment Failure in vCloud Air

By | 23/08/2017

Today while working on one of the production issue, I came across situation where I deployed a new vDC from vCloud Air portal, but after the vDC was created successfully, I was not able to list the edge gateways/org networks for this vDC.

gwt-1

On checking vCloud Director, I found that edge gateway creation failed and was showing below errors:

 I logged in Web Client and navigated to Administration > Licenses view and was bot able to see any licenses there. This was a bit strange. I understood right at that moment that there is some issue with licensing service.

In our environment we are using vCenter with external PSC, so I logged onto PSC node and found that /storage/log partition was at 100%. In such situation services will start to behave abnormally and can even go to stopped state as they can’t write anything in log files.

Note: I have changed the name of server for security reasons.

To fix the issue for that moment I archived and then zeroed out some old log files which reduced the utilization of /storage/log partition and then bounced the services.

On further discussion with my peers, I came across VMware KB-2143565 which explained everything. Actually its a known problem in vSphere 6.0 ad we need to decrease the maximum backup size and the maximum backup index in the log4j.properties file.

Here are the steps:

1: Perform snapshot of PSC node.

2: Login to PSC node via root user.

3: Naviagte to directory /usr/lib/vmware-sso/vmware-sts/webapps/sts/WEB-INF/classes

5: Backup the current configuration file

# cp log4j.properties log4j.properties.bak

4: Edit the log4j.properties and search for below entries

log4j.appender.LOGFILE.MaxFileSize=100MB

log4j.appender.LOGFILE.MaxBackupIndex=10

Change log4j.appender.LOGFILE.MaxFileSize to 10MB (or greater) and log4j.appender.LOGFILE.MaxBackupIndex to 5

5: Restart STS service:

# service vmware-stsd restart

Alternatively you can increase the size of /storage/log partition to a greater value.

Note: This issue is resolved in vCenter Server 6.0 update 3

Once the above issue was fixed, I was able to see the licenses in vCenter Web Client. I spun a new vDC from VCA portal and this time things went pretty smooth.

gwt-2.PNG

I hope you find this post informational. Feel free to share this on social media if it is worth sharing. Be sociable 🙂