Troubleshooting Edge Gateway Deployment Failure in vCloud Air

Today while working on one of the production issue, I came across situation where I deployed a new vDC from vCloud Air portal, but after the vDC was created successfully, I was not able to list the edge gateways/org networks for this vDC.

gwt-1

On checking vCloud Director, I found that edge gateway creation failed and was showing below errors:

[ 7267fcbe-194f-47c6-bee1-029a2e445e48 ] Deployment of edge gateway gateway failed.
com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (1256006): Operation is not allowed by the applied NSX license.
- com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (1256006): Operation is not allowed by the applied NSX license.
 
- VSM response error (1256006): Operation is not allowed by the applied NSX license.

I logged in Web Client and navigated to Administration > Licenses view and was bot able to see any licenses there. This was a bit strange. I understood right at that moment that there is some issue with licensing service.

In our environment we are using vCenter with external PSC, so I logged onto PSC node and found that /storage/log partition was at 100%. In such situation services will start to behave abnormally and can even go to stopped state as they can’t write anything in log files.

Note: I have changed the name of server for security reasons.

psc-fqdn:~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 11G 2.6G 7.6G 26% /
udev 4.0G 164K 4.0G 1% /dev
tmpfs 4.0G 32K 4.0G 1% /dev/shm
/dev/sda1 128M 38M 84M 31% /boot
/dev/mapper/core_vg-core 5.0G 139M 4.6G 3% /storage/core
/dev/mapper/log_vg-log 5.0G 5.0G 0 100% /storage/log

To fix the issue for that moment I archived and then zeroed out some old log files which reduced the utilization of /storage/log partition and then bounced the services.

On further discussion with my peers, I came across VMware KB-2143565 which explained everything. Actually its a known problem in vSphere 6.0 ad we need to decrease the maximum backup size and the maximum backup index in the log4j.properties file.

Here are the steps:

1: Perform snapshot of PSC node.

2: Login to PSC node via root user.

3: Naviagte to directory /usr/lib/vmware-sso/vmware-sts/webapps/sts/WEB-INF/classes

5: Backup the current configuration file

# cp log4j.properties log4j.properties.bak

4: Edit the log4j.properties and search for below entries

log4j.appender.LOGFILE.MaxFileSize=100MB
log4j.appender.LOGFILE.MaxBackupIndex=10

Change log4j.appender.LOGFILE.MaxFileSize to 10MB (or greater) and log4j.appender.LOGFILE.MaxBackupIndex to 5

5: Restart STS service:

# service vmware-stsd restart

Alternatively you can increase the size of /storage/log partition to a greater value.

Note: This issue is resolved in vCenter Server 6.0 update 3

Once the above issue was fixed, I was able to see the licenses in vCenter Web Client. I spun a new vDC from VCA portal and this time things went pretty smooth.

gwt-2.PNG

I hope you find this post informational. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Add a Comment