Today while working on one of the production issue, I came across situation where I deployed a new vDC from vCloud Air portal, but after the vDC was created successfully, I was not able to list the edge gateways/org networks for this vDC.
On checking vCloud Director, I found that edge gateway creation failed and was showing below errors:
[ 7267fcbe-194f-47c6-bee1-029a2e445e48 ] Deployment of edge gateway gateway failed. com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (1256006): Operation is not allowed by the applied NSX license. - com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (1256006): Operation is not allowed by the applied NSX license. - VSM response error (1256006): Operation is not allowed by the applied NSX license.
I logged in Web Client and navigated to Administration > Licenses view and was bot able to see any licenses there. This was a bit strange. I understood right at that moment that there is some issue with licensing service.
In our environment we are using vCenter with external PSC, so I logged onto PSC node and found that /storage/log partition was at 100%. In such situation services will start to behave abnormally and can even go to stopped state as they can’t write anything in log files.
Note: I have changed the name of server for security reasons.
psc-fqdn:~ # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda3 11G 2.6G 7.6G 26% / udev 4.0G 164K 4.0G 1% /dev tmpfs 4.0G 32K 4.0G 1% /dev/shm /dev/sda1 128M 38M 84M 31% /boot /dev/mapper/core_vg-core 5.0G 139M 4.6G 3% /storage/core /dev/mapper/log_vg-log 5.0G 5.0G 0 100% /storage/log
To fix the issue for that moment I archived and then zeroed out some old log files which reduced the utilization of /storage/log partition and then bounced the services.
On further discussion with my peers, I came across VMware KB-2143565 which explained everything. Actually its a known problem in vSphere 6.0 ad we need to decrease the maximum backup size and the maximum backup index in the log4j.properties file.
Here are the steps:
1: Perform snapshot of PSC node.
2: Login to PSC node via root user.
3: Naviagte to directory /usr/lib/vmware-sso/vmware-sts/webapps/sts/WEB-INF/classes
5: Backup the current configuration file
# cp log4j.properties log4j.properties.bak
4: Edit the log4j.properties and search for below entries
Change log4j.appender.LOGFILE.MaxFileSize to 10MB (or greater) and log4j.appender.LOGFILE.MaxBackupIndex to 5
5: Restart STS service:
# service vmware-stsd restart
Alternatively you can increase the size of /storage/log partition to a greater value.
Note: This issue is resolved in vCenter Server 6.0 update 3
Once the above issue was fixed, I was able to see the licenses in vCenter Web Client. I spun a new vDC from VCA portal and this time things went pretty smooth.
I hope you find this post informational. Feel free to share this on social media if it is worth sharing. Be sociable 🙂