Deleting Stubborn Interconnect Configuration in HCX

I had a working HCX setup in my lab and I was doing some modification in my setup and tried chopping off my interconnect networking configuration in HCX Cloud side. Deletion of interconnect configuration was failing for me with below error

hcx-pool-delete error.JPG

Let me first explain how I landed in this situation. 

I was deleting the interconnect appliances from my on-prem to show demo to my peers on how the interconnects are deployed via HCX plugin in vSphere webclient. During the demo I did not noticed that site pairing between my on-prem HCX and cloud side HCX was broken (due to vCenter upgrade in cloud side, cert mismatch issue occurred).

When I kicked the CGW appliance removal, the backing VM got deleted and the appliance configuration disappeared from on-prem. But when I checked in cloud side, the peer CGW appliance and the Mobility Agent host was still intact.

That is when I discovered that communication between the two sites are broken. Since cloud side CGW appliance is automatically deleted when deletion is called from on-prem, I couldn’t find any option in HCX Cloud UI to delete the same.

Frustrated of this, I deleted the mobility agent host and CGW manually from cloud side vCenter. But the interconnect configuration was still intact in HCX Cloud UI. Since this is a lab setup, I could not open SR with VMware, and hence I decided to do some R&D on this and finally I was able to fix the issue.

Disclaimer: The below resolution is not supported by VMware and its totally an outcome of R&D in lab setup. Don’t perform these steps ever in your production without engaging VMware support.

 HCX configuration data is saved in MongoDB and since UI was not helping me, I jumped into DB and examined few tables. 

1: Connect to HCX DB: Connect to HCX appliance over putty and run command: mongo hybridity

HCX DB connect.JPG

2: List tables: To list the tables in database, run command: show collections

This command give you a long list of tables. The important tables related to interconnect configuration are:

  • FleetAppliance
  • FleetConfigIP
  • FleetConfig
  • VirtualNetwork

I found details of the offending IP in above tables. To examine contents of a table in mongo, you need to run command: db.<table-name>.find().pretty()

Examples are as shown below

Once I verified details in above table, I dropped them using below commands

> db.FleetConfig.drop ()

> db.FleetConfigIP.drop ()

> db.FleetAppliance.drop()

> db.VirtualNetwork.drop()

Post dropping these tables, I refreshed the HCX Cloud UI and the stubborn interconnect configuration disappeared from there.

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing:)