Delete Stale Org Networks and Edge Gateway from vCloud Director

Today while working in production, I came across an issue where the edge VM’s backing the edge gateway were not present in vCenter (no idea how they got deleted).

Due to this I was not able to delete the Org network from vCD. Any attempt to delete the Org network was failing with error

[ f3fcd1fd-cf1c-4a57-a920-504204dceba7 ] Cannot delete organization VDC network DMZ (0dff6592-df7c-4e02-a106-5f0ed722d601)
 Cannot update edge gateway "urn:uuid:f41b9f5a-0389-4781-ae0d-4b9cd19e9756"

java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive.

- Cannot update edge gateway "urn:uuid:f41b9f5a-0389-4781-ae0d-4b9cd19e9756"
 java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive.

This is what I was seeing in vCloud Director

1.PNG

On further investigation I found that the virtual wires associated with the problematic network were present in vCenter but on selecting the virtual machine tab for the virtualwire (distributed portgroup), I was not seeing any VM connected. Typically an edge VM should appear in the list.

That is when I found that edge VM’s were gone from vCenter.

Tried to re-deploy edge gateway and it also failed. Got same error about edge VM’s not being present

[ 8580da07-1ce4-440f-b126-0e5c8cca6e3e ] Cannot redeploy edge gateway XYZ (urn:uuid:f41b9f5a-0389-4781-ae0d-4b9cd19e9756)
 com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive.

- com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive.

- VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive.

In this situation there is nothing much you can do. You need to hack your VCD DB and remove the stale entries from the database which would cause those entries to disappear from vCD UI.

First of all delete all virtual wires corresponding  to the stale Org network from vCenter.

Secondly you need to analyze the database to look into the tables where info about those Org networks is stored.

Update: 13-07-2017: Before performing below steps, please check if the issue is similar to this. DB hack should be last weapon for any administrator dealing with this issue

Typically you have to  use following queries:

1: Find Org vDC ID where your stale networks exists

select id ,vdc_id ,name FROM vdc_logical_resource where lr_type= ‘NETWORK’ AND name = ‘Org_nw_name’;

Output (Assuming your org network name is ABC)

id           vdc_id      name
0xAAAAAA     0xBBBBBB    ABC

2: List all Org Networks in your Org VDC (In case you have more than one stale network)

Select id, name from vdc_logical_resource where vdc_id = 0xBBBBB AND lr_type = ‘NETWORK’;

In this example I am assuming that you have 3 stale Org network namely ABC,DEF and XYZ

 id                name
 0xAAAAAAA         ABC
 0xNNNNNNN         DEF
 0xTTTTTTT         XYZ

Note: In case you have several org networks associated with your edge gateway but there is only one problematic (stale) org network, then you can modify the above query as shown below.

Select id, name from vdc_logical_resource where vdc_id = 0xBBBBB  lr_type = ‘NETWORK’ AND name = ‘Org_nw_name’;

This query will specifically return record of the stale network and thus will prevent accidental deletion of any healthy org network

3: Find Logical network id and gateway id

select display_name, id, gateway_id, logical_network_id from gateway_interface where display_name = ‘ABC’;

display_name    id        gateway_id logical_network_id
 ABC          0xF41B9F      0x8EEB1C   0x9FCD1Y

4: Query for link_lnet_id

select id,link_lnet_id FROM logical_network WHERE name = ‘ABC’

id                                 link_lnet_id
0x8EEB1C757916483893A633FF3E459ECE    NULL

Once you have all the information in place, use following delete queries to remove the records of the stale networks from vCD UI

delete FROM vdc_logical_resource WHERE id = 0xAAAAAAA;

delete FROM gateway_assigned_ip WHERE gateway_interface_id = 0xF41B9F;

delete FROM gateway_interface WHERE logical_network_id = 0x9FCD1Y;

delete FROM logical_network WHERE name = ‘ABC’;

Once you have deleted all stale networks and intend to delete edge gateway as well, you can do so by firing the below query

delete FROM gateway WHERE id = Edge_gw_id;

Example:

delete FROM gateway WHERE id = 0xF41B9F5A03894781AE0D4B9CD19E9756;