Cannot Redeploy Edge Gateway “VSM response error (10020): Failed to deploy edge appliance vse-XXXX-0. The name ‘vse-XXXX-0’ already exists”

This post is very similar to issue described in my last post. The only difference in last issue and this was I was not able to redeploy edge gateway to get rid of stubborn Org Networks whereas in previous case Edge redeploy fixed the issue quite comfortably.

Let me start with a little bit background of how was this issue discovered and what challenges I faced.  I was working investigating a failed deprovision issue when this issue was discovered. Deprovision tasks in our environment are fully automated and we have some portal where these tasks arrives and there is a Resume button which when clicked, kicks the deprovision process.

When the Resume button is clicked that portal initiates API calls to vCD and start deleting stuffs. It starts with deleting vApps, vApp Templates and then proceed to Org Network deletion and then the edge gateway and at last deletes the Org vDC and Org.

Sometimes stuffs at vCD level are in inconsistent state and thus API calls are unable to delete that element and deprovision is halted in portal.

During my investigation I checked the logs and found that API calls were unable to remove one of the Org Network.

Following errors were visible in vCD UI for network deletion failure

[ 695e10af-1677-4c64-bbe1-42250b6c249d ] Cannot delete organization VDC network default-routed (0694f25a-78b9-45b0-be44-e5c8ccda4b91)
Failed to delete interface of edge gateway urn:uuid:5286e85d-afb0-4821-b4f4-db87b390ba11

- Failed to delete interface of edge gateway urn:uuid:5286e85d-afb0-4821-b4f4-db87b390ba11
 
- com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : vm-3768 could not be found. Object identifiers are case sensitive.

From the logs it was very clear that there are issues with edge backing VM’s. I went ahead with performing edge gateway redeploy without checking the edge VM’s status in vCenter. I was thinking that redeploy fixes this issue 9 out of 10 times so just give it a shot.

To my surprise edge gateway redeploy also failed and also I observed that redeploy task took around 20 minutes (usually it takes 5-7 minutes) and eventually timed out. 

Errors related to edge redeploy task failing was

[ e04b76e6-7bb1-4d97-a85c-0df2813a06be ] Cannot redeploy edge gateway M738162563-11503 (urn:uuid:5286e85d-afb0-4821-b4f4-db87b390ba11) com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (10020): Failed to deploy edge appliance vse-xxxxx-0. (The name 'vse-xxxxx-0' already exists.) - com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (10020): Failed to deploy edge appliance vse-xxxxx-0. (The name 'vse-xxxxx-0' already exists.) - VSM response error (10020): Failed to deploy edge appliance vse-xxxxx-0. (The name 'vse-xxxxx-0' already exists.) read more

Edge Gateway Network deletion failed with error “Failed to communicate with NSX Edge vm. Error code VIX_E_PROGRAM_NOT_STARTED was returned by VIX API”

Today while working on one production issue, I came across one incident where I was unable to delete one of the Org Network in vCloud Director. Observed following errors in vCD UI for the Org network deletion failure: 

On checking vcloud-container.debug.log I observed similar log entries as seen in vCD UI

This was entirely new error for me so I started googling this around and unfortunately did not found helpful article. The only article which I got for this error was this but of no use for me.  read more

Detaching and Deleting Independent Disks in vCloud Director via REST API

Yesterday while working on one of the production issue where we had to deprovision a tenant environment in vCloud Air, I noticed that independent disks were preventing automated deprovision of the environment and the error messages were loud and clear in the log files.

It was a new issue for me so I started reading about independent disks in vCloud Director and want to share few things about this.

First of all independent Disk feature in vCD is completely different from an Independent Disk in vSphere. Independent disks can be shared across multiple vApps/VM’s in vCloud Director. This feature was first introduced in vCD v5.1.

Following quote from vCloud Architecture Toolkit document rightly explains about independent disks

The use of independent disks with vCloud Director allows updates of virtual machines without impacting the underlying data.

The feature is designed to enable users to create virtual disks which can be attached to and detached from virtual machines. There is no functionality to control this feature from the vCD UI and this can be controlled via API’s only. 

When you create an independent disk, it is associated with an organization vDC but not with a virtual machine. After the disk has been created, the disk owner or an administrator can attach it to any virtual machine deployed in that vDC, detach it from a virtual machine, and remove it from the vDC.

Presence of Independent disks in vCD can be seen on navigating to Org > Administration > Org vDC > Independent Disks tab. If you right click on any of the disk you will not see any action window opening. 

0.PNG

In this post I am going to demonstrate how we can detach/delete independent disks from VM via API calls. Lets get started.

For sake of this demonstration, I have used some hypothetical names for Org and Org vDC.

Step 1: Obtain vCD Auth token code

# curl -sik -H “Accept:application/*+xml;version=5.6” -u “admin@system” -X POST https://vCD-FQDN/api/sessions | grep auth

Enter host password for user ‘admin@system’:

x-vcloud-authorization: Auth

Step 2: Locate your Org 

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization:Auth-X GET https://vCD-FQDN/api/org/

On using the above API call, you will see a href to all your Org that are present in vCD. For your next query chose the href of the org where independent disks are lying.  

<Org href="https://vCD-FQDN:443/api/org/08356307-2939-42d3-a2a2-aeccef6478e4" name="ABC" type="application/vnd.vmware.vcloud.org+xml"/>

<Org href="https://vCD-FQDN:443/api/org/2b729e6f-588e-49c4-964f-89b2e744c075" name="DEF" type="application/vnd.vmware.vcloud.org+xml"/>

<Org href="https://vCD-FQDN:443/api/org/fc432145-f1f3-42f6-a26f-eeb3d306a405" name="GHI" type="application/vnd.vmware.vcloud.org+xml"/>

Step 3: Locate your Org vDC

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: Auth” -X GET https://vCD-FQDN:443/api/org/fc432145-f1f3-42f6-a26f-eeb3d306a405 | grep vdc

<Link rel="down" href="https://vCD-FQDN:443/api/vdc/adf0929b-a107-4671-9f85-b629b744c2b7" name="VDC1" type="application/vnd.vmware.vcloud.vdc+xml"/> read more

Replacing vCD SSL Certificates in a Multi Cell Environment

After a long wait I finally got chance to work on vCloud Director ssl certificates. This was the only component in my lab which was still using self-signed certs and that encouraged me to do something new in lab.

A note on vCD SSL certificates

vCloud Director like any other VMware product needs a certificate to be installed on the device that it uses for communication with the other products. By default vCD uses a self-signed certificate. If you have a certificate authority in your environment then you can get the certs created in advance before installing vCloud director and save your self from pain of messing with certificates at later stages. read more

Troubleshooting Failed Org Network Creation in vCloud Director

Today while working in my lab, I observed that while creating a new VDC in vCD was failing because org network failed to create.

On navigating to Org VDC list and clicking on error, it read the error load and clear that org vdc network can’t be created.

vcd-1

On navigating to Org VDC Networks section and clicking on error, I was able to identify what has caused the network creation failure.

vcd-2

The error stack was reading as below:

[ 114db22d-fc14-4c87-9030-36d2316aff8b ] Cannot deploy organization VDC network (f1514426-647e-4a03-a5a9-fafa4d73bb58)
com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Cannot create network “dvs.VCDVSRouted-NW-9ab02973-9ded-4c4b-8826-4a52bdf2d6cf” from VXLAN network pool “urn:uuid:5c9de104-0f40-4cec-898f-985ee1fce1d6”. Make sure vShield Manager infrastructure is properly configured and there are segment IDs available. read more

Troubleshooting Mysterious Catalog Item in vCloud Director

Few days back while working on one of the customer ticket, I came across an incident where customer was reporting that he is not able to see one of his template in his catalog. He sent the VM via ODT to vCloud Air to be imported in his catalog.

I verified that VM was missing from catalog but was present in vCenter. I tried to import the VM again in Catalog by selecting the option “Import from vSphere” but to my surprise the VM was showing up in the list for selection.

corpsql01 not listing

I have never seen such behaviour in VCD. Even if import of template is failed midway, the item is listed in catalog with a question mark against it and status of item reads as “failed to create read more

Installing RabbitMQ for vCloud Director

In this post we will learn how to install RMQ for vCloud Director. Before jumping into any lab activity, lets learn first what is RabbitMQ nd why we need it.

What is RabbitMQ?

RabbitMQ is an open source message-queuing software which helps in facilitating message exchange between 2 or more applications. The exchange of messages is done via queue which is defined by administrator. An application can publish a message to the queue which can be retrieve or consumed by a different application. read more

Find Snapshot Creation Date of a vCloud Director VM

Last month while working on a customer ticket, I came across a request from customer where he wanted to know snapshot creation date for one of his VM as he can not find this detail from vCD UI.

To confirm this, I logged into vCD and navigated to my test lab to see what are the information available.

On navigating through vCD I found that vCD only tells that whether or not snapshot exists for a vApp/VM.

You can see in below screenshot in top right corner that there is no option for selection snapshot creation date. read more

Delete Stale Org Networks and Edge Gateway from vCloud Director

Today while working in production, I came across an issue where the edge VM’s backing the edge gateway were not present in vCenter (no idea how they got deleted).

Due to this I was not able to delete the Org network from vCD. Any attempt to delete the Org network was failing with error

[ f3fcd1fd-cf1c-4a57-a920-504204dceba7 ] Cannot delete organization VDC network DMZ (0dff6592-df7c-4e02-a106-5f0ed722d601) Cannot update edge gateway "urn:uuid:f41b9f5a-0389-4781-ae0d-4b9cd19e9756" java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive. - Cannot update edge gateway "urn:uuid:f41b9f5a-0389-4781-ae0d-4b9cd19e9756" java.util.concurrent.ExecutionException: com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : edge-xyz could not be found. Object identifiers are case sensitive. read more

Install vCloud Director 8 with High Availability

vCloud Director 8.0 is the latest version available for service providers and can be downloaded from here.

It’s been quite sometime that I am dealing with vCloud Director in our production environment and as well as my test lab. In past I have written a post on how to install vCloud Director 5.5. You can also read the entire vCloud Director post series from Here

Since v8 is out there in market for sometime, I decided to try my hands on it and implement that in my homelab.

There are various posts available on internet about what is vCloud Director and what it does. So I will not talk much about it and jump directly into action. read more

Unable to start vApp-Invalid vApp properties:Invalid property value size

I recently deployed vSphere replication appliance in vCloud Director and while powering it on I was facing one error

Unable to start vAPP- Invalid vApp properties:Invalid property value size

Due to this power on operation on vApp was failing time and again.

I checked the vCD logs and did not found any error messages for my vApp. All I got was few debug messages:

2016-05-29 09:56:25,597 | INFO | nf-activity-pool-272 | VC20VirtualServer | Created device change list for [name = vrs01 (3fd990d6-6ee2-4618-8446-9067f8ecdd02), valref = [vcId=03dc414f-6385-43e5-b95f-1d7ca8db3aa8, moref=vm-5205]] for synchronizing nics. | requestId=cf0ea3dc-8541-4d5a-b37b-fad3b3cde71b,request=POST https://x.x.x.x/cloud/amf,requestTime=1464515782276,remoteAddress=172.20.1.2:58308,userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML...,accept=*/* method=vAppService.updateVAppVmWithStorageClass vcd=1d114aae-e514-4fc6-a30d-8d5fc7c8c286,task=1fb5f553-e269-4a76-8632-ccf9bb1c5b9d activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:1fb5f553-e269-4a76-8632-ccf9bb1c5b9d) activity=(com.vmware.vcloud.fabric.net.activities.ConstituteNetworkedVmActivity,urn:uuid:56dc4e55-a460-4ac9-9dab-7eb5f76e5248)</em> 2016-05-29 09:56:25,599 | INFO | nf-activity-pool-272 | VC20VirtualServer | Invoking reconfigure vm [name = vrs01 (3fd990d6-6ee2-4618-8446-9067f8ecdd02), valref = [vcId=03dc414f-6385-43e5-b95f-1d7ca8db3aa8, moref=vm-5205], changeVersion 2016-05-29T09:52:33.186794Z] for synchronizing nics. | requestId=cf0ea3dc-8541-4d5a-b37b-fad3b3cde71b,request=POST https://x.x.x.x/cloud/amf,requestTime=1464515782276,remoteAddress=172.20.1.2:58308,userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML...,accept=*/* method=vAppService.updateVAppVmWithStorageClass vcd=1d114aae-e514-4fc6-a30d-8d5fc7c8c286,task=1fb5f553-e269-4a76-8632-ccf9bb1c5b9d activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:1fb5f553-e269-4a76-8632-ccf9bb1c5b9d) activity=(com.vmware.vcloud.fabric.net.activities.ConstituteNetworkedVmActivity,urn:uuid:56dc4e55-a460-4ac9-9dab-7eb5f76e5248) 2016-05-29 09:56:26,617 | DEBUG | val-activity-pool-273 | TaskManager | result null (or not a moref type) for task [moref=task-55625, state=SUCCESS, taskName=ReconfigVM_Task, progress=null, entityName=vrs01 (3fd990d6-6ee2-4618-8446-9067f8ecdd02)] when waiting for inventory update | requestId=cf0ea3dc-8541-4d5a-b37b-fad3b3cde71b,request=POST https://x.x.x.x/cloud/amf,requestTime=1464515782276,remoteAddress=172.20.1.2:58308,userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML...,accept=*/* method=vAppService.updateVAppVmWithStorageClass vcd=1d114aae-e514-4fc6-a30d-8d5fc7c8c286,task=1fb5f553-e269-4a76-8632-ccf9bb1c5b9d activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:1fb5f553-e269-4a76-8632-ccf9bb1c5b9d) activity=(com.vmware.vcloud.fabric.net.activities.ConstituteNetworkedVmActivity,urn:uuid:56dc4e55-a460-4ac9-9dab-7eb5f76e5248) activity=(com.vmware.vcloud.val.internal.impl.ReconfigureVmActivity,urn:uuid:0b07e9fd-fb2c-4926-95d1-22f063ca77db) 2016-05-29 09:56:27,163 | DEBUG | backend-activity-pool-46 | JobString | Job object - Object : vrs01(com.vmware.vcloud.entity.vm:3fd990d6-6ee2-4618-8446-9067f8ecdd02) operation name: VAPP_UPDATE_VM | requestId=cf0ea3dc-8541-4d5a-b37b-fad3b3cde71b,request=POST https://x.x.x.x/cloud/amf,requestTime=1464515782276,remoteAddress=172.20.1.2:58308,userAgent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML...,accept=*/* method=vAppService.updateVAppVmWithStorageClass vcd=1d114aae-e514-4fc6-a30d-8d5fc7c8c286,task=1fb5f553-e269-4a76-8632-ccf9bb1c5b9d activity=(com.vmware.vcloud.backendbase.management.system.TaskActivity,urn:uuid:1fb5f553-e269-4a76-8632-ccf9bb1c5b9d) read more

Building a Private Cloud with vCloud Director-Part 14-Migrating VM from vSphere Infrastructure to private Cloud using vCloud Connector

In our last post  of this series we have learnt what is vCloud connector and what it does.

Now its time to see vCloud Connector in action. In this post we will learn how to migrate a VM from existing vSphere Infrastructure to a private cloud running on vCloud Director.

vCloud Connector has server/node type architecture where there will be one server and one node for each type of cloud. Also we require a node to be deployed for vSphere Infrastructure.

In my last post Installing/Configuring vCloud Connector I have deployed a server and also a node for my vSphere infrastructure. After that we have to deploy a new node for the private cloud (running on vCloud director) read more

Troubleshooting Datastore Cluster Disappearance Issue in vCloud Director

Today I was facing a weird issue in my vCloud Lab. All my datastores have disappeared from vCloud Director and the storage policy was not showing any datastore. I have included a few snapshots from my lab here:

1: Storage Policies was not listing any datastore

ste-1

2: Selecting Datastore and Datastore Cluster was not listing any datastore either

ste-2

3: Due to this my Provider vDC was not able to see any datastores

ste-3

4: And since Provider vDC had no visibility to any datastore, my Organization vDC was throwing this alert read more

Building a Private Cloud with vCloud Director-Part 13-Installing and Configuring vCloud Connector

In this post we will see how to deploy vCloud Connector and use it to facilitate a hybrid cloud architecture. Before going into installation and configuration part first lets see what is vcloud connector

What is vCloud connector?

The VMware vCloud Connector is a tool that facilitates hybrid cloud computing for organizations. The vCloud Connector essentially helps to orchestrate and administer the migration of VM’s across different data centers and clouds.

VMware vCloud Connector links your internal private cloud with public clouds, so you can manage them as a single hybrid environment and transfer workloads back and forth. read more

Importing vApp into vCloud Director catalog from vSphere

In this post I am going to cover how to import a vApp running on vSphere to a catalog in vCloud director.

I have a VM “server2k3” in my vSphere environment which I will be exporting as vApp in my public catalog in vcloud director.

I have selected my organization which have rights to publish catalogs to other organizations in my vCloud director environment. From Catalogs tab I selected option import from vSphere.

From vCenter server list select your vCenter and then select the VM which will be imported. Give a name and description for the imported VM. read more

Building a Private Cloud with vCloud Director-Part 12-Deploying vApps from Catalogs

In our last post Creating and Publishing Catalogs we have seen how to create a public catalog and share it with organizations present in your cloud infrastructure. Also I have populated the public catalog with one server 2003 vApp.

In this post we will see how an organization named “developers” deploy a new vApp from the public catalog that is shared with it. So Lets begin

Login to vcd web interface and go to Manage and Monitor tab and select organizations. In right hand side of the window double click on developers to open it. read more

Building a Private Cloud with vCloud Director-Part 11-Creating and Publishing Catalogs

In our last post of this series we have seen how to  Create and Configure org vDC. Now since Org vDC have been setup, next task is to create some vApps in the organization. But before creating vApps there must be catalogs defined.

What is catalog?

A catalog is a container for vApp templates and media files in an organization. Organization administrators and catalog authors can create catalogs in an organization. Catalog contents can be shared with other users in the organization and can also be published to all organizations in the vCloud Director installation. read more

Setting up LDAP Authentication in vCloud Director

Logging in to your vCloud Director system/organisation via the web interface can be achieved in a number of ways. You can use local authentication (users local to vCloud Director), your Active Directory, or another LDAP v3 compliant directory service for authentication and group membership lookup.

After you connect vCloud Director to an LDAP server, you can import system administrators from the groups and users in the LDAP directory. You can also use the system LDAP settings to import users and groups to an organization, or you can specify separate LDAP settings for each organization. An LDAP user cannot log in to vCloud Director until you import them to the system or an organization. read more

Building a Private Cloud with vCloud Director-Part 10: Creating and Configuring Org vDC

In our last post Creating and configuring Organizations we have created a new organization and went through the configuration settings. In this post we will see how to allocate resources to an organization.

Basically allocating resources means configuring org vDC’s and org vDC will be deriving their resources from Provider vDC (PvDC).

Lets begin with configuring the org vDC in this post.

Login to vCloud Director web interface and navigate to Home Tab and select item no 6 “Allocate resources to an organization read more

Building a Private Cloud with vCloud Director-Part 9: Creating and Configuring Organizations in vCloud Director

In our last post of Configuring Networks in vCloud Director we have created External Network and Network Pools. Network Pools and External Network is consumed by the organizations and org vDC in vCloud Director.

In this post we will learn how to create organization and org vDC and will see how they utilize the network components.

1: Creating Organization

Login to web interface of vCloud Director and navigate to Home tab and click on Item no 5 “Create An Organization”

It will launch the New Organization wizard as shown below. This page lets you create the organization and generates the URL to access your organization. read more

Building a Private Cloud with vCloud Director-Part 8: Configuring Networks in vCloud Director

In our last post Configuring vCloud Director we have seen the initial configuration of vCloud director. Also we have created one Provider VDC and prepared the Esxi hosts for setting up the vCloud infrastructure.

In this post we will learn how to create networking components in vCloud Director. In my post Understanding vCloud Networking i tried to touch down on theoretical concepts of vCloud Networking. Now we will try our hands out on configuring the different network components in this post.

1: Creating External Network read more

Understanding vCloud Networking

One of the most complex concept of VMware vCloud Director is understanding the Networking and at times it can be very confusing.

In this post I will try to shed some light on the concepts of networking within vCD. Chris Wahl has explained vCD networking in his video series “vCloud Director Essentials” in a very beautiful way but still I was not 100% confident about concepts.

In this post I will try to explain the concepts which I learned from Chris video series as well as other sources. Feel free to do more research if any info provided here is not clear to you as I said earlier vCD networking is bit difficult to understand in very first attempt. read more

vCenter User has not been assigned a role on vShield Manager

Problem: Today I faced a new problem in my home lab when I was trying to install the vShield components on my Esxi Hosts. Selecting the Esxi host and navigation to vShield tab was displaying an error

” vCenter User ‘Alex\vcadmin’ has not been assigned a role on vShield Manager”

error-1

Reason: The reason for this error was I had not defined any role for the vCenter user (vcadmin in my case) on vShield manager appliance.

Resolution: To fix this issue login to web interface of vShield Manager, select Settings and Reports and navigate to Users tab and click on Add button to add a new user. read more

Building a Private Cloud with vCloud Director-Part 7: Configuring vCloud Director

In our last post Configuring vShield Manager we have seen the initial configuration of vShield Manager. Now its time to start doing configuration of the vCloud Director. In this post I will touch down on different configurable options available in vCloud Director. So Lets start.

Open console of the vCloud Director by typing the URL https://vcloud_director_ip

It will present the welcome page listing the initial configuration items.

vcd-conf1

Hit Next to proceed. The very first configuration item is accepting the license agreement. read more