Today while working on one production issue, I came across one incident where I was unable to delete one of the Org Network in vCloud Director. Observed following errors in vCD UI for the Org network deletion failure:
On checking vcloud-container.debug.log I observed similar log entries as seen in vCD UI
This was entirely new error for me so I started googling this around and unfortunately did not found helpful article. The only article which I got for this error was this but of no use for me.
Like last post of NSX series, this post is also focused on exploring a new tool which helps automatic NSX stuffs in your infrastructure. I first came across this tool when I was watching a VMworld 2016 Session titled NET7514 – PowerNSX and PyNSXv, but never got chance to play around this tool.
Now since I am exploring NSX automation these days, I decided to deploy the tool, in lab and use it.
This post will be focused on just installation/configuration part and some examples on how to use this tool.
So what is PyNSXv?
PyNSXv is a high python based library that exposes ready to use work-flows and a CLI tool that can be used to control and automate NSXv in your infrastructure.
It an opensource tool and is not supported by VMware and before using this in production, it is recommended to test it thoroughly in lab deployments.
Currently PyNSXv functionality covers the following key areas:
PyNSXv can be used in two different ways, as a library by importing the files in the /library sub-directory into your code, or as a CLI tool by executing pynsxvon the command line after installation. To install PyNSXv you can use PIP on your system.
You should have some basic knowledge of Python and Powershell (and of-course of NSX) to begin with this tool. Below diagram just shows a very high level architectural view of this tool.
PyNSXv can be downloaded from VMware’s GitHub directory. Instructions for installation are available on PyNSXv Wiki page. Lets the fun begin.
Installing PyNSXv on Windows
Step 1: Install python v2.7 on your windows box. Python v2.7 can be downloaded from Here. Attention: Do Not install python v 3.X
Step 2: Add python installation folder in the Windows path (So that PyNSXv can be run from any folder).
To do so navigate to Control Panel > System and Security > System > Advance System Settings > Advanced > Environment Variables.
Under System Variables edit PATH variable and add python installation\script folder path (typically it is C:\Python27\Scripts)
Step 3: Install PynSXv
Run command : pip install pysxv
Step 4: After you installed PyNSXv, the first thing you have to do is to create your a ini file that contains the host names and credentials of your vCenter and NSX Manager. Save this file in location : C:\Python27\Lib\site-packages\pynsxv
Typically nsx.ini file looks like below:
# variables for PyNSXv [nsxv] nsx_manager = 192.168.109.6 nsx_username = admin nsx_password = NSX-PWD [vcenter] vcenter = 192.168.109.20 vcenter_user = administrator@vsphere.local vcenter_passwd = VC-PWD [defaults] transport_zone = Cloud-pVDC-VXLAN-NP datacenter_name = Cloud-DC edge_datastore = iSCSI-2 edge_cluster = Resource-Cluster
After placing the nsx.ini file in you path, you can run pynsxv from your shell or cmd prompt.
Now you can use pynsx,exe command to play around with various options. One example is shown below
To see list of all available options that can be used with pynsxv command, please read this document.
Alternatively PyNSXv can be installed on windows using git command as well. Make sure git is installed on your windows machine.
You can clone the PyNSXv repository by using command: git clone https://github.com/vmware/pynsxv.git
This will typically place the installation file under C:\Users\Username\Documents\GitHub. From there navigate to pynsxv directory and run setup.py command. It will configure the necessary modules/library.
The nsx.ini file is placed in directory C:\Users\Username\Documents\GitHub\pynsxv\pynsxv. Modify this file as per example shown earlier.
Linux Installation
Step 1: If you are using Centos/Redhat 6 then default python version that is shipped with the v6 distribution is 2.6. We have to install python 2.7 on CentOS/Redhat. Instructions for doing so are explained here.
This post also demonstrates how to install pip. Once Python2.7 and Pip2.7 is installed, PyNSXv can be installed using command: pip2.7 install pynsxv
This command places the modules/library files under directory /usr/local/lib/python2.7/site-packages/pynsxv. In the same directory you will find the nsx.ini file.
Alternatively on linux boxes PyNSXv can be installed via git using command git clone https://github.com/vmware/pynsxv.git.
This will create a directory pynsx in your present working directory. Again you can find nsx.ini file here and modify it as per your environment details.
Now its time to play around various options available.
[root@linjump ~]# pynsxv -i /root/pynsxv/pynsxv/nsx.ini lswitch list +-----------------------------------------------------------+---------------+ | LS name | LS ID | |-----------------------------------------------------------+---------------| | dvs.VCDVSProd-Routed-c48ebd51-e791-4d06-8bf4-0f2c04ee3eff | virtualwire-1 | +-----------------------------------------------------------+---------------+
These days I am busy exploring NSX Rest API in my lab and during the process I came to know about a cool tool named PowerNSX and decided to dedicate a blog on this to give respect to creator of this tool.
What is PowerNSX
PowerNSX is a PowerShell module that abstracts the NSX API to a set of easily used PowerShell functions. PowerNSX enables NSX administrators to drive their infrastructure programmatically.
PowerNSX add additional functionality to extend the capabilities of NSX along with exposing the existing Update, Remove and Get operations for all key NSX functions beyond the native UI or API.
In this post we will explore how NSX manager system info can be retrieved via Rest API. NSX manager appliance home page is itself very descriptive and provides all system info.
In this post we will learn how the same system info can be explored via API calls. Let’s get started.
Query NSX Manager Information
Below API query will provide you info like what is the major and minor version of NSX appliance you ae running along with patch number and build number
# curl -k -u “admin:Password” -X GET https://nsxmgr.alex.local/api/1.0/appliance-management/global/info xmllint –format –
<?xml version="1.0" encoding="UTF-8"?> <globalInfo> <currentLoggedInUser>admin</currentLoggedInUser> <versionInfo> <majorVersion>6</majorVersion> <minorVersion>3</minorVersion> <patchVersion>2</patchVersion> <buildNumber>5672532</buildNumber> </versionInfo> </globalInfo>
Query NSX Manager Summary Information
This API query will present you with all info which you used to see from the NSX manager homepage. This call can be used to obtain all system related info in one shot.
# curl -k -u “admin:Password” -X GET https://nsxmgr.alex.local/api/1.0/appliance-management/summary/system | xmllint –format –
<?xml version="1.0" encoding="UTF-8"?> <systemSummary> <ipv4Address>192.168.109.6</ipv4Address> <dnsName>nsxmgr.alex.local</dnsName> <hostName>nsxmgr</hostName> <domainName>alex.local</domainName> <applianceName>vShield Virtual Appliance Management</applianceName> <versionInfo> <majorVersion>6</majorVersion> <minorVersion>3</minorVersion> <patchVersion>2</patchVersion> <buildNumber>5672532</buildNumber> </versionInfo> <uptime>14 days, 23 hours, 56 minutes</uptime> <cpuInfoDto> <totalNoOfCPUs>4</totalNoOfCPUs> <capacity>2599 MHZ</capacity> <usedCapacity>187 MHZ</usedCapacity> <freeCapacity>2412 MHZ</freeCapacity> <usedPercentage>7</usedPercentage> </cpuInfoDto> <memInfoDto> <totalMemory>16025 MB</totalMemory> <usedMemory>5761 MB</usedMemory> <freeMemory>10264 MB</freeMemory> <usedPercentage>36</usedPercentage> </memInfoDto> <storageInfoDto> <totalStorage>81G</totalStorage> <usedStorage>20G</usedStorage> <freeStorage>61G</freeStorage> <usedPercentage>25</usedPercentage> </storageInfoDto> <currentSystemDate>Tuesday, 27 June 2017 04:29:52 PM IST</currentSystemDate> </systemSummary>
In this post we will learn how can we configure some of the network settings like DNS/Syslog and NTP configurations in NSX manager via Rest API.
We can do all this from NSX manager GUI also but if you are thinking about automating NSX manager deployment, then these Rest API knowledge can be pretty handy for configuring the appliance post its deployment.
Lets get started.
Query Network Settings
Below API query will give you an overview of NSX Manager IP settings, Hostname, DNS settings and domain name
# curl -k -u “admin:adminpwd” -X GET https://nsxmgr.alex.local/api/1.0/appliance-management/system/network/ | xmllint –format –
<?xml version="1.0" encoding="UTF-8"?> <network> <hostName>nsxmgr</hostName> <domainName>alex.local</domainName> <networkIPv4AddressDto> <ipv4Address>192.168.109.6</ipv4Address> <ipv4NetMask>255.255.255.0</ipv4NetMask> <ipv4Gateway>192.168.109.1</ipv4Gateway> </networkIPv4AddressDto> <dns> <ipv4Address>192.168.109.2</ipv4Address> <domainList>alex.local</domainList> </dns> </network>
In this post We will learn how to view generate self-signed certificate for NSX and replace the certificates after getting them signed from CA. We will be doing this via Rest API.
I wrote a post in past on how to replace SSL certs for NSX from GUI. In this post I am trying to achieve the same via Rest API
Following are the API queries which you need to execute in order to generate and replace certs.
Generate CSR Certificate
# curl -k -u “admin:Telstra@123” -d @csr.xml -X PUT https://nsxmgr.alex.local/api/1.0/appliance-management/certificatemanager/csr/nsx
<?xml version="1.0" encoding="UTF-8"?> <csr> <algorithm>RSA</algorithm> <keySize>4096</keySize> <subjectDto> <commonName>nsxmgr.alex.local</commonName> <organizationUnit>Cloud</organizationUnit> <organizationName>Alex.Co</organizationName> <localityName>Bangalore</localityName> <stateName>Karnataka</stateName> <countryCode>IN</countryCode> </subjectDto> </csr>
In this post I will be demonstrating how to enable and disable high availability on NSX edge gateway using Rest API.
If you are new to NSX and do not know what edge gateway high availability means then I would recommend to read this Blog by Gabe Rosas.
We can enable disable high availability on edge gateway from vSphere Web Client by navigating to Home > Networking & Security > NSX Edges > Selecting Edge > Manage > HA Configuration
Enabling HA on edge gateway will create a new vse vm in vCenter and both VM start exchanging heartbeat and exchanging other configuration etc.
Now we will see how to achieve this via NSX Rest API.
Step 1: Query HA Status
# curl -k -u “admin:AdminPWD” -X GET https://nsxmgr.alex.local/api/4.0/edges/edge-2/highavailability/config | xmllint –format –
<?xml version="1.0" encoding="UTF-8"?> <highAvailability> <version>3</version> <enabled>false</enabled> <declareDeadTime>15</declareDeadTime> <logging> <enable>false</enable> <logLevel>info</logLevel> </logging> <security> <enabled>false</enabled> </security> </highAvailability>
From the above output we can see HA ha snot been enabled on edge gateway yet.
Step 2: Enable HA
To enable HA on edge gateway, we need to supply few parameter in the request body of the API call. If you are using curl you can create an xml file as shown below and can supply it with API query with -d option.
<?xml version="1.0" encoding="UTF-8"?> <highAvailability> <version>4</version> <enabled>True</enabled> <declareDeadTime>15</declareDeadTime> <logging> <enable>false</enable> <logLevel>info</logLevel> </logging> <security> <enabled>false</enabled> </security> </highAvailability>
In this post I will demonstrate how to redeploy edge gateway in vCloud Director using Rest API
Disclaimer: This is not any fancy post and I am going to perform very simple task here. Most of you may be already aware of this. This post is for those who are new to API and also a reference post for me for future.
Lets get started.
We have to follow below steps for redeploying an edge gateway using API calls
Step 1: Generate Auth Token
# curl -sik -H “Accept:application/*+xml;version=5.6” -u “admin@system” -X POST https://vcd-b.alex.local/api/sessions | grep auth
Enter host password for user ‘admin@system’:
x-vcloud-authorization: 3fc8a5425f804c9d94eeff04e0272ed7
Step 2: Get Org UUID
# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization:3fc8a5425f804c9d94eeff04e0272ed7” -X GET https://vcd-a.alex.local/api/org/
<Org href="https://vcd-a.alex.local/api/org/58d92de4-4aa5-4a14-9b39-28e1de5e9809" name="Production" type="application/vnd.vmware.vcloud.org+xml"/>
Step 3: Get vDC UUID
# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization:3fc8a5425f804c9d94eeff04e0272ed7” -X GET https://vcd-a.alex.local/api/org/58d92de4-4aa5-4a14-9b39-28e1de5e9809 | grep vdc
<Link rel="down" href="https://vcd-a.alex.local/api/vdc/1c32832c-e4db-48b0-9cfc-116fbb302625" name="Prod-DC" type="application/vnd.vmware.vcloud.vdc+xml"/>
Step 4: Get Edge Gateway UUID
# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization:3fc8a5425f804c9d94eeff04e0272ed7” -X GET https://vcd-a.alex.local/api/vdc/1c32832c-e4db-48b0-9cfc-116fbb302625 | grep edgeGateways
<Link rel="edgeGateways" href="https://vcd-a.alex.local/api/admin/vdc/1c32832c-e4db-48b0-9cfc-116fbb302625/edgeGateways" type="application/vnd.vmware.vcloud.query.records+xml"/>
Step 5: Get Edge Gateway UUID
# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization:3fc8a5425f804c9d94eeff04e0272ed7” -X GET https://vcd-a.alex.local/api/admin/vdc/1c32832c-e4db-48b0-9cfc-116fbb302625/edgeGateways | grep GW_Name
<EdgeGatewayRecord gatewayStatus="READY" haStatus="DISABLED" isBusy="false" name="Prod-GW" numberOfExtNetworks="1" numberOfOrgNetworks="1" vdc="https://vcd-a.alex.local/api/vdc/1c32832c-e4db-48b0-9cfc-116fbb302625" href="https://vcd-a.alex.local/api/admin/edgeGateway/b37b059b-be98-4806-8535-9bbdcd4b6575" isSyslogServerSettingInSync="true"/>
Yesterday while working on one of the production issue where we had to deprovision a tenant environment in vCloud Air, I noticed that independent disks were preventing automated deprovision of the environment and the error messages were loud and clear in the log files.
It was a new issue for me so I started reading about independent disks in vCloud Director and want to share few things about this.
First of all independent Disk feature in vCD is completely different from an Independent Disk in vSphere. Independent disks can be shared across multiple vApps/VM’s in vCloud Director. This feature was first introduced in vCD v5.1.
Following quote from vCloud Architecture Toolkit document rightly explains about independent disks
The use of independent disks with vCloud Director allows updates of virtual machines without impacting the underlying data.
The feature is designed to enable users to create virtual disks which can be attached to and detached from virtual machines. There is no functionality to control this feature from the vCD UI and this can be controlled via API’s only.
When you create an independent disk, it is associated with an organization vDC but not with a virtual machine. After the disk has been created, the disk owner or an administrator can attach it to any virtual machine deployed in that vDC, detach it from a virtual machine, and remove it from the vDC.
Presence of Independent disks in vCD can be seen on navigating to Org > Administration > Org vDC > Independent Disks tab. If you right click on any of the disk you will not see any action window opening.
In this post I am going to demonstrate how we can detach/delete independent disks from VM via API calls. Lets get started.
For sake of this demonstration, I have used some hypothetical names for Org and Org vDC.
Step 1: Obtain vCD Auth token code
# curl -sik -H “Accept:application/*+xml;version=5.6” -u “admin@system” -X POST https://vCD-FQDN/api/sessions | grep auth
Enter host password for user ‘admin@system’:
x-vcloud-authorization: Auth
Step 2: Locate your Org
# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization:Auth” -X GET https://vCD-FQDN/api/org/
On using the above API call, you will see a href to all your Org that are present in vCD. For your next query chose the href of the org where independent disks are lying.
<Org href="https://vCD-FQDN:443/api/org/08356307-2939-42d3-a2a2-aeccef6478e4" name="ABC" type="application/vnd.vmware.vcloud.org+xml"/> <Org href="https://vCD-FQDN:443/api/org/2b729e6f-588e-49c4-964f-89b2e744c075" name="DEF" type="application/vnd.vmware.vcloud.org+xml"/> <Org href="https://vCD-FQDN:443/api/org/fc432145-f1f3-42f6-a26f-eeb3d306a405" name="GHI" type="application/vnd.vmware.vcloud.org+xml"/>
Step 3: Locate your Org vDC
# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: Auth” -X GET https://vCD-FQDN:443/api/org/fc432145-f1f3-42f6-a26f-eeb3d306a405 | grep vdc
<Link rel="down" href="https://vCD-FQDN:443/api/vdc/adf0929b-a107-4671-9f85-b629b744c2b7" name="VDC1" type="application/vnd.vmware.vcloud.vdc+xml"/>
After a long wait I finally got chance to work on vCloud Director ssl certificates. This was the only component in my lab which was still using self-signed certs and that encouraged me to do something new in lab.
A note on vCD SSL certificates
vCloud Director like any other VMware product needs a certificate to be installed on the device that it uses for communication with the other products. By default vCD uses a self-signed certificate. If you have a certificate authority in your environment then you can get the certs created in advance before installing vCloud director and save your self from pain of messing with certificates at later stages.
Today while working in my lab, I observed that while creating a new VDC in vCD was failing because org network failed to create.
On navigating to Org VDC list and clicking on error, it read the error load and clear that org vdc network can’t be created.
On navigating to Org VDC Networks section and clicking on error, I was able to identify what has caused the network creation failure.
The error stack was reading as below:
[ 114db22d-fc14-4c87-9030-36d2316aff8b ] Cannot deploy organization VDC network (f1514426-647e-4a03-a5a9-fafa4d73bb58)
com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Cannot create network “dvs.VCDVSRouted-NW-9ab02973-9ded-4c4b-8826-4a52bdf2d6cf” from VXLAN network pool “urn:uuid:5c9de104-0f40-4cec-898f-985ee1fce1d6”. Make sure vShield Manager infrastructure is properly configured and there are segment IDs available.
Few days back while working on one of the customer ticket, I came across an incident where customer was reporting that he is not able to see one of his template in his catalog. He sent the VM via ODT to vCloud Air to be imported in his catalog.
I verified that VM was missing from catalog but was present in vCenter. I tried to import the VM again in Catalog by selecting the option “Import from vSphere” but to my surprise the VM was showing up in the list for selection.
I have never seen such behaviour in VCD. Even if import of template is failed midway, the item is listed in catalog with a question mark against it and status of item reads as “failed to create“
In last post of this series we learnt how to install/configure RMQ for vCloud Director. This post is extension of my last post where I will be adding one more node to my RMQ setup to form a cluster for high availability.
What data is replicated in a RMQ Cluster?
All data/state required for the operation of a RabbitMQ broker is replicated across all nodes. An exception to this are message queues, which by default reside on one node, though they are visible and reachable from all nodes. To replicate queues across nodes in a cluster, see the documentation on high availability
In this post we will learn how to install RMQ for vCloud Director. Before jumping into any lab activity, lets learn first what is RabbitMQ nd why we need it.
What is RabbitMQ?
RabbitMQ is an open source message-queuing software which helps in facilitating message exchange between 2 or more applications. The exchange of messages is done via queue which is defined by administrator. An application can publish a message to the queue which can be retrieve or consumed by a different application.
In last post of this series,we learnt about Recovery options available with vCloud Air Disaster Recovery and we discussed about Test recovery and actual recovery and saw the use case for both type.
In this post we will discuss about failback of a VM (which is recovered in vCloud Air) to on-premise. If you have landed straight to this post accidentally then I will urge you to check out previous blog posts which I have wrote on vCloud Air Disaster Recovery.
Disaster Recovery to vCloud Air is now a tested and reliable solution and helps customer to replicate their critical workloads to vCloud Air so that in event of Disaster in on-premise, customer can recover their workloads and continue their operation without much impact to business.
In this post we will learn how to recover VM’s in vCloud Air in event of disaster or in case when scheduled maintenace is planned in on-premise. For those who are not much familiar with what is vCloud Air Disaster Recovery solution, I urge you to read about that from below links:
After a long back I dived into my lab yesterday and was working again on setting up DR to vCloud Air.I have wrote few blog posts in past on vSphere Replication and DR to vCloud Air using VR. If you are interested in reading those posts, you can read them from below links
1: Replicating and Recovering VM’s using vSphere Replication
2: Disaster Recovery with vcloud Air
3: Replicate VM to vCloud Air using vSphere Replication
This time in my lab I noticed that all my VM’s replication state were going to ‘Not Active’ as soon as its configured for replication. This was a brand new setup and I was sure that I followed all required steps in deploying and configuring the replication appliance. This was strange as I never faced this issue before.
In last post of this series we learnt about how to add nodes in a cassandra cluster. If you have missed earlier posts of this series
then you can read them from below links:
2: Understanding Cassandra Read/Write Mechanism
3: Installing Cassandra on RHEL6
4: Adding Node To Cassandra Cluster
In this post we will explore nodetool and will see how we can manage a cassandra cluster using nodetool utility.
What is nodetool?
The nodetool utility is a command line interface for managing a cluster. It provides a simple command line interface to expose operations and attributes available with cassandra.
In last post of this series we learnt how to install cassandra on Rhel 6. In this post we will look into additional configuration parameter that is needed to configure in order to facilitate other nodes to join the cassandra cluster.
If you have missed earlier posts of this series then you can read them from below links:
2: Understanding Cassandra Read/Write Mechanism
3: Installing Cassandra on RHEL6
At the moment we have one node cassandra cluster. Before going ahead and installing cassandra on other nodes, we will first perform following configuration changes in cassandra.yaml file.
In last 2 posts of this series we learnt about Cassandra architecture and understood the cassandra Read/Write process.
If you have missed earlier posts of this series then you can read them from below links:
2: Understanding Cassandra Read/Write Mechanism
In this post we will learn about installing cassandra on Redhat Linux.
Before jumping into lab and start with installation, I want to touch down on few concepts first which will help in understanding installation process.
In first post of this series we discussed about what is cassandra, what are the benefits of using cassandra. We also discussed a little bit about from where cassandra came and finally we looked at the architecture of cassandra and discussed some important terms like snitch, gossip, data replication, partitioner etc.
In this post we will see how cassandra Read/Write mechanism works.
Let’s discuss about some key components of cassandra first before discussing about Read/Write. Important components of cassandra can be summarized as below:
Cassandra was very new to me when I joined the vCloud Air operations team back in 2015. Over last 1.5 years I have got a bit of understanding about cassandra now and it provoked me to learn this wonderful database technology.
Last month while working on a customer ticket, I came across a request from customer where he wanted to know snapshot creation date for one of his VM as he can not find this detail from vCD UI.
To confirm this, I logged into vCD and navigated to my test lab to see what are the information available.
On navigating through vCD I found that vCD only tells that whether or not snapshot exists for a vApp/VM.
You can see in below screenshot in top right corner that there is no option for selection snapshot creation date.
In last 2 post of this series, we learn about digging out info about datacenter,cluster and virtual machines.
In this post we will learn about API options available for Esxi hosts.
If you have missed earlier posts of this series, you can read them from here:
1: Exploring vSphere 6.5 API-Datacenter & Cluster
2: Exploring vSphere 6.5 API-Virtual Machines
Let’s get started with fetching info about esxi hosts.
1: List all hosts present in a vCenter
Following query will list all esxi hosts that are present across all cluster/datacenter which are there in a vcenter.
If you are using vCloud Director in your environment and if you have ever tried doing a Storage vMotion of a VM from vSphere directly, you will notice a warning saying that its not recommended to modify the entity since its managed by VCD.