Category Archives: vCloud Director

vCloud Director Installation/Configuration/Management/Troubleshooting Posts

Configuring vCenter SSO Federation in vCloud Director 8.20

By | 31/01/2018

There are 3 authentication methods that are supported by vCloud Director:

1: Local: These are the local users which are created at the time of installing vCD or creating any new organization.  If you have configured vCD with default configuration, then the first local account that is created is “administrator” user who is system admin for the vCD.

2: LDAP service: A LDAP service enables the organization to use their own LDAP servers for authentication. Users can then be imported into vCD from the configured LDAP. If you have a multi-tenant based vCD deployment, then each organization can use their own LDAP service for authentication. read more

Configuring vCD 9.0 To Send Metric Data to Cassandra DB

By | 05/10/2017

In last post of this series, we learnt how to install and configure Cassandra DB for collecting metrics data from vCD. We also discussed that kairosdb is no longer needed to be installed alongwith cassandra for this purpose.

In this post we will learn how to configure vCD 9.0 to send metrics data to Cassandra DB.

This configuration is done by using cell management tool utility which is located in /opt/vmware/vcloud-director/bin directory. 

Run cell-management-tool cassandra –help command to see all available options which you need to specify to configure vCD correctly so that it start sending all metrics data to cassandra.

vcd-cass.PNG

Typically this is the command to do so:

[root@vcd90 ~]# /opt/vmware/vcloud-director/bin/cell-management-tool cassandra –configure –create-schema –cluster-nodes 192.168.109.53 –username cassandra –password cassandra –port 9042 –ttl 15

Verifying Cassandra settings... Cassandra setting valid for node: 192.168.109.51 Cassandra configuration settings verified successfully vcloud_metrics keyspace created... vm_metrics table created... adding configured metrics to the schema... adding counter: cpu.usage.average adding counter: cpu.usage.maximum adding counter: cpu.usagemhz.average adding counter: disk.provisioned.latest adding counter: disk.read.average adding counter: disk.used.latest adding counter: disk.write.average adding counter: mem.usage.average Persisting Cassandra settings... Success. The monitoring service is now configured to persist data into cassandra nodes(192.168.109.51), vCD cell(s) must be restarted if they are already running. read more

Installing Cassandra DB for collecting vCD 9.0 Metrics Data

By | 04/10/2017

Cassandra DB is needed for capturing and storing vCloud Director metrics data so that it can be displayed in portal to end users so that users are aware of VM resource utilization etc.

Prior to vCD v9.0, we needed kairosdb + cassandra together for capturing and storing the metrics data, but things have changed now. VMware has removed the requirement of kairosdb and now metrics data can be sent straight to cassandra database.  This metric data in turn can be viewed in tenant UI.

As per vCD 9.0 documentation 

Cassandra is an open source database that you can use to provide the backing store for a scalable, high-performance solution for collecting time series data like virtual machine metrics. If you want vCloud Director to support retrieval of historic metrics from virtual machines, you must install and configure a Cassandra cluster and use the cell-management-tool to connect the cluster to vCloud Director. Retrieval of current metrics does not require optional database software.

If you are new to Cassandra, then In past I have wrote some blog posts about cassandra architecture and how it works. Feel free to read those posts from below links:

1:Introduction to cassandra

2: Understanding Cassandra Read/Write Mechanism

3: Installing Cassandra on RHEL6

In production you may want a Cassandra cluster with minimum 3-4 nodes, but in lab environment 1 or two nodes is sufficient. In my lab I am running only two node. I am configuring Cassandra on my CentOS 6 box.

vCD 9.0 needs Cassandra version 2.2.6 which can be downloaded from here

Before installing cassandra, we need to install jdk 1.7 or later and set the JAVA_HOME variable. Instructions for this is covered in blog link given above. Although the same blog have steps for installing cassandra, I am including the steps here as well.

Follow below steps for a successful installation of cassandra.

1: Create cassandra user with sudo permissions.

You can use below script which will create a user on server with sudo permissions.

# wget https://raw.githubusercontent.com/zubayr/create_user_script/master/create_user_script.sh
# chmod 777 create_user_script.sh
# sh create_user_script.sh -s cassandra

2: Verify presence of cassandra user/group

[root@cassdb01 ~]# cat /etc/passwd | grep cassandra
cassandra:x:501:501::/home/cassandra:/bin/bash
[root@cassdb01 ~]# cat /etc/group | grep cassandra
cassandra:x:501:

3: Extract cassandra zip file and setup cassandra directory structure

# cp /root/apache-cassandra-2.2.6-bin.tar.gz /opt/ # cd /opt; # tar -zxvf apache-cassandra-2.2.6-bin.tar.gz # ln -s /opt/apache-cassandra-2.6 /opt/apache-cassandra # mkdir -p /var/lib/cassandra/{data,commitlog};mkdir -p /var/log/cassandra read more

Find vCloud Director Orphaned VM’s

By | 03/10/2017

We all are familiar with concept of orphaned VM’s in vSphere. However orphaned VM’s in vCloud Director have slightly different meaning. 

From vCD perspective, virtual machines that are referenced in the vCenter database but not in the vCloud Director database are considered orphan VMs because vCD cannot access them even though they may be consuming compute and storage resources. This kind of reference mismatch can arise for a number of reasons, including high-volume workloads, database errors, and administrative actions.

Starting with vCD 8.2, VMware added one more option to cell management utility to locate such orphaned VM’s so that they can be removed or re-imported into vCloud Director. This utility is not available with any vCD version prior to 8.20.

The command to find orphaned VM’s is find-orphan-vms command which is used in conjunction with cell-management-tool and enables an administrator to list these VMs .

To list the options available with this command, run command: 

# /opt/vmware/vcloud-director/bin/cell-management-tool find-orphan-vms –help

vcd-orphaned-vms.PNG

If you are using self-signed certificates in vCD, then you have to specify truststore file and truststore password along with supplying vcd username/password and vcenter credentials etc. This is the full command that one can use: 

[root@vcd-a bin]# ./cell-management-tool find-orphan-vms –host vcd-a.alex.local -u admin –password vCD-Pwd –trustStore /opt/vmware/vcloud-director/vcd.ks –trustStorePassword vCloud@2017 –vc-name resourcevc01.alex.local –vc-user vcadmin@alex –vc-password passwd –output-file vmlist.txt

The above command will query all all Org VDC’s that are present in vCD and present the list of VM’s in the output file specified in above command.

Querying for VC by name resourcevc01.alex.local
Querying all VDCs associated with VC: resourcevc01.alex.local (https://resourcevc01.alex.local:443/sdk)
Querying all VDC<->resource pool mappings associated with VC: resourcevc01.alex.local (https://resourcevc01.alex.local:443/sdk)
Querying all VDC<->VM Moref mappings associated with VC: resourcevc01.alex.local (https://resourcevc01.alex.local:443/sdk)
Processing 1 VMs on 2 VDCs across 2 resource pools
Analysis complete.

You can use cat or less utility to see the contents of the output file. The file will list all orphaned VM’s/Org VDC. In my lab I did not had any orphaned VM’s so the list is empty

[root@vcd-a bin]# cat vmlist.txt VDC: "Service_VDC [urn:vcloud:vdc:fcdf292a-b886-4dc9-9b6f-3a4df1a97782]" (org: "Service-vDC") VDC: "Prod-DC [urn:vcloud:vdc:1c32832c-e4db-48b0-9cfc-116fbb302625]" (org: "Production") read more

Migrate vCloud Director 9.0 DB from MSSQL to Postgres

By | 02/10/2017

With vCloud Director 9.0, VMware introduced postgres as supported database for vCD. If you are planning to use postgres as DB, then you should install Postgres v 9.5 on a supported OS.

In our last Post I mentioned that I purposefully configured MSSQL as DB from my new vCD 9.0 installation, as I wanted to test the migration of vCDDB from MSSQL to Postgres. This post is focussed on how to do so.

If you are new to postgres and do not know how to install it, then follow this blog for installation instructions which are pretty easy and straight forward.

Once you have installed postgres and started services, next is to create database for vCD. Follow below commands to do so

1: Create Database

postgres=# CREATE DATABASE vcloud;
CREATE DATABASE

2: Verify presence of newly created database

postgres=# \l
 List of databases
 Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+-------+-----------------------
 postgres | postgres | UTF8 | C | C |
 template0 | postgres | UTF8 | C | C | =c/postgres +
 | | | | | postgres=CTc/postgres
 template1 | postgres | UTF8 | C | C | =c/postgres +
 | | | | | postgres=CTc/postgres
 vcloud | postgres | UTF8 | C | C |
(4 rows)

3: Create vCloud user and assign password to user

postgres=# create user vcloud; CREATE ROLE postgres=# alter user vcloud password 'Telstra@123'; ALTER ROLE read more

vCloud Director 9.0 Single Cell Installation

By | 30/09/2017

vCD 9.0 was announced yesterday and in my last post I touch based on some of the new features that are included in this new release. In this post we will see how to install and configure vCD 9.0.

vCloud Director Hardware and Software Requirements

Before starting with vCloud director 9.0 installation, please make sure all hardware and software requirements are met. Instructions are laid out here

Installing and Configuring a vCloud Director Database

In my lab I am using MSSQL for vCD database. Although I wanted to try the installation with Postgres database, but I refrained myself from doing so as I wanted to test cell management utility for migrating vCD database from MSSQL to Postgres later.

Below script will be used in configuring the database. You have to make minor changes in the script to setup according to your infrastructure

USE [master] GO CREATE DATABASE [vcloud] ON PRIMARY (NAME = N'vcloud', FILENAME = N'C:\vcloud.mdf', SIZE = 100MB, FILEGROWTH = 10% ) LOG ON (NAME = N'vcdb_log', FILENAME = N'C:\vcloud.ldf', SIZE = 1MB, FILEGROWTH = 10%) COLLATE Latin1_General_CS_AS GO USE [vcloud] GO ALTER DATABASE [vcloud] SET SINGLE_USER WITH ROLLBACK IMMEDIATE; ALTER DATABASE [vcloud] SET ALLOW_SNAPSHOT_ISOLATION ON; ALTER DATABASE [vcloud] SET READ_COMMITTED_SNAPSHOT ON WITH NO_WAIT; ALTER DATABASE [vcloud] SET MULTI_USER; GO USE [vcloud] GO CREATE LOGIN [vcloud] WITH PASSWORD = 'vcloudpass', DEFAULT_DATABASE =[vcloud], DEFAULT_LANGUAGE =[us_english], CHECK_POLICY=OFF GO CREATE USER [vcloud] for LOGIN [vcloud] GO USE [vcloud] GO sp_addrolemember [db_owner], [vcloud] GO read more

vCloud Director 9.0 Released

By | 30/09/2017

Yesterday VMware released vCloud Director 9.0 and this version of vCD brings many new things on table which customer’s were looking for some time. We will discuss about Whats new with this release. Vcloud Director 9.0 was first anoounced by VMware in VMworld 2017 on 28/08/2017.

The GA version of vCloud Director 9.0 is build 6681978 and is available for download for service providers from here

Whats new with this release?

Below new features are in this release:

  • New HTML5 based UI which is fast (same like HTML client for vSphere). However, the old Flex based UI is still available.
  • New VM lifecycle workflows: VM creation task is now completed in one screen as opposed to maximum of seven in previous releases of vCD.
  • Multisite management:  If a tenant has workloads running across different Org vDC’s and in more than once vCloud Director instance, then tenant can access all of them via single portal.
  • vCloud Director Extender: It offers migrations between Cloud, an extension of L2 where NSX is not required in local environments. This means that customers with a simple vCenter will be able to connect to resources offered by vCloud Director providers and move workloads there. You can read more about vCloud Extender form here
  • Support for PostgreSQL: Postgres v 9.5 is now included as supported database. Earlier only MSSQL and Oracle was supported. If you want to migrate from MSSQL or Orcale DB to Postgres, then you can do so via cell Management Tool utility (I am pretty excited to test this in lab today)
  • Support for vSAN and VVol (Customers were looking for both of these for quiet some time)
  • Application Extensibility for the UI to allow 3rd party plugins is a significant new feature and will allow vendors to tap into vCD even more to enhance the client experience.
  • Trunking support for External and Routed Org Networks.
  • Up to 100 milliseconds latency between vCenter Server and vCloud Director is supported.
  • VM metrics available in UI: Earlier the VM metrics were accessible only via API and that too required routing metrics data from vCD to cassandra database. With vCD 9.0 metrics data is now part of the new HTML5 Tenant UI.

Useful links:

Video Tutorials

vCloud Director 9.0 HTML 5 Tenant Portal read more

How to locate iso file uploaded in vCloud Director on backend datastore

By | 08/09/2017

6 Months back I got a case where one customer deployed a Cisco ASA v10 appliance in his on-prem and attached 2 CD drives in that VM and then transferred that VM in vCloud Air. Post transfer of VM, customer was not able to power on the VM as the second CD drive of VM was not mapped to iso which customer uploaded in his catalog.

If you are familiar with vCD UI, then you might be aware of the fact that vCD do not provides an option to end user to specify particular CD ROM device for inserting ISO file. The only option which user gets is “Insert CD/DVD from Catalog’ and when an iso is inserted, it is always mapped to first CD ROM device at vCenter level. read more

vCloud Availability for vCloud Director-Part-1-Introduction

By | 01/09/2017

Few days back I was reading a blog by Tom Fojta about vCloud Availability for vCloud Director and had a quick look into what this product is and how it works. I was busy with some other assignments so did not got much chance to read about that in greater detail, so today I decided to know more about this products (as it is based on DRaaS, which is always an eye catching topic)

This post is focused on very basic stuffs about vCloud Availability for vCloud Director. So lets get started.

What is vCloud Availability for vCloud Director? read more

Cannot Redeploy Edge Gateway “VSM response error (10020): Failed to deploy edge appliance vse-XXXX-0. The name ‘vse-XXXX-0’ already exists”

By | 12/07/2017

This post is very similar to issue described in my last post. The only difference in last issue and this was I was not able to redeploy edge gateway to get rid of stubborn Org Networks whereas in previous case Edge redeploy fixed the issue quite comfortably.

Let me start with a little bit background of how was this issue discovered and what challenges I faced.  I was working investigating a failed deprovision issue when this issue was discovered. Deprovision tasks in our environment are fully automated and we have some portal where these tasks arrives and there is a Resume button which when clicked, kicks the deprovision process.

When the Resume button is clicked that portal initiates API calls to vCD and start deleting stuffs. It starts with deleting vApps, vApp Templates and then proceed to Org Network deletion and then the edge gateway and at last deletes the Org vDC and Org.

Sometimes stuffs at vCD level are in inconsistent state and thus API calls are unable to delete that element and deprovision is halted in portal.

During my investigation I checked the logs and found that API calls were unable to remove one of the Org Network.

Following errors were visible in vCD UI for network deletion failure

[ 695e10af-1677-4c64-bbe1-42250b6c249d ] Cannot delete organization VDC network default-routed (0694f25a-78b9-45b0-be44-e5c8ccda4b91)
Failed to delete interface of edge gateway urn:uuid:5286e85d-afb0-4821-b4f4-db87b390ba11

- Failed to delete interface of edge gateway urn:uuid:5286e85d-afb0-4821-b4f4-db87b390ba11
 
- com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (202): The requested object : vm-3768 could not be found. Object identifiers are case sensitive.

From the logs it was very clear that there are issues with edge backing VM’s. I went ahead with performing edge gateway redeploy without checking the edge VM’s status in vCenter. I was thinking that redeploy fixes this issue 9 out of 10 times so just give it a shot.

To my surprise edge gateway redeploy also failed and also I observed that redeploy task took around 20 minutes (usually it takes 5-7 minutes) and eventually timed out. 

Errors related to edge redeploy task failing was

[ e04b76e6-7bb1-4d97-a85c-0df2813a06be ] Cannot redeploy edge gateway M738162563-11503 (urn:uuid:5286e85d-afb0-4821-b4f4-db87b390ba11) com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (10020): Failed to deploy edge appliance vse-xxxxx-0. (The name 'vse-xxxxx-0' already exists.) - com.vmware.vcloud.fabric.nsm.error.VsmException: VSM response error (10020): Failed to deploy edge appliance vse-xxxxx-0. (The name 'vse-xxxxx-0' already exists.) - VSM response error (10020): Failed to deploy edge appliance vse-xxxxx-0. (The name 'vse-xxxxx-0' already exists.) read more

Edge Gateway Network deletion failed with error “Failed to communicate with NSX Edge vm. Error code VIX_E_PROGRAM_NOT_STARTED was returned by VIX API”

By | 11/07/2017

Today while working on one production issue, I came across one incident where I was unable to delete one of the Org Network in vCloud Director. Observed following errors in vCD UI for the Org network deletion failure: 

On checking vcloud-container.debug.log I observed similar log entries as seen in vCD UI

This was entirely new error for me so I started googling this around and unfortunately did not found helpful article. The only article which I got for this error was this but of no use for me.  read more

Detaching and Deleting Independent Disks in vCloud Director via REST API

By | 21/06/2017

Yesterday while working on one of the production issue where we had to deprovision a tenant environment in vCloud Air, I noticed that independent disks were preventing automated deprovision of the environment and the error messages were loud and clear in the log files.

It was a new issue for me so I started reading about independent disks in vCloud Director and want to share few things about this.

First of all independent Disk feature in vCD is completely different from an Independent Disk in vSphere. Independent disks can be shared across multiple vApps/VM’s in vCloud Director. This feature was first introduced in vCD v5.1.

Following quote from vCloud Architecture Toolkit document rightly explains about independent disks

The use of independent disks with vCloud Director allows updates of virtual machines without impacting the underlying data.

The feature is designed to enable users to create virtual disks which can be attached to and detached from virtual machines. There is no functionality to control this feature from the vCD UI and this can be controlled via API’s only. 

When you create an independent disk, it is associated with an organization vDC but not with a virtual machine. After the disk has been created, the disk owner or an administrator can attach it to any virtual machine deployed in that vDC, detach it from a virtual machine, and remove it from the vDC.

Presence of Independent disks in vCD can be seen on navigating to Org > Administration > Org vDC > Independent Disks tab. If you right click on any of the disk you will not see any action window opening. 

0.PNG

In this post I am going to demonstrate how we can detach/delete independent disks from VM via API calls. Lets get started.

For sake of this demonstration, I have used some hypothetical names for Org and Org vDC.

Step 1: Obtain vCD Auth token code

# curl -sik -H “Accept:application/*+xml;version=5.6” -u “admin@system” -X POST https://vCD-FQDN/api/sessions | grep auth

Enter host password for user ‘admin@system’:

x-vcloud-authorization: Auth

Step 2: Locate your Org 

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization:Auth-X GET https://vCD-FQDN/api/org/

On using the above API call, you will see a href to all your Org that are present in vCD. For your next query chose the href of the org where independent disks are lying.  

<Org href="https://vCD-FQDN:443/api/org/08356307-2939-42d3-a2a2-aeccef6478e4" name="ABC" type="application/vnd.vmware.vcloud.org+xml"/>

<Org href="https://vCD-FQDN:443/api/org/2b729e6f-588e-49c4-964f-89b2e744c075" name="DEF" type="application/vnd.vmware.vcloud.org+xml"/>

<Org href="https://vCD-FQDN:443/api/org/fc432145-f1f3-42f6-a26f-eeb3d306a405" name="GHI" type="application/vnd.vmware.vcloud.org+xml"/>

Step 3: Locate your Org vDC

# curl -sik -H “Accept:application/*+xml;version=5.6” -H “x-vcloud-authorization: Auth” -X GET https://vCD-FQDN:443/api/org/fc432145-f1f3-42f6-a26f-eeb3d306a405 | grep vdc

<Link rel="down" href="https://vCD-FQDN:443/api/vdc/adf0929b-a107-4671-9f85-b629b744c2b7" name="VDC1" type="application/vnd.vmware.vcloud.vdc+xml"/> read more

Replacing vCD SSL Certificates in a Multi Cell Environment

By | 20/06/2017

After a long wait I finally got chance to work on vCloud Director ssl certificates. This was the only component in my lab which was still using self-signed certs and that encouraged me to do something new in lab.

A note on vCD SSL certificates

vCloud Director like any other VMware product needs a certificate to be installed on the device that it uses for communication with the other products. By default vCD uses a self-signed certificate. If you have a certificate authority in your environment then you can get the certs created in advance before installing vCloud director and save your self from pain of messing with certificates at later stages. read more

Troubleshooting Failed Org Network Creation in vCloud Director

By | 08/04/2017

Today while working in my lab, I observed that while creating a new VDC in vCD was failing because org network failed to create.

On navigating to Org VDC list and clicking on error, it read the error load and clear that org vdc network can’t be created.

vcd-1

On navigating to Org VDC Networks section and clicking on error, I was able to identify what has caused the network creation failure.

vcd-2

The error stack was reading as below:

[ 114db22d-fc14-4c87-9030-36d2316aff8b ] Cannot deploy organization VDC network (f1514426-647e-4a03-a5a9-fafa4d73bb58)
com.vmware.vcloud.api.presentation.service.InternalServerErrorException: Cannot create network “dvs.VCDVSRouted-NW-9ab02973-9ded-4c4b-8826-4a52bdf2d6cf” from VXLAN network pool “urn:uuid:5c9de104-0f40-4cec-898f-985ee1fce1d6”. Make sure vShield Manager infrastructure is properly configured and there are segment IDs available. read more