Objective 6.1 of VCAP6-Deploy exam covers following topics:
- Configure esxtop / resxtop custom profiles
- Evaluate use cases for and apply esxtop / resxtop Interactive, Batch and Replay modes
- Use esxtop / resxtop to collect performance data
- Given esxtop / resxtop output, identify relative performance data for capacity planning purposes
Before starting discussing on these topics, I want to cover a few basics of vSphere Management Assistant (vMA) as we will be using it for performing few tasks listed in this objective.
What is vSphere Management Assistant (vMA)?
The vSphere Management Assistant (vMA) is a virtual machine that includes prepackaged software such as a Linux distribution, the vSphere command‐line interface, and the vSphere SDK for Perl. Basically it is the missing service console for ESXi. But it’s more than that too.
This allows administrators to run scripts or agents that interact with ESX/ESXi and vCenter Server systems without having to explicitly authenticate each time. vMA can also collect ESX/ESXi and vCenter Server logs and store the information for analysis.
vMA 6.0 can be downloaded from here. It is available in the form of .ova file and installation is pretty much straight forward like any other VMware appliance and I am not covering installation steps. Basic installation/configuration steps can be read from this article.
Getting started with vMA
SSH to vMA appliance and login with user ‘vi-admin’ and password set during deployment. The first task is to add remote servers to vMA so that we can execute commands on those servers without logging onto them.
1: Adding a remote server
vi-admin@vmasrv:~> vifp addserver esxi01.alex.local --authpolicy fpauth --username root --password <password>
2: List Servers : Once you have added all your remote servers, you can list them by executing below command
vi-admin@vmasrv:~> vifp listservers
esxi01.alex.local ESXi 6.0.0 5050593
esxi02.alex.local ESXi 6.0.0 5050593
esxi03.alex.local ESXi 6.0.0 5050593
3: Set Remote Server : To execute command on a remote server which is added to vMA previously, we have to first set it as target server
vi-admin@vmasrv:~> vifptarget --set esxi01.alex.local
4: Add Esxi SSH Thumbprint: When an Esxi host is set as target server, and you try to execute a command against that server, you will see a message about connection failed as the server thumbprint is not trusted. We have to add SSH thumbprint of target server on using credstore_admin.pl utility
vi-admin@vmasrv:~[esxi01.alex.local]> esxcli network ip route ipv4 list
Connect to esxi01.alex.local failed. Server SHA-1 thumbprint: CB:06:FE:BB:89:88:79:18:68:79:AD:42:67:02:CF:15:E5:5F:D6:25 (not trusted).
vi-admin@vmasrv:~[esxi01.alex.local]> /usr/lib/vmware-vcli/apps/general/credstore_admin.pl add -s esxi01.alex.local -t CB:06:FE:BB:89:88:79:18:68:79:AD:42:67:02:CF:15:E5:5F:D6:25
New entry added successfully
5: Execute command : Once a thumbprint is added, you can execute command on the added target server
vi-admin@vmasrv:~[esxi01.alex.local]> esxcli network ip route ipv4 list
Network Netmask Gateway Interface Source
------------- ------------- ------------- --------- ------
default 0.0.0.0 192.168.109.1 vmk0 MANUAL
192.168.106.0 255.255.255.0 0.0.0.0 vmk2 MANUAL
192.168.107.0 255.255.255.0 0.0.0.0 vmk4 MANUAL
192.168.108.0 255.255.255.0 0.0.0.0 vmk1 MANUAL
192.168.109.0 255.255.255.0 0.0.0.0 vmk0 MANUAL
Now since basics of vMA has been covered, lets move on to main topics of this objective.
Configure esxtop / resxtop custom profiles
To gather performance data and monitor an ESXi host, following 2 tools can be used:
- esxtop. This command runs locally on an Esxi host and provides real time CPU, memory, disk and network data for hosts and virtual machines.
- resxtop. This is a remote version of esxtop. It is included as part of vCLI and is present on the vMA.
By default when you run esxtop command, you are presented with cpu utilization related stats of an Esxi host, but esxtop has many different displays and options around which you can play.
When running esxtop simply select ‘h’ to see the options available. Typically you will see the following options on your screen
Now suppose you want to see network related statistics of Esxi host, just press ‘n’ and the view will be changed
If you wish to exclude some fields which you do not want to monitor, then press ‘f’ button and press the letter to toggle the field.
For example if I do not want to Include field MegaBits Rx/s, then I will press ‘K’ and if I want to include Multicast packets, I will press ‘P’ and hit enter. You will see 2 new fields added in the network statistics.
There are lot and lot of options that can be used with esxtop. This article covers a full list of available options.
As we discussed earlier, resxtop is available with vMA. To run resxtop against an Esxi host, fire below command
vi-admin@vmasrv:~> resxtop --server esxi01.alex.local
and it will show you cpu statistics of the host by default. You can toggle the fields which you want to see and can create custom profiles and save it so that next time when you run that profile, you will see only that data which you need.
To save the current metrics into a profile we can save by using a W, and then naming the profile. For example I am saving esxi01 cpu statistics in a file named esxi01-cpu.
Now when you want to run resxtop against this host and want to see the custom fields, you don’t have to toggle the options. Just run the profile which you saved earlier
vi-admin@vmasrv:~[esxi01.alex.local]> resxtop -c esxi01cpu
Evaluate use cases for and apply esxtop / resxtop Interactive, Batch and Replay modes
There are 3 modes in which esxtop/resxtop can be used. These modes are:
- Interactive: This is default mode in which statistics are displayed in real time in different panels.
- Batch: In batch mode, resxtop (or esxtop) does not accept interactive commands. This mode is used for collecting statistics over a set period of time. Further analyisis is performed retrospectively using several different tools including Microsoft Excel, ESXlot and Perfmon.
- Replay: In replay mode, esxtop replays resource utilization statistics collected using vm-support. We can view the captured performance information for a particular duration or time period as like real time to view what was happened during that time.
Running esxtop/resxtop in Batch mode
To collect esxtop/resxtop data in batch mode, fire a command similar to:
[root@esxi01:~] esxtop -b -a -d 5 -n 10 > /tmp/esxi01.csv
vi-admin@vmasrv:~[esxi01.alex.local]> resxtop -b -a -d 5 -n 10 > /tmp/resxtop-esxi01.csv
- b = batch mode
- a = all of the statistics
- d = delay in seconds between statistics snapshots
- n = number statistic snapshots saved before batch is stopped.
Running esxtop in Replay mode
To run esxtop in replay mode, login to Esxi host via root user over SSH session and use a command similar to as shown below, to capture performance statistics:
[root@esxi01:~] vm-support -p -d 15 -i 10 -w /vmfs/volumes/591ac3ec-cc6af9a9-47c5-0050560346b9/esxi01-perfdata
- p : is to only gather performance data
- d : delay in seconds between statistics snapshots
- i : is the number of iteration
- w : is to specify a datastore for the output file
Once the command completes, the collected data will be saved in a zipped file stored on the location provided in command
Before running the esxtop command in replay mode, we have to extract the zipped file
[root@esxi01:/vmfs/volumes/591ac3ec-cc6af9a9-47c5-0050560346b9/esxi01-perfdata] tar -xzf esx-esxi01-2017-12-01--12.00.tgz
Once the file is extracted, you will find a new directory.
[root@esxi01:/vmfs/volumes/591ac3ec-cc6af9a9-47c5-0050560346b9/esxi01-perfdata] ls -l
drwxr-xr-x 1 root root 3080 Dec 1 12:14 esx-esxi01-2017-12-01--12.00
To run the esxtop in replay mode, run esxtop command with -R and name of extracted directory. In my case I had to run
# esxtop -R esx-esxi01-2017-12-01--12.00
If replay mode does not start and you get an error message similar to “No such file or directory esxtop […] all vm-support snapshots have been used“, then don’t be disheartened. The fix to this problem is “go inside the directory and run reconstruct.sh file.
[root@esxi01:/vmfs/volumes/591ac3ec-cc6af9a9-47c5-0050560346b9/esxi01-perfdata] esxtop -R esx-esxi01-2017-12-01--12.00
VsiCache_Init():Failed to open(esx-esxi01-2017-12-01--12.00/hungvm/vsi/vsi.0): No such file or directory
esxtop: cannot find esx-esxi01-2017-12-01--12.00/hungvm/vsi/vsi.0, all vm-support snapshots have been used.
Now when you run extop -R command, you will be able to see the data.
Note: The last line in the output: “Read stats from …” indicates that ESXTOP is running in replay mode
Use vScsiStats to gather storage performance data
vscsiStats is a tool controls vscsi data collection for virtual machine disk disk I/O workload characterization. Users can start and stop online data data collection for particular VMs as well as print out online histogram data.
To use vscsiStats for Storage Performance Analysis, there is an awesome document created by drummonds on VMTN
To see a full list of options that can be used with vscsi command, run vscsiStats -h
We need to have VMWorldID before we can use vscsiStats command to analyze VM disk. To do so run vscsiStats -l command and note the world groupID
To start collecting data: vscsiStats -s -w <WorldGroupID>
To stop collecting data: vscsiStats -x
You can choose one of the following to see the information in histogram:
Note: vscsiStat should be running before we can view the info in histogram
To see the collection: vscsiStats -p latency
You can redirect the screen output to a file by running command:
[root@esxi01:~] vscsiStats -p all -c > /vmfs/volumes/iSCSI-1/vscsiStatoutput.csv
You can reset the statistics by typing: vscsiStats -r
Use esxtop / resxtop to collect performance data:
I am not pretty sure what is the goal of this objective. This may or may not include deciphering verious performance related issues via esxtop command or using vm-support command with (-p) option. For general troubleshooting I followed this article by Matt Callaway
Given esxtop / resxtop output, identify relative performance data for capacity planning purposes
We aready covered this part earlier in this blogpost.
I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂
ESXTOP Troubleshooting Poster
Using vscsiStats for Storage Performance Analysis
Interpreting esxtop Statistics