VCAP6-DCV Deploy Objective 5.3

Objective 5.3 of VCAP6-Deploy exam covers following topics:

  • Generate vSphere log bundles
  • Configure and test centralized logging
  • Analyze log entries to obtain configuration information
  • Analyze log entries to identify and resolve issues
  • Configure logging levels for vSphere

                                                        Generate vSphere log bundles

There are various ways to view/generate log bundles of Esxi host and vCenter server. We will look at all of them one by one. I will start with Esxi host logs first.

1: From the DCUI 

logs-1.PNG

2: Esxi host Web Browser: https://esxi_fqdn_or_ip/host

logs-2

3: C# client : Connect directly to Esxi host and from home menu click on system logs

logs-3

From the drop-down menu, select the log and entry you want to view.

logs-18.PNG

4: Web Client : Login to vSphere Web Client and select a vcenter server from inventory and navigate to Monitor > System Logs and click on Export System Logs and select an Esxi host from the list. Optionally you can include vCenter server and web client logs as well alongwith host logs. 

logs-4

Specify which log files you need to export.

logs-5

5: Power CLI: Run below command to generate host log via PowerCLI

Get-VMHost Esxi-FQDN-IP | Get-Log -Bundle -DestinationPath <path>

logs-8.PNG

6: Via Esxi command line: Connect to Esxi user via SSH and login via root user and run command: /usr/bin/vm-support

log-11.PNG

vCenter Logs

1: Web browser : https://vcenter-FQDN/appliance/support-bundle or https://psc-FQDN/appliance/support-bundle

After you enter credentials, a support bundle will start downloading

logs-6

2: PowerCLI: Run below command to generate vCenetr server log via PowerCLI

Get-Log -Bundle -DestinationPath <path>

logs-9.PNG

3: Web Client: Login to vSphere Web Client and navigate to Administration > System Configuration > Objects > Nodes. Select a node and from Action dropdown menu select Export Support Bundles

logs-10.PNG

Select what you want to export and whats not and click on “Export Support Bundle”

logs-12

4: Command Line: Connect to the vCenter Server or PSC appliance via root user over SSH and switch to bash shell by typing “Shell” and run command : vc-support -l

Command will take 5-7 minutes to generate the log bundle. 

logs-13.PNG

Once it is completed, the Support bundle will be saved in the format of “vc-<FQDN_of-PSC>-<Date>.tgz.” under /Storage/log

                                               Configure and test centralized logging

An Esxi host or vCenter server (VCSA) can be configured to forward the system logs to a remote Syslog Server. 

Forwarding Esxi host logs to syslog server:

ESXi 5.0 and higher hosts run a syslog service (vmsyslogd) that provides a standard mechanism for logging messages from the VMkernel and other system components. By default in ESXi, these logs are placed on a local scratch volume or a ramdisk.

To preserve the logs further, ESXi can be configured to place these logs to an alternate storage location on disk and to send the logs across the network to a syslog server. Follow below steps to configure syslog settings for an Esxi host

1: SSH to Esxi host and login via root user and run below command: 

# esxcli system syslog config set -- loghost='tcp://syslog-ip-fqdn:514'

2: Reload the syslog service

# esxcli system syslog reload

3: Open the firewall ports for syslog on Esxi host

# esxcli network firewall ruleset set -r syslog -e true

4: Test the connectivity to syslog server

# nc -z syslogsrv.alex.local 514
Connection to syslogsrv.alex.local 514 port [tcp/shell] succeeded!

There are other ways to achieve the same. I have wrote an article on this topic in past so I am not covering those steps again. 

Forwarding VCSA logs to syslog server:

VCSA comes installed with syslog-ng by default which is used to provide the vSphere Syslog Collector functionality as well as the local syslog client for the VCSA itself.

To forward VCSA logs to a remote syslog server login to vSphere Web Client using administrator@sso-domain and navigate to Administration > System Configuration > Nodes > Select a node from list > Related Objects and search for syslog

log-19.PNG

Edit the syslog settings using teh pencil button and fill up following details

Common Log Level: The supported values for this field are:

  • * : include all log files.
  • info: Only informational log files are redirected to the remote machine.
  • notice: Only notices are redirected to the remote machine. A notice message indicates a normal but significant condition.
  • warn: Only warnings are redirected to the remote machine.
  • error: Only error messages are redirected to the remote machine.
  • crit: Only critical log files are redirected to the remote machine.
  • alert: Only critical log files are redirected to the remote machine.
  • emerg: Only emergency log files are redirected to the remote machine. An emergency message indicates that the system has stopped responding and cannot be used.

Remote Syslog Host: The IP address of the host you want to use for storing ESXi messages and logs. This is also the IP address of the remote syslog server on the network you use to redirect logs and ESXi messages

Remote Syslog Port: The port number to use for communication with the machine to which you want to export log files.

Remote Syslog Protocol: The communication protocol that Syslog uses. Available protocols are TCP, UDP, and TLS.

log-20.PNG

After supplying all the values, click on Restart from Actions menu for changes to take effect. 

For forwarding logs from a Windows based vCenter, William lam has wrote and excellent article. Feel free to check it out.

                            Analyze log entries to obtain configuration information

To check the configuration file browse to the host using a web browser https://host-fqdn-ip/host

logs-2

The common configuration files that can be viewed from here are: host file, license file, certificate files and host agent config file etc.

Auditing Esxi Shell Logins and Commands

Esxi logs reside in /var/log directory. To review these logs, ssh to an Esxi host directly and login via root user and review following log files

Auth.log : Here you can see login failure/success messages when a user attempt to login to Esxi host

auth-log.PNG

Shell.log: This log file gives you a list of commands that were fired via Esxi shell.

shell-log.PNG

                                    Analyze log entries to identify and resolve issues

Analyzing logs and troubleshooting issues comes with experience and its very hard to write on this topic as every problem is unique in some way and the problems varies from environment to environment as every environment has their own design and stuffs are setup’ed differently.

However VMware has published some very handy KB’s/Docs for how to get started with anaylyzing issues and performing troubleshooting. I have listed few of them:

Esxi log: Esxi logs are present in /var/run/log

esxi-logs.png

vCenter Logs: 

vcenter-logs.png

vcenter-logs2.png

VMware KB-2110014 tells the log location of various log files that resides on vCenter appliance. And this article explains about Esxi Log File Locations.

                                        Configure logging levels for vSphere

Changing Esxi host logging level

Global Log Level: Select an Esxi host and navigate to Manage > Settings > Advanced System Settings and edit the setting Config.HostAgent.log.level

log-13.PNG

You can choose one of the option as shown in below screenshot

log-14

To change logging level of vpxa, edit the parameter Vpx.Vpxa.config.log.level

log-15

Changing vCenter server logging level

To change overall logging level of vCenter server, select a VC from inventory and navigate to Manage > Settings > general and click on Edit button.

Choose one of the logging level from the list as shown in below screenshot. 

log-16

This is explanation of each logging level

log-17.PNG

Note: Changes done to the logging level via the vSphere Client or vSphere Web Client do not persist after a reboot and are overwritten by the default values in the vpxd.cfg file. To make permanent log level modifications, you must edit the vpxd.cfg file. VMware KB-1004795 has the steps for modifying the vpxd.cfg file. 

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂