Configure Centralized Logging on ESXi 6 Hosts

In this post we will learn how to configure Esxi-6 hosts to send the logs to a centralized syslog server.

Purpose of configuring syslog server?

As per VMware KB-2003322

ESXi 5.0 and higher hosts run a syslog service (vmsyslogd) that provides a standard mechanism for logging messages from the VMkernel and other system components. By default in ESXi, these logs are placed on a local scratch volume or a ramdisk.

To preserve the logs further, ESXi can be configured to place these logs to an alternate storage location on disk and to send the logs across the network to a syslog server.

Retention, rotation, and splitting of logs received and managed by a syslog server are fully controlled by that syslog server. ESXi cannot configure or control log management on a remote syslog server.

How to configure Esxi hosts for centralized logging?

There are various ways to configure syslog settings on Esxi hosts. These includes:

1: Using esxcli command on Esxi host.

2: Using vSphere Web-Client.

3: Using vSphere Thick client.

4: Using PowerCli.

5: Using Host Profiles.

We will look individually on all available method one by one. Let’s get started.

Before configuring esxi hosts to send logs to syslog server, we need to have a syslog server in our environment. I have configured my syslog server on a CentOS 6 box following instructions illustrated here

I added additional 2 lines at the bottom of rsyslog.conf file so that all hosts should have their logs in their individual folder

$template TmplAuth, "/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?TmplAuth

1 2	$template TmplAuth, "/var/log/%HOSTNAME%/%PROGRAMNAME%.log" . ?TmplAuth

Configuring Syslog Using esxcli utility

The command to configure syslog settings on Esxi hosts is esxcli system syslog config

Lets first see what are the available options with this command.

[root@esxi01:~] esxcli system syslog config set –help

With this command we have following options available:

--check-ssl-certs : Verify remote SSL certificates against the local CA Store

--default-rotate :  Default number of rotated local logs to keep

--default-size : Default size of local logs before rotation, in KiB

--default-timeout :  Default network retry timeout in seconds if a remote server fails to respond

--drop-log-rotate : Number of rotated dropped log files to keep

--drop-log-size : Size of dropped log file before rotation, in KiB

--logdir : The directory to output local logs to

--logdir-unique : Place logs in a unique subdirectory of logdir, based on hostname

--loghost : The remote host(s) to send logs to

--queue-drop-mark : Message queue capacity after which messages are dropped

--reset : Reset values to default

--check-ssl-certs : Verify remote SSL certificates against the local CA Store

--default-rotate : Default number of rotated local logs to keep

--default-size : Default size of local logs before rotation, in KiB

--default-timeout : Default network retry timeout in seconds if a remote server fails to respond

--drop-log-rotate : Number of rotated dropped log files to keep

--drop-log-size : Size of dropped log file before rotation, in KiB

--logdir : The directory to output local logs to

--logdir-unique : Place logs in a unique subdirectory of logdir, based on hostname

--loghost : The remote host(s) to send logs to

--queue-drop-mark : Message queue capacity after which messages are dropped

--reset : Reset values to default

Next is to retrieve the current syslog configuration.

[root@esxi01:~] esxcli system syslog config get
Default Network Retry Timeout: 180
 Dropped Log File Rotation Size: 100
 Dropped Log File Rotations: 10
 Enforce SSLCertificates: false
 Local Log Output: /scratch/log
 Local Log Output Is Configured: false
 Local Log Output Is Persistent: true
 Local Logging Default Rotation Size: 1024
 Local Logging Default Rotations: 8
 Log To Unique Subdirectory: false
 Message Queue Drop Mark: 90
 Remote Host: <none>

[root@esxi01:~] esxcli system syslog config get

Default Network Retry Timeout: 180

Dropped Log File Rotation Size: 100

Dropped Log File Rotations: 10

Enforce SSLCertificates: false

Local Log Output: /scratch/log

Local Log Output Is Configured: false

Local Log Output Is Persistent: true

Local Logging Default Rotation Size: 1024

Local Logging Default Rotations: 8

Log To Unique Subdirectory: false

Message Queue Drop Mark: 90

Remote Host: <none>

As you can see Remote Host is currently set to none, which means there is no syslog server configured on Esxi host.

To set syslog server on host, we can use the below command

[root@esxi01:~] esxcli system syslog config set –loghost syslogsrv.alex.local

Next is to reload the syslog configuration

[root@esxi01:~] esxcli system syslog reload

Open the firewall ports for syslog on Esxi host

[root@esxi01:~] esxcli network firewall ruleset set -r syslog -e true

Test the connectivity to syslog server

[root@esxi01:~] nc -z syslogsrv.alex.local 514
Connection to syslogsrv.alex.local 514 port [tcp/shell] succeeded!

And that’s it. On checking /var/log/messages on my syslog server, I was able to see logs getting populated

Aug 4 15:54:03 esxi01 Vpxa: info vpxa[50A90B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-77188b05-47] [VpxLRO] -- BEGIN task-internal-134340 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 52a22f84-3c8d-eee9-f2a5-dfa16fdc22ac
Aug 4 15:54:03 esxi01 Vpxa: info vpxa[50A90B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-77188b05-47] [VpxLRO] -- FINISH task-internal-134340
Aug 4 15:54:14 esxi01 Hostd: info hostd[37F40B70] [Originator@6876 sub=Libs] SOCKET creating new socket, connecting to /var/run/vmware/usbarbitrator-socket
Aug 4 15:54:14 esxi01 Hostd: info hostd[37F40B70] [Originator@6876 sub=Libs] SOCKET connect failed, error 2: No such file or directory
Aug 4 15:54:14 esxi01 Hostd: warning hostd[39540B70] [Originator@6876 sub=UserDirectory] Group lookup failed for 'ALEXESX Admins'

Aug 4 15:54:03 esxi01 Vpxa: info vpxa[50A90B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-77188b05-47] [VpxLRO] -- BEGIN task-internal-134340 -- vpxa -- vpxapi.VpxaService.fetchQuickStats -- 52a22f84-3c8d-eee9-f2a5-dfa16fdc22ac

Aug 4 15:54:03 esxi01 Vpxa: info vpxa[50A90B70] [Originator@6876 sub=vpxLro opID=PollQuickStatsLoop-77188b05-47] [VpxLRO] -- FINISH task-internal-134340

Aug 4 15:54:14 esxi01 Hostd: info hostd[37F40B70] [Originator@6876 sub=Libs] SOCKET creating new socket, connecting to /var/run/vmware/usbarbitrator-socket

Aug 4 15:54:14 esxi01 Hostd: info hostd[37F40B70] [Originator@6876 sub=Libs] SOCKET connect failed, error 2: No such file or directory

Aug 4 15:54:14 esxi01 Hostd: warning hostd[39540B70] [Originator@6876 sub=UserDirectory] Group lookup failed for 'ALEXESX Admins'

Also i verified that under/var/log i have a folder for esxi01 host and it has all the log files from my host

[root@syslogsrv esxi01]# pwd
/var/log/esxi01
[root@syslogsrv esxi01]# ls
crond.log Fdm.log Hostd.log hostd-probe.log lwsmd.log mark.log Rhttpproxy.log shell.log smartd.log syslog.log vmkernel.log vobd.log Vpxa.log

[root@syslogsrv esxi01]# pwd

/var/log/esxi01

[root@syslogsrv esxi01]# ls

crond.log Fdm.log Hostd.log hostd-probe.log lwsmd.log mark.log Rhttpproxy.log shell.log smartd.log syslog.log vmkernel.log vobd.log Vpxa.log

Configuring syslog settings via Web-Client

To configure syslog on Esxi host via Web-Client, navigate to Host and Cluster and select Esxi host > Manage > Settings > Advanced System Settings and in search box type syslog.global and verify value for Syslog.global.loghost

Also you can define following parameters:

Syslog.global.logDirUnique: True/False. Set to true will create individual directories per host (w/ host name) in the specified folder.
Syslog.global.defaultRotate: Max number of logs to keep locally. Does not effect the remote log server retention.
Syslog.global.defaultSize: Max size (in KB) of each log file before it is rotated or rolled over.

If the value is empty, Edit the settings by clicking on the pencil icon and add the entry as shown below:

Open firewall posts for syslog server by Navigating to Security Profile and click on Edit button

Look for “syslog” rule and if the check box is not selected then select it and click “OK”

Under outgoing connections, verify syslog is now listing. If not then refresh web-client to see entry appears there or not.

Under Services, verify syslog server setting is listed as Running

Also verify syslog service startup policy is set to “Start and stop with host” if you want service to be persistent across host reboot.

Configuring Syslog via vSphere Thick Client

Select Esxi host and go to Configuration tab and select Advanced Settings. Locate Syslog and expand the settings and under global configuration verify if there is any entry for Syslog.global.logHost.

If there are no settings present, enter a new one by adding udp://syslog-IP:514 and hit OK.

Under firewall settings, select the syslog service and hit OK.

Under Service Properties, verify Syslog Server is Running. if not select the service and click on Options button and set service policy to “Start and Stop with host” and start service.

Configuring syslog on Esxi hosts using PowerCli

1: Set the syslog server on all Esxi host using command

get-vmhost| Set-VMHostAdvancedConfiguration -NameValue @{'Config.HostAgent.log.level'='info';'Vpx.Vpxa.config.log.level'='info';'Syslog.global.logHost'='udp://syslogsrv.alex.local:514'}

1	get-vmhost\| Set-VMHostAdvancedConfiguration -NameValue @{'Config.HostAgent.log.level'='info';'Vpx.Vpxa.config.log.level'='info';'Syslog.global.logHost'='udp://syslogsrv.alex.local:514'}

2: Open the firewall ports

get-vmhost| Get-VMHostFirewallException |?{$_.Name -eq 'syslog'} | Set-VMHostFirewallException -Enabled:$true

1	get-vmhost\| Get-VMHostFirewallException \|?{$_.Name -eq 'syslog'} \| Set-VMHostFirewallException -Enabled:$true

3: Restart syslog service on all hosts

$ESXhost=get-vmhost

Get-VMHost -name $ESXhost | Get-VMHostService | where {$_.key -eq 'vmsyslogd'} | Start-VMHostService -Confirm:$start

$ESXhost=get-vmhost

Get-VMHost -name $ESXhost | Get-VMHostService | where {$_.key -eq 'vmsyslogd'} | Start-VMHostService -Confirm:$start

4: Confirm Esxi hosts syslog settings

get-vmhost | Get-AdvancedSetting -Name Syslog.global.logHost

1	get-vmhost \| Get-AdvancedSetting -Name Syslog.global.logHost

Setting syslog via Host profile

Edit your Host Profile and expand Advanced Configuration Settings profile and locate Syslog.global.logHost setting. This is by default set to “Set the option to its default value”

Change the option to “Configure a fixed option” and provide the value of the option in formal protocol://syslogsrv-ip:port. Typically this is udp://syslogsrv-ip:514

Expand Security and Services profile and locate syslog. Checkmark the boxes under Ruleset.

Expand Service Configuration profile and add a sub-profile to it. Set service name as “vmsyslogd” and set service configuration to start and stop with host.

Save the host profile and navigate to Host and Cluster, select the cluster and check Esxi hosts for compliance. You will see your hosts as non-compliant if syslog server is not configured on hosts.

As expected one of my host was non-complaint with profile and was complaining about syslog settings not present on host.

I placed the host in MM and remediated it and things were back to green. Hosts were happy,cluster was also happy and compliant.

And that’s it. We have learnt 5 different ways in which hosts can be configured to send logs to a centralized syslog server. Happy syslogging!!!!

I hope you enjoyed reading this post. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Configure Centralized Logging on ESXi 6 Hosts

Like this:

Related

Leave a ReplyCancel reply

Spread the Love

Like this:

Related

Leave a ReplyCancel reply