Troubleshooting Esxi host Disconnection from vCenter issue

By | 22/03/2015

Last week I was facing a serious issue in my home lab where my esxi host is getting disconnected from my vCenter Server randomly. Whenever I am doing any configuration changes like enabling ssh or creating a new vSwitch the host got disconnected immediately. I was damn frustrated and was looking for a solution because it was very hard for me to work.

So I started troubleshooting by going through my vCenter log files and found following:

So I guess something wrong was happening related to heartbeat exchange between my host and vCenter server. I started my troubleshooting by following below steps:

1: Checked whether Esxi is able to reach my vCenter server or not by pinging and doing a telnet from Esxi host to vCenter Server on port 902

Note: Telnet command wont work in Esxi so you have to use “nc -z” command

Res-1

Res-2

So as you can see I  was able to reach my vCenter from my Esxi host successfully.

2: Next I checked whether or not my Esxi host is listening on port 902 (heartbeat port)

Res-3

The above command verified yes my host is listening on port 902

4: I added the host disconnection timeout string in Advance Settings of vCenter and increased the value to 120

Res-7

I verified once again that value has been added.

Res-8

4: Next I check my vCenter Server for “Managed IP Setting”. Sometimes if the vCenter IP is not listed then also you can face this issue.

Res-5

In my case I manually entered IP under Run Time Settings as shown in above image.

5: I checked the same settings on my Esxi host.

Res-4

So from above image it is pretty clear that my Esxi host is configured to managed by correct vCenter server.

6: Next I checked for Heartbeat Port Value on my Esxi host by running the command:

# grep -i serverport /etc/vmware/vpxa/vpxa.cfg

The output which I got was something strange as my Esxi host was using port 922 for heartbeats exchange instead of using default port 902.

According to VMware KB Article 2040630

This issue is caused by dropped, blocked, or lost heartbeat packets between the vCenter Server and the ESXi/ESX host. If there is an incorrect configuration of the vCenter Server managed IP address, the host receives the heartbeat from vCenter Server but cannot return it.

It is important to remember that the default heartbeat port is UDP 902, and these packets must be sent between vCenter Server and the ESXi/ESX host for the host to stay connected and remain in the vCenter Server inventory.

Res-9

I changed the port to 902 by editing the vpxa.cfg file and removed and added back my Esxi host to vCenter Server and hoped that my issue is now resolved. But surprisingly I was still getting the disconnection problem. Once again I connected my Esxi host using ssh and checked vpxa.cfg file and found the port has been again changed to 922. This was strange.

On digging more I found that this is happening because of heartbeat port specified as 922 in the registry key of vCenter server. I got this clue from one of the issue 2437489 posted in VMware Community group.

The full registry key is :

HKEY_LOCAL_MACHINE\SOFTWARE\VMware, Inc.\VMware VirtualCenter

Res-10

As you can see in above image the heartbeat port is 922 which is causing all the troubles. I changed it to 902 and restarted my vCenter Server and bingo my issue is resolved.

Hit Like and share on social media if above information is helpful to you. Happy Learning!!!

Category: VMware

About Alex Hunt

Hi All I am Manish Jha. I am currently working in OVH US as Operations Support Engineer (vCloud Air Operations). I have around 7 Years of IT experience and have exposure on VMware vSphere, vCloud Director,vSphere Replication, vRealize Automation, NSX and RHEL. If you find any post informational to you please press like and share it across social media and leave your comments if you want to discuss further on any post. Disclaimer: All the information on this website is published in good faith and for general information purpose only. I don’t make any warranties about the completeness, reliability and accuracy of this information. Any action you take upon the information you find on this blog is strictly at your own risk. The Views and opinions published on this blog are my own and not the opinions of my employer or any of the vendors of the product discussed.

10 thoughts on “Troubleshooting Esxi host Disconnection from vCenter issue

  1. Sri

    Hey Alex, That’s a Good one you’ve found out, So While troubleshooting we’ve to check listening port after your Step-2 and the rest troubleshooting part would have been easier, This was much useful for me!!

  2. santosh

    I faced a same issue.by seeing dis post i resolved the issue.Thanks for the post bhai.Keep going