virtizik.com
VCF 9.1 / ESXi Upgrade Troubleshooting

ESXi 9.1 Upgraded Successfully — But vCenter Said “Not Responding”

During the ESXi portion of my VVF 8.0 U3 to VCF 9.1 upgrade, one host completed the 9.1 image update and came back on the network, but vCenter could no longer manage it. The fix turned out to be much smaller than the symptoms suggested.

Keywords: VVF 8.0 U3, VMware Cloud Foundation 9.1, ESXi 9.1, vLCM, vCenter, vpxa, hostd, Not Responding, NSX

The situation

I was in the ESXi upgrade phase of a brownfield transition from VVF 8.0 U3 to VMware Cloud Foundation 9.1. vCenter had already been upgraded and the initial NSX upgrade had completed. I then moved on to the ESXi hosts using the VCF-managed lifecycle workflow and the VCF 9.1 BOM image.

esxi-04.home.local was the first host to complete its ESXi upgrade. When it came back, however, vCenter showed the host as Not Responding.

The confusing part was that the host itself was clearly alive. I could still log directly into the Host Client at 192.168.1.73. More importantly, the Host Client's Networking view appeared to show no VMkernel adapters at all.

vCenter showing esxi-04 as Not Responding while the ESXi Host Client remains reachable after an ESXi 9.1 upgrade
After the ESXi 9.1 upgrade, vCenter reported esxi-04.home.local as Not Responding even though the host itself remained reachable through the Host Client.
This looked worse than it really was.
The vSphere Client suggested a host/networking failure, but I still had direct management access. That made it worth validating the ESXi networking state from the CLI before rebuilding anything or rolling back the upgrade.

First check: did the ESXi 9.1 upgrade actually succeed?

I SSH'd to the host and checked the installed ESXi version:

[root@esxi-04:~] vmware -vl
VMware ESXi 9.1.0 build-25370933
VMware ESXi 9.0 Update 1

That immediately told me something important: this was not a failed image installation. The host had successfully booted the ESXi 9.1 build from the VCF 9.1 BOM.

Upgrade stateESXi 9.1.0 build 25370933 was active.
Direct accessThe Host Client and SSH were both reachable.
vCenter stateThe host remained Not Responding from the management plane.

The VMkernel adapters were actually still there

Because the Host Client showed an empty VMkernel list, I checked the real host configuration from the CLI:

esxcli network ip interface list
esxcfg-vmknic -l

The result was reassuring. All of the expected VMkernel interfaces were still present:

vmk0  192.168.1.73   defaultTcpipStack
vmk1  10.20.10.73    defaultTcpipStack
vmk2  10.20.11.104   vmotion
vmk10 10.20.20.101   vxlan
vmk50 169.254.1.1    hyperbus

They were also still connected to the same distributed switch, DVS-B-MGMT. That told me the empty Host Client page was not evidence that the VMkernel configuration had been erased.

Do not rebuild networking based on the UI alone.
In this case the CLI proved that the DVS and VMkernel configuration had survived the upgrade intact.

I also verified the physical NIC and driver

The management path for this lab depends on an Intel I226-V 2.5 GbE adapter, so I checked whether the NIC had survived the jump to ESXi 9.1 and whether the driver was loaded:

[root@esxi-04:~] esxcli network nic list
Name    Driver    Admin  Link  Speed
vmnic0  cndi_igc  Up     Up    2500
vmnic1  i40en     Up     Up   10000
vmnic2  i40en     Up     Down     0

The ESXi 9.1 image contained the expected cndi-igc VIB:

cndi-igc  1.2.12.0-1vmw.910.0.25370933  VMwareCertified

At that point I had eliminated the most disruptive possibilities: the host had upgraded successfully, the physical NIC was online, the driver was present, the DVS existed, and the VMkernel interfaces were still configured.

The clue was vpxa health

I then checked the vCenter agent health directly:

localcli system health report get -r vmw.vpxaStatus

The output showed:

VC_server_IP: null
connected_to_hostd: false
last_heartbeat_to_VC: null
managed_by_VC: false
responsive: false

This lined up almost perfectly with what vCenter was showing. The host itself was healthy, but the vCenter agent was not currently participating in the normal management relationship.

I checked the service state and both agents were technically running:

/etc/init.d/hostd status
/etc/init.d/vpxa status

hostd is running.
vpxa is running.
Running did not mean healthy.
vpxa had a process, but its health state showed that it was not connected to hostd and was not managed by vCenter.

The fix: restart the management agents

I first restarted vpxa, but the health status did not immediately recover. I then restarted both management agents:

/etc/init.d/hostd restart
/etc/init.d/vpxa restart

The vpxa.log then showed the important transition:

Vmomi service started
[VpxaModulesStart] DONE
Version negotiation; server selected: TCP:localhost:443
[Release: '9.1.0.0']
Started heartbeating..

That was the point where the management plane recovered. I did not have to remove the host from inventory, rebuild the DVS configuration, roll ESXi back, or reboot the host a second time.

vCenter reconnected the host automatically.
After hostd and vpxa restarted cleanly and the heartbeat resumed, esxi-04.home.local returned to a connected state on its own.

Why I did not roll back ESXi

The screen initially made a rollback tempting. A freshly upgraded ESXi host was Not Responding and the Host Client appeared to show no VMkernel adapters. But the CLI evidence told a different story.

1
ESXi 9.1 was actually running.
The host had booted build 25370933 successfully.
2
The NICs and drivers were healthy.
The I226-V was online at 2.5 Gb/s using cndi_igc.
3
The VMkernel configuration survived.
vmk0, vmk1, vmk2, vmk10, and vmk50 all existed.
4
The DVS was still intact.
The interfaces remained attached to DVS-B-MGMT.
5
The management-agent state was the outlier.
vpxaStatus showed the missing hostd/vCenter relationship.

My recovery checklist for the remaining hosts

If another host behaves the same way during the ESXi 9.1 portion of the VCF upgrade, this is the sequence I would use before doing anything invasive:

vmware -vl
esxcli network ip interface list
esxcfg-vmknic -l
esxcli network nic list
esxcli network vswitch dvs vmware list
localcli system health report get -r vmw.vpxaStatus
/etc/init.d/hostd status
/etc/init.d/vpxa status

If the host is reachable, networking is intact, and the problem is isolated to the management agents, my next step is:

/etc/init.d/hostd restart
/etc/init.d/vpxa restart

Then I give vCenter a short window to reconnect the host before attempting anything more disruptive.

Lessons learned

1
“Not Responding” does not always mean the ESXi host is down.
Direct Host Client and SSH access can quickly separate a host failure from a management-agent failure.
2
Trust the CLI when the UI looks incomplete.
The Host Client showed no VMkernel adapters, while the host CLI showed every expected interface still configured.
3
Validate the image before blaming the upgrade.
The host had correctly booted the ESXi 9.1 BOM build and the required NIC driver was present.
4
Check vpxaStatus before rebuilding anything.
It immediately exposed that the vCenter management relationship was the broken layer.
5
A management-agent restart can be enough.
In my case, restarting hostd and vpxa restored heartbeats and vCenter reconnected the host automatically.

This was a good reminder that an upgrade problem can look like a networking problem even when the data plane is perfectly healthy. The safest troubleshooting path was to prove each layer before changing it.