virtizik.com
VMware VCF 9.1 / Cross-vCenter Troubleshooting

Cross-vCenter Relocation Fails with “Cannot Connect to Host”

How a powered-off VM exposed an NFC routing problem that looked like a vMotion failure

Keywords: VCF 9.1, ESXi 9.1, cross-vCenter migration, cold migration, NFC, TCP 902, HostConnectFault, VMkernel routing, vMotion, Provisioning

The Situation

I stood up a temporary vCenter on my standalone storage host so I could move a VM out of my VCF lab. The standalone ESXi host lived on the same physical lab network as the VCF hosts, but the networking was intentionally split:

VCF source host: esxi-01
  vmk0  192.168.1.70/24   lab / local management
  vmk1  10.20.10.7/24     VCF management
  vmk2  10.20.11.101/24   vMotion

Standalone destination: esxi-05
  vmk0  192.168.1.74/24   management
  vmk1  10.20.11.105/24   vMotion

vRouter:
  192.168.1.90
  10.20.10.1

The relocation wizard validated, the task started, and then the copy failed with:

Cannot connect to host.
vSphere task showing Relocate virtual machine failed with Cannot connect to host
The original cross-vCenter relocation failure: Cannot connect to host while copying VM files.

My First Suspect Was vMotion

Because this was a cross-vCenter relocation, my first assumption was that the standalone host simply could not reach the VCF vMotion network. The VCF hosts were using 10.20.11.0/24, while the standalone host initially had no matching vMotion VMkernel.

I added a vMotion VMkernel to esxi-05 as 10.20.11.105. That let me validate the dedicated vMotion path without changing the VCF hosts:

# esxi-01
vmkping -S vmotion -I vmk2 -d -s 1472 10.20.11.105

# esxi-05
vmkping -S vmotion -I vmk1 -d -s 1472 10.20.11.101

Both directions returned 0% packet loss at a full 1500-byte MTU. Both VMkernels were tagged VMotion, and both had a valid vmotion

At this point the Layer-3 vMotion path was healthy. The cross-vCenter relocation still failed anyway.
vSphere Client relocation failure with ESXi SSH packet capture troubleshooting windows
While the task continued to fail, I was validating the vMotion path and capturing traffic directly from both ESXi hosts.

The Important Detail I Almost Missed: The VM Was Powered Off

The VM I was moving, i-nfs, was powered off. That changed the data path completely.

A live migration would make TCP/8000 on the vMotion network a primary suspect. But this was a cold cross-vCenter relocation. The task itself was sitting at Copying Virtual Machine files, which meant I needed to look at NFC and the host management / provisioning path instead.

Live vMotionMemory and execution state normally travel over the vMotion VMkernel, commonly using TCP/8000.
Cold relocationVM file movement uses NFC and TCP/902.
My mistakeI had proven the vMotion network, but that was not the path currently failing.

hostd.log Showed the Real Source IP

The breakthrough came from /var/run/log/hostd.log on esxi-01. The NFC manager was explicit about the connection it was trying to build:

[NFC INFO]NfcEstablishAuthCnxToServer:
Connecting from client 10.20.10.7 to host 192.168.1.74 using port 902

CnxOpenTCPSocket:
Cannot connect to server 192.168.1.74:902: Connection timed out

[NFC ERROR]NfcNewAuthdConnectionEx:
Failed to connect to peer.

Unable to connect to NFC server:
Failed to connect to server 192.168.1.74:902

Copy operation failed with error:
vim.fault.HostConnectFault

That one line changed the entire problem:

10.20.10.7  ---- TCP/902 NFC ---->  192.168.1.74
   esxi-01                              esxi-05
   vmk1                                 vmk0

NFC was not sourcing the copy from 192.168.1.70. It was binding to the VCF management VMkernel at 10.20.10.7.

Why My TCP/902 Test Had Been Misleading

I had already tested this:

nc -zv 192.168.1.74 902

and it succeeded. The problem was that this test naturally used the host's directly connected 192.168.1.x path:

192.168.1.70  ---- TCP/902 ---->  192.168.1.74
      SUCCESS

That was not the source address NFC had selected. When I forced the source VMkernel instead, the real problem appeared immediately:

[root@esxi-01:~] vmkping -I vmk1 192.168.1.74
PING 192.168.1.74 (192.168.1.74): 56 data bytes
sendto() failed (Network is unreachable)
This was the smoking gun. The host could reach 192.168.1.74, but the specific VMkernel that NFC chose could not.

Why the vRouter Was Not Involved… Until It Was

Earlier in the troubleshooting, I checked the vRouter while testing 10.20.11.101 to 10.20.11.105. Nothing appeared on the router, which was correct: those vMotion IPs are on the same /24 and communicate directly at Layer 2.

The NFC path was different. It crossed two subnets:

10.20.10.7/24
     |
     |  gateway 10.20.10.1
     v
  vRouter
     ^
     |  192.168.1.90
     |
192.168.1.74/24

Once I knew the real NFC source address, the vRouter became part of the path.

The Fix: Add a Host Route on the Source VCF ESXi Host

I first added a return route on the standalone host so esxi-05 knew how to get back to the VCF management subnet:

[root@esxi-05:~] esxcli network ip route ipv4 add \
  --network 10.20.10.0/24 \
  --gateway 192.168.1.90

That gave esxi-05 this route:

10.20.10.0   255.255.255.0   192.168.1.90   vmk0

But the migration still could not work because the source NFC VMkernel itself had no path to 192.168.1.74.

The decisive change was on esxi-01:

[root@esxi-01:~] esxcli network ip route ipv4 add \
  --network 192.168.1.74/32 \
  --gateway 10.20.10.1

The resulting routing table was:

Network       Netmask          Gateway      Interface  Source
------------  ---------------  -----------  ---------  ------
default       0.0.0.0          192.168.1.1  vmk0       MANUAL
10.20.10.0    255.255.255.0    0.0.0.0      vmk1       MANUAL
192.168.1.0   255.255.255.0    0.0.0.0      vmk0       MANUAL
192.168.1.74  255.255.255.255  10.20.10.1   vmk1       MANUAL

The /32 route is the important part. It is more specific than the existing directly connected 192.168.1.0/24 route, so traffic for exactly 192.168.1.74 is forced out the VCF management path through 10.20.10.1.

The Test That Finally Passed

After adding the source route, the same forced-source VMkernel test that previously returned Network is unreachable worked immediately:

[root@esxi-01:~] vmkping -I vmk1 192.168.1.74
PING 192.168.1.74 (192.168.1.74): 56 data bytes
64 bytes from 192.168.1.74: icmp_seq=0 ttl=63 time=0.388 ms
64 bytes from 192.168.1.74: icmp_seq=1 ttl=63 time=0.449 ms
64 bytes from 192.168.1.74: icmp_seq=2 ttl=63 time=0.380 ms

--- 192.168.1.74 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss

The return path from the standalone host also worked:

[root@esxi-05:~] vmkping -I vmk0 10.20.10.7
64 bytes from 10.20.10.7: icmp_seq=0 ttl=63 time=0.316 ms
64 bytes from 10.20.10.7: icmp_seq=1 ttl=63 time=0.299 ms
64 bytes from 10.20.10.7: icmp_seq=2 ttl=63 time=0.312 ms

3 packets transmitted, 3 packets received, 0% packet loss
The migration immediately moved beyond the point where it had previously failed. I watched the relocation progress through 49% while it continued copying VM files.
vSphere relocation task at 49 percent copying virtual machine files
After the source /32 route was added, the relocation progressed beyond the previous failure point and continued copying VM files.

My Root-Cause Chain

What VMware selectedNFC bound to 10.20.10.7 on the VCF source host and targeted 192.168.1.74:902.
What was missingThe source VMkernel had no route from 10.20.10.7 to the standalone host.
What fixed itA specific 192.168.1.74/32 route through 10.20.10.1.
Powered-off cross-vCenter relocation
        ↓
VM file copy uses NFC / TCP 902
        ↓
NFC binds to source 10.20.10.7
        ↓
Destination is 192.168.1.74
        ↓
Source vmk1 has no route to 192.168.1.74
        ↓
hostd: CnxOpenTCPSocket timeout
        ↓
vim.fault.HostConnectFault
        ↓
vSphere: "Cannot connect to host"
        ↓
Add 192.168.1.74/32 via 10.20.10.1
        ↓
NFC path becomes routable
        ↓
Relocation proceeds

The Commands I Would Keep Handy

Validate the dedicated vMotion path

# Source
vmkping -S vmotion -I vmk2 -d -s 1472 10.20.11.105

# Destination
vmkping -S vmotion -I vmk1 -d -s 1472 10.20.11.101

Check which IP NFC is actually using

grep -iE \
'NFC ERROR|NFC INFO|NfcEstablishAuthCnxToServer|CnxOpenTCPSocket|HostConnectFault|902' \
/var/run/log/hostd.log | tail -200

Test from a specific source VMkernel

vmkping -I vmk1 192.168.1.74

Add the temporary source host route

esxcli network ip route ipv4 add \
  --network 192.168.1.74/32 \
  --gateway 10.20.10.1

Add the standalone return route

esxcli network ip route ipv4 add \
  --network 10.20.10.0/24 \
  --gateway 192.168.1.90

Verify the final routing tables

esxcli network ip route ipv4 list

Lessons Learned

1
“Cannot connect to host” is too generic to trust by itself. I needed hostd.log to see the actual source and destination of the failed NFC connection.
2
A successful generic TCP/902 test was not enough. The manual test used 192.168.1.70, while NFC was binding to 10.20.10.7.
3
Power state matters. I initially chased TCP/8000 because I was thinking “vMotion,” but the powered-off VM was being copied through NFC.
4
Always test from the exact VMkernel VMware is using. vmkping -I vmk1 exposed the missing route immediately.
5
A narrow host route was safer for this lab move. I could make exactly one standalone destination reachable from the VCF management VMkernel without redesigning the VCF host networking.

What I Would Do in a Cleaner Long-Term Design

For this lab move, the host-specific route was exactly what I needed because I did not want to disturb the VCF networking just to evacuate a standalone storage VM.

For a repeatable or production design, I would avoid depending on an incidental management route. I would make sure the hosts have a deliberate, routable Provisioning/NFC path between environments, or use a dedicated Provisioning VMkernel where appropriate.

The key point is not “add this exact route everywhere.” The key point is to identify the source VMkernel selected by NFC and make sure that source has a valid bidirectional path to the destination host on TCP/902.

Final Takeaway

This looked like a vMotion problem because I was performing a cross-vCenter relocation and the two environments had different network layouts. I even found that TCP/8000 to the new standalone vMotion VMkernel was not behaving the way I expected.

But the VM was powered off, and the actual failure was much simpler: NFC was sourcing from 10.20.10.7, that VMkernel had no route to 192.168.1.74, and the copy timed out on TCP/902.

The fix was a specific source-host route:

192.168.1.74/32 via 10.20.10.1

Once that route existed, the same forced-source ping passed and the relocation immediately moved beyond the point where it had always failed.