Fixing VCF 9.1 Installer “General error” at Distributed Switch on Nested ESXi

The symptom: “General error” on Distributed Switch

I hit this while deploying another VMware Cloud Foundation instance in my nested lab. The first six Prepare steps completed, but step 7, Distributed Switch, stopped with an almost useless message:

General error:

The preconfigured profile area was blank and Retry simply returned the same error.

First problem found: hostname and certificate mismatch

The first real error in domainmanager.log was not networking at all. Several nested hosts had incorrect or incomplete hostnames, and their self-signed ESXi certificates had been generated with the wrong identity.

SSLPeerUnverifiedException:
Certificate for <esxi-a0X.home.local> doesn't match any subject alternative names: [localhost.localdomain]

ESXI_HOST_CERTIFICATE_CN_NOT_VALID.error

I corrected each host to the intended home.local domain, regenerated the ESXi certificates, rebooted the hosts, and then removed/re-added them in VCF so the new thumbprints could be accepted.

# Example hostname correction
esxcli system hostname set --host esxi-a01
esxcli system hostname set --domain home.local

# Regenerate the ESXi self-signed certificate
/sbin/generate-certificates
reboot

That fixed the certificate validation error, but the Distributed Switch step still failed. The new failure was much more interesting: intermittent read timeouts while VCF connected to https://esxi-a0X.home.local/sdk.

After the certificate fix: intermittent ESXi /sdk TLS timeouts

Domain Manager now resolved the correct hostnames and could reach TCP/443, but QuickStartNetworkProfileService would randomly fail against different hosts:

ConnectTimeoutException: Connect to esxi-a0X.home.local:443 failed: Read timed out

QuickStartNetworkProfileService Failed to fetch host info
PUBLIC_INTERNAL_SERVER_ERROR

The Java stack showed Domain Manager waiting inside BouncyCastle's TLS handshake:

org.bouncycastle.tls.RecordStream.readRecord
org.bouncycastle.tls.TlsProtocol.blockForHandshake
org.bouncycastle.tls.TlsClientProtocol.connect
java.net.SocketTimeoutException: Read timed out

I worked through the obvious suspects one by one. DNS was good. Routing was correct. The Debian router was forwarding the sessions. nftables was not blocking them. Conntrack was nowhere near full. TCP/443 established immediately. TLS 1.2 and TLS 1.3 both worked in serial tests. Envoy was nowhere near its external HTTPS connection limit.

CPU scheduling looked innocent

Because these are nested ESXi VMs, I also checked the outer ESXi host with esxtop. The physical host was lightly loaded, and the nested hosts showed tiny CPU Ready and Co-Stop values. Adding vCPUs would have treated the wrong layer.

The packet capture finally isolated the delay

The most useful test was capturing the actual VCF Retry on my Debian router. That removed the ambiguity between “the installer timed out” and “where did it actually wait?”

During a failing burst, VCF completed the TCP three-way handshake, sent its TLS ClientHello, and ESXi acknowledged the payload promptly. Then ESXi simply waited several seconds before emitting the TLS server flight.

# The pattern on the wire
VCF Installer  -- SYN --------------------------> ESXi
VCF Installer  <---------------------- SYN/ACK -- ESXi
VCF Installer  -- TLS ClientHello -------------> ESXi
VCF Installer  <--------------------------- ACK -- ESXi

                     ... 5 to 10 seconds ...

VCF Installer  <------------- TLS server flight -- ESXi

I also compared the two ClientHello sizes generated by different VCF connection paths. One was 353 bytes and another was 323 bytes. tshark showed both had the correct SNI, TLS versions, P-256 key share, groups, and signature algorithms. The 30-byte difference was simply 15 fewer two-byte cipher-suite entries. The larger ClientHello could stall too, so the cipher-list difference was not the cause.

The root cause: slow entropy on AMD Zen 5

At this point the symptom matched a very specific ESXi behavior: cryptographic work was waiting even though TCP and CPU scheduling were healthy.

The nested hosts were running on a Minisforum MS-A2 with an AMD Ryzen 9 9955HX, a Zen 5 processor. ESXi exposes a kernel setting named entropySources:

0 = defaults
1 = interrupts
2 = RDRAND
4 = RDSEED
8 = entropyd

With the default value of 0, ESXi can select RDSEED when the CPU supports it. William Lam documented a VMware Engineering finding that RDSEED entropy generation can be dramatically slower on AMD Zen 4, with similar behavior observed on Zen 5, and specifically notes that the issue also applies to nested ESX VMs on Zen 4/5 systems.

That fit my packet capture perfectly: ESXi received the ClientHello immediately, but the cryptographic server response was delayed for seconds.

The fix: force ESXi to use RDRAND

I changed one nested host first so I could do a clean A/B test:

esxcli system settings kernel set \
  -s entropySources \
  -v 2

reboot

After reboot, I verified that the configured and runtime values were both 2:

[root@esxi-a01:~] esxcli system settings kernel list -o entropySources
Name            Type    Configured  Runtime  Default
--------------  ------  ----------  -------  -------
entropySources  uint32  2           2        0

Then I hammered esxi-a01 from the VCF Installer with five rounds of four concurrent HTTPS requests — 20 TLS handshakes total.

for round in 1 2 3 4 5; do
  for i in 1 2 3 4; do
  (
    curl -ksS \
      --connect-timeout 10 \
      --max-time 15 \
      -o /dev/null \
      -w "$round-$i ssl=%{time_appconnect} total=%{time_total} code=%{http_code}\n" \
      https://esxi-a01.home.local/sdk
  ) &
  done
  wait
done

That was too large a change to dismiss as normal variance. I applied the same setting to esxi-a02, esxi-a03, and esxi-a04, rebooted them, and verified Runtime: 2 on all four hosts.

# Apply to each affected nested ESXi host
esxcli system settings kernel set -s entropySources -v 2
reboot

# Verify after reboot
esxcli system settings kernel list -o entropySources

VCF Installer moved past Distributed Switch

After all four nested hosts were using RDRAND, I retried the VCF workflow. This time the Prepare phase completed and the installer advanced to Validate & Deploy.

The complete troubleshooting path

Commands I would keep handy

# Check the ESXi entropy source
esxcli system settings kernel list -o entropySources

# Force RDRAND on an affected AMD Zen 4/5 nested ESXi host
esxcli system settings kernel set -s entropySources -v 2
reboot

# Confirm the setting is active after reboot
esxcli system settings kernel list -o entropySources

# Quick HTTPS/TLS timing test from the VCF Installer
curl -ksS \
  --connect-timeout 10 \
  --max-time 15 \
  -o /dev/null \
  -w 'code=%{http_code} connect=%{time_connect} ssl=%{time_appconnect} total=%{time_total}\n' \
  https://esxi-a01.home.local/sdk

# Capture VCF Installer -> ESXi HTTPS on the router
tcpdump -ni any -s0 -w /tmp/vcf-retry.pcap \
  'host 192.168.1.190 and net 10.10.10.0/24 and tcp port 443'

References

The final entropy lead matched existing VMware/Broadcom and community guidance:

The exact VCF Installer failure in this article was reproduced and isolated in my lab. The external references above support the ESXi entropy behavior and AMD Zen 4/5 workaround; they do not describe this exact VCF Distributed Switch UI symptom.

Takeaway

The VCF Installer's “General error” was several layers removed from the actual problem. The distributed switch page was only where the workflow surfaced the failure. The real sequence was:

VCF network-profile discovery
        -> ESXi /sdk TLS handshake
        -> ESXi waits on slow entropy generation
        -> Domain Manager hits its read timeout
        -> QuickStartNetworkProfileService fails
        -> UI shows "General error"

The packet capture was what broke the case open. Once I could see that ESXi had already received the ClientHello but was waiting seconds before generating the TLS response, the entropy path became a plausible target. Switching the nested hosts to RDRAND changed handshake time from seconds to milliseconds and the VCF deployment immediately moved forward.