Recovering an Interrupted VCF 9.1 Deployment After NFS Failure, Cloud Proxy Loss, and Stale Fleet State

The lab and the failure domain

This incident happened while I was building a second VMware Cloud Foundation 9.1 instance in a nested two-site lab. Site B already had a working VCF instance and hosted the shared VCF Operations/Fleet services. Site A was being deployed as a separate VCF instance and was joining the existing Operations/Fleet environment.

Existing Operations

vcf-ops.home.local · Site B

Existing Site-B Cloud Proxy

vrops-c-b01.home.local · 10.20.10.60

New Site-A Cloud Proxy

vrops-c-a01.home.local · 10.10.10.60

Site-A SDDC Manager

vcfa-sddc.home.local · 10.10.10.167

VCF Installer

vcf-installer.home.local · 192.168.1.190

Lab router / DNS

192.168.1.90

The deploy had already survived the normal sharp edges of running VCF inside nested ESXi: VLAN tagging, management-vmk migration, vSAN, NSX sizing, DNS, and service-runtime placement. Then an NFS/storage interruption destabilized the lab. VMs were restarted, NSX briefly became unavailable, and the in-flight VCF deployment was left in a partially completed state.

Table of contents

Before the storage interruption: nested VCF had already exposed several lab-specific traps

It is worth separating the storage incident from the earlier deployment problems. These were real blockers, but they had already been solved before the final Cloud Proxy/Fleet recovery.

Side incident: the existing Site-B Cloud Proxy would not boot

While the Site-A deployment was being recovered, the Cloud Proxy associated with the original VCF instance also failed after the storage disruption. VCF Operations showed the proxy offline.

The console made the cause much clearer:

[FAILED] Failed to start File System Check on /dev/disk/by-label/root
[DEPEND] Dependency failed for /sysroot
[DEPEND] Dependency failed for Initrd Root File System
[DEPEND] Dependency failed for Mounts Configured in the Real Root

Broadcom documents this class of Photon OS failure as filesystem corruption commonly following storage loss, power interruption, or an abrupt crash. The successful repair was to edit the GRUB boot entry, append fsck.repair=yes to the Linux line, and allow the appliance to repair the filesystem before continuing the boot.

# GRUB: append to the linux line
fsck.repair=yes

If the automatic repair had failed, the next-level recovery would have been single-user/read-only boot and a targeted e2fsck against the actual root/boot partitions. I did not need that fallback in this incident.

The Site-A Cloud Proxy was a different problem: the VM was gone, but Fleet still remembered it

The new Site-A proxy was not simply offline. After the host/storage disruption, the VM that had been deployed during the VCF workflow was no longer present. The VCF Installer still believed it was in the middle of Join the existing operations appliance and sat at 4/7 tasks complete.

At first glance this looked like an inactive Fleet Lifecycle task. It was not. The more important clue was inventory: Fleet still contained a Site-A Cloud Proxy node even though the actual VM no longer existed.

On the Fleet Build service, the stale node had the Site-A IP 10.10.10.60. Earlier inventory also showed a confusing duplicate-name condition where the Site-B and Site-A proxy objects could both surface the Site-A display name. That made it especially important to identify proxies by UUID, IP address, and actual hostname rather than by friendly name alone.

Proving what VCF Operations actually knew

The source of truth check was the CASA Cloud Proxy inventory on the primary VCF Operations node:

curl --insecure -s -u "admin" \
  https://localhost/casa/onprem/v1/cloudproxies \
  | python -m json.tool

After deleting the stale Site-A proxy object from Operations and synchronizing inventory, CASA returned only the legitimate Site-B proxy:

UUID:     <site-b-proxy-uuid>
IP:       10.20.10.60
hostname: vrops-c-b01.home.local
role:     ONE_WAY_REMOTE_COLLECTOR
type:     UNIFIED_CLOUD_PROXY

DNS was also clean in both directions:

vrops-c-b01.home.local  -> 10.20.10.60
10.20.10.60             -> vrops-c-b01.home.local

vrops-c-a01.home.local  -> 10.10.10.60
10.10.10.60             -> vrops-c-a01.home.local

Broadcom KB 447501 is useful here: deleting a Cloud Proxy in Operations does not necessarily remove its VM, and Fleet inventory can lag because of synchronization. The CASA endpoint is an excellent verification point.

Manually redeploying the Site-A Unified Cloud Proxy

Broadcom KB 438951 describes a closely related VCF 9.1 multi-instance failure: completing Fleet/Operations integration requires a healthy, configured Cloud Proxy, but a missing health precheck can let the workflow proceed too far. The documented remediation is to deploy a new Cloud Proxy from the VCF Operations UI.

I redeployed Site A with the intended identity:

FQDN:    vrops-c-a01.home.local
IP:      10.10.10.60
Gateway: 10.10.10.1
DNS:     192.168.1.90

At this point the new proxy existed and VCF Operations could see Site A — but the Installer did not advance.

The next layer: SDDC Manager still thought Operations and the Cloud Proxy were IN_PROGRESS

The Site-A SDDC Manager exposed the real inconsistency. Its vcf_management_component table showed the Services Runtime and SDDC Lifecycle as complete, but the existing Operations node and the new Cloud Proxy were still marked IN_PROGRESS:

SELECT component_type, fqdn, deployment_type, deployment_status
FROM vcf_management_component
ORDER BY component_type;

SDDC_LCM                    vcf-a-instance.home.local  NEW       SUCCEEDED
VCF_OPERATIONS_CLOUD_PROXY  vrops-c-a01.home.local    NEW       IN_PROGRESS
VCF_OPERATIONS:MASTER       vcf-ops.home.local         EXISTING  IN_PROGRESS
VSP                         vcf-a-msr.home.local       NEW       SUCCEEDED

The public API confirmed the same stale state:

GET /v1/vcf-management-components

vcfOperations.deploymentStatus          = IN_PROGRESS
vcfOperationsCollector.deploymentStatus = IN_PROGRESS
vspCluster.deploymentStatus              = SUCCEEDED
sddcLcm.deploymentStatus                 = SUCCEEDED

Meanwhile, /v1/vcf-management-components/tasks/latest did not return a populated active task. In other words, there was no real component task doing work anymore — only state left behind by the interrupted orchestration.

BEGIN;

UPDATE vcf_management_component
SET deployment_status = 'SUCCEEDED'
WHERE component_type IN (
    'VCF_OPERATIONS_CLOUD_PROXY',
    'VCF_OPERATIONS:MASTER'
)
AND deployment_status = 'IN_PROGRESS';

COMMIT;

The update changed exactly two rows, and the public API immediately reflected SUCCEEDED for both Operations and the Site-A collector. That confirmed the stale state had been in the database itself, not in a Commons Services cache.

The Installer still had a zombie parent workflow

Fixing component state did not automatically wake the top-level Installer execution. Restarting domainmanager on the VCF Installer caused it to clean up the old interrupted EXTEND_VCF execution, mark it failed, run the error handlers, and make the deployment retryable.

systemctl restart domainmanager
systemctl status domainmanager --no-pager

After the retry, the workflow finally started executing again — which was progress, because it exposed the next real problem instead of sitting indefinitely with no active subtasks.

An important lesson here was that a restart is not always “just a restart.” Domain Manager has startup cleanup behavior for interrupted orchestrator contexts. In this case that behavior was exactly what converted a zombie execution into a state the Installer could retry.

The retry exposed a networking regression: the Installer route was gone

The retried workflow immediately tried to collect the Site-A SDDC Manager SSH host key and timed out:

ssh: connect to host vcfa-sddc.home.local port 22: Connection timed out

DNS was fine:

getent hosts vcfa-sddc.home.local
10.10.10.167 vcfa-sddc.home.local

But the route was wrong:

ip route get 10.10.10.167

10.10.10.167 via 192.168.1.1 dev eth0 src 192.168.1.190

The VCF Installer had previously been given static routes through the lab router, but they were non-persistent and had disappeared. The correct next hop was 192.168.1.90:

ip route replace 10.10.0.0/16 via 192.168.1.90 dev eth0
ip route replace 10.20.0.0/16 via 192.168.1.90 dev eth0

After that change, the Installer could reach vcfa-sddc.home.local:22 and successfully collected the RSA/ECDSA SSH host keys. The ssh-dss and ssh-ed25519 negotiation messages in the debug log were harmless probes; supported keys had already been retrieved.

The next retry failed on admin@local — because I had changed it during troubleshooting

Once routing worked, the Installer reached the Site-A SDDC Manager API and failed for a completely different reason:

FAILED_TO_GET_SDDC_MANAGER_TOKEN

POST https://vcfa-sddc.home.local/v1/tokens
401 IDENTITY_UNAUTHORIZED_ENTITY
User is not authorized

Earlier in the troubleshooting I had reset admin@local directly on Site-A SDDC Manager to regain API access. That fixed my manual token test, but it created a mismatch: the VCF Installer still had the original generated password in its deployment specification.

Broadcom KB 403099 documents the local password storage and reset mechanism:

mkdir -p /etc/security/local
chown root:vcf_services /etc/security/local
chmod 650 /etc/security/local

touch /etc/security/local/.localuserpasswd
chown root:vcf_services /etc/security/local/.localuserpasswd
chmod 660 /etc/security/local/.localuserpasswd

printf '%s' "$PASSWORD" \
  | openssl dgst -sha512 -binary \
  | openssl enc -base64 \
  | tr -d '\n' \
  > /etc/security/local/.localuserpasswd

The fix was to put admin@local back to the original value shown by the VCF Installer under Review Passwords, so the SDDC Manager and the Installer agreed again.

Repeated bad token attempts then produced:

HTTP 403
LOCAL_ACCOUNT_LOCKED
Local account is locked.

That was another valuable clue: the password mismatch was no longer theoretical. The old workflow had been hammering the SDDC Manager token endpoint with a stale credential. After stopping the retries, restoring the original password, and clearing the lockout/cooldown, token generation succeeded and the Installer could continue.

Once the state, route, and credential all matched, VCF resumed normally

The visible change was immediate: Join the existing operations appliance went from 4/7 to 7/7, and VCF Management Services started deploying.

Telemetry was installed through Fleet Lifecycle and reached SUCCEEDED. The Installer then created a new Fleet LCM task for Salt, polled it normally, and continued.

The occasional getSystemInfo errors seen in the debug log were not the blocker at this stage. The code fell back to discovering the local VSP cluster from Site-A SDDC Manager, found vcf-a-msr.home.local and vcf-a-instance.home.local, discovered the shared fleet-lcm service from VCF Operations, exchanged a session token, and continued.

At 04:23 UTC the Installer patched the Site-A management-component record for Salt to SUCCEEDED and the parent orchestration reached its end state.

What I learned from this recovery

The recovery sequence that finally worked

Quick command reference

Cloud Proxy inventory on VCF Operations

curl --insecure -s -u "admin" \
  https://localhost/casa/onprem/v1/cloudproxies \
  | python -m json.tool

VCF management-component state on SDDC Manager

psql -h localhost -U postgres -d platform -c "
SELECT component_type, fqdn, deployment_type, deployment_status
FROM vcf_management_component
ORDER BY component_type;"

Create an SDDC Manager API token

curl -sk -X POST https://localhost/v1/tokens \
  -H 'Content-Type: application/json' \
  -d '{"username":"admin@local","password":"<password>"}'

Read management-component state

curl -sk \
  -H "Authorization: Bearer $TOKEN" \
  https://localhost/v1/vcf-management-components | jq

Restore lab routes on the VCF Installer

ip route replace 10.10.0.0/16 via 192.168.1.90 dev eth0
ip route replace 10.20.0.0/16 via 192.168.1.90 dev eth0

ip route get 10.10.10.167
nc -vz -w 3 10.10.10.167 22

Watch the Installer/Fleet transitions

LOG=/var/log/vmware/vcf/domainmanager/domainmanager.log

tail -f "$LOG" | \
grep --line-buffered -Ei \
'Fleet LCM Task ID|componentType|deploymentStatus|SUCCEEDED|FAILED|FAILURE|End of Orchestration'

Broadcom references that were useful

These were the most relevant vendor references for the individual failure modes. The selective SQL state change described above is intentionally not presented as a Broadcom-supported procedure.

• KB 439615 — Aria/VCF Operations boots into emergency mode after power/storage interruption
• KB 326392 — Manual filesystem check / emergency mode recovery for VCF Operations
• KB 326323 — Photon OS “Failed to start File System Check” recovery with fsck.repair=yes
• KB 438951 — Existing Operations/Fleet deployment can require a healthy Cloud Proxy; deploy a new proxy from Operations
• KB 447015 — Redeploy a Unified Cloud Proxy when its management relationship becomes invalid
• KB 447501 — Cloud Proxy deletion, Fleet synchronization, and CASA verification
• KB 443106 — VCF 9.1 instance integrations must use a healthy Cloud Proxy / collector group
• KB 403099 — Reset the SDDC Manager admin@local password
• KB 436034 — Stale Installer execution can repeatedly use an old password and lock admin@local
• KB 435569 — Troubleshooting Fleet/SDDC lifecycle services on the VCF Services Runtime

Full screenshot timeline

The main article uses the screenshots that best show the turning points. For completeness, the rest of the screenshots captured during this deployment troubleshooting thread are included below as a collapsible gallery.

Show all additional screenshots (33)

Final takeaway

The hardest part of this incident was that every fix revealed another valid failure underneath it. The Cloud Proxy really was missing. Fleet really did have stale inventory. SDDC Manager really did have orphaned IN_PROGRESS state. The Installer really had lost its route. And admin@local really was out of sync and eventually locked.

Once those layers were repaired in order, VCF did not need to be rebuilt from scratch. The same deployment execution recovered, completed the Operations join, installed the remaining management services through Fleet Lifecycle, and finished successfully.