VMware Cloud Foundation Troubleshooting

Building a VCF 9.1 Manual Depot: How-To, Pro Tips, and Lessons Learned

How I manually staged VCF 9.1 binaries, fixed certificate and tool problems, and learned why a successful metadata upload can still produce exactly zero usable components.

Environment: VCF 9.1, VCF Installer, VCF Download Tool, manual depot, productVersionCatalog, NFS staging

What I was trying to accomplish

I needed to populate the software depot on my VCF Installer appliance, but I did not have a usable Broadcom download activation code or download token for the automated workflow. I could still obtain the entitled files manually from the Broadcom Support Portal, so I built a local depot hierarchy and used the VCF Download Tool only for the final upload and indexing operation.

The important distinction: acquiring binaries from Broadcom and uploading already-downloaded binaries to my local Installer were two different operations. Automated Internet downloads require the appropriate entitlement and authentication. The local binaries upload step authenticated to my VCF Installer with admin@local; it did not use an activation-code argument.

What looked like a simple copy-and-upload process turned into a useful lesson about VCF 9.x metadata, component build matching, certificate identity, staging cleanup, and the architectural changes introduced in VCF 9.1.

GoalManually populate the VCF Installer with catalog-recognized VCF 9.1 installation binaries.
Staging root/nfs/vmware/vcf/nfs-mount/manual-depot-910
Incoming root/nfs/vmware/vcf/nfs-mount/incoming-910

The earlier warning signs

I had already used the manual-depot pattern with VCF 9.0.2. That earlier run showed how the Binary Management page could contain a mixture of successful components and components that were still missing or unresolved.

VCF Installer Binary Management showing a partially downloaded VCF 9.0.2 repository
My earlier VCF 9.0.2 Binary Management view: SDDC Manager, NSX, and vCenter were recognized, while Operations-related components were still incomplete. This became my reference point for the 9.1 workflow.

The deployment validator made the missing pieces explicit. In that run, the Operations Fleet Management and Operations Collector files could not be retrieved.

VCF Installer validation failure for Operations fleet management and collector files
An earlier validation failure identifying unresolved Operations Fleet Management and Collector artifacts. VCF 9.1 changes this component model, but the lesson remains: a file on disk is not the same as a catalog-recognized binary.
VCF 9.1 architecture pro tip: I stopped looking for a standalone Fleet Manager OVA. In VCF 9.1, Fleet functions move into VCF Operations and the Kubernetes-based management-services platform. The catalog instead includes components such as VCF_FLEET_LCM, VCF_SDDC_LCM, VSP, and DEPOT_SERVICE.

Step 1: Create clean staging locations

I created separate locations for incoming files and the structured depot. Keeping them separate let me inspect and classify downloads before moving them under PROD.

DEPOT=/nfs/vmware/vcf/nfs-mount/manual-depot-910
INCOMING=/nfs/vmware/vcf/nfs-mount/incoming-910

mkdir -p "$DEPOT" "$INCOMING"
chown vcf:vcf "$INCOMING"
chmod 755 "$DEPOT" "$INCOMING"

I also kept a master copy outside the active depot. The upload process can clean staged binary files after a successful run, so I treated manual-depot-910 as working space rather than my only copy.

Pro tip: never use the active VCFDT upload directory as the only copy of files that took hours to download. Keep a separate master repository or patch-hold location.

Step 2: Start with signed metadata

The metadata package became the source of truth for the rest of the work. I downloaded and extracted:

vcf-9.1.0.0100-offline-depot-metadata.zip

I extracted it so that PROD was directly under my depot root—without an accidental extra directory level:

unzip \
  "$INCOMING/vcf-9.1.0.0100-offline-depot-metadata.zip" \
  -d "$DEPOT"

find "$DEPOT/PROD" -maxdepth 5 -type f | sort

The important metadata included the unified release manifest, product version catalog and signature, compatibility data, and vSAN HCL data. I verified the catalog explicitly:

VCF_META="$DEPOT/PROD/metadata"
PVC="$VCF_META/productVersionCatalog/v1/productVersionCatalog.json"

ls -lh "$PVC"
ls -lh "$VCF_META/productVersionCatalog/v1/productVersionCatalog.sig"
ls -lh "$VCF_META/manifest/v1/vcfManifest.json"
Why this matters: VCFDT does not accept a binary merely because its filename looks like VCF 9.1. The component, version, build, artifact name, and release metadata must line up with the signed catalog.

Compatibility data was a separate gotcha

I also ensured this file existed where the upload workflow expected it:

mkdir -p "$DEPOT/PROD/COMP/SDDC_MANAGER_VCF/Compatibility"

ls -lh \
  "$DEPOT/PROD/COMP/SDDC_MANAGER_VCF/Compatibility/VmwareCompatibilityData.json"

If VmwareCompatibilityData.json is missing, the metadata portion can fail even when the manifest and product catalog are present.

Step 3: Confirm the VCF Download Tool itself

One of my first attempts did not reach VCF at all. The tool failed before authentication because its bundled Java path was invalid:

VCF Download Tool failing because JAVA_HOME points to a missing bundled Java executable
The tool reported that JAVA_HOME was not defined correctly and could not execute its bundled Java binary.
Error: JAVA_HOME is not defined correctly.
We cannot execute /opt/vmware/vcf/lcm/vcf-download/jre/lin64/bin/java

I checked the configured value and whether the expected executable really existed:

echo "$JAVA_HOME"
ls -l /opt/vmware/vcf/lcm/vcf-download/jre/lin64/bin/java
find /opt/vmware/vcf/lcm/vcf-download -type f -path '*/bin/java'

The practical lesson was to use a complete VCFDT extraction/install appropriate for the target VCF release rather than creating a blind symlink to an arbitrary system Java. My later functional run used VCFDT 9.1.0.0400.25570101.

Pro tip: validate the tool version and bundled runtime before troubleshooting the depot. A broken VCFDT installation can make a perfectly structured repository look bad.

Step 4: Download the exact catalog builds

This was the most important lesson. I initially downloaded newer files from the portal and placed them in the expected component directories. The depot grew to 26 GB and contained legitimate VCF 9.1 artifacts—but they belonged to later patch levels than my signed metadata knew about.

ComponentWhat I staged initiallyBuild represented by my metadataResult
NSX9.1.0.0200.255241729.1.0.0.25318227 or catalog-listed 0100Ignored
vCenter9.1.0.0300.256295309.1.0.0.25370922 or catalog-listed 0100Ignored
VCF Operations9.1.0.0400.255415619.1.0.0.25346025 or catalog-listed 0100Ignored
Operations Cloud Proxy9.1.0.0400.255415629.1.0.0.25346033 or catalog-listed 0100Ignored

I proved the mismatch directly against the local metadata:

grep -RhoE '25629530|25524172|25541561|25541562' \
  "$VCF_META" | sort -u

The command returned nothing. By contrast, enumerating the 9.1 versions showed GA and 0100 builds:

grep -RhoE '9\.1\.0\.(0|0100|0200|0300|0400)\.[0-9]+' \
  "$VCF_META" | sort -Vu
What my catalog told me: it contained the VCF 9.1 GA family and 9.1.0.0100. It did not contain the staged 0200, 0300, or 0400 component builds.

I chose a consistent GA BOM for the initial deployment and planned to apply supported patches after the platform was operational. The key was consistency—not a universal rule that GA is always better.

Pro tip: “newest file visible in the portal” and “newest file recognized by my signed catalog” are not necessarily the same thing. Read the catalog first, then download.

Step 5: Build the VCF 9.1 component hierarchy

I created the component directories required by the 9.1 BOM:

COMP="$DEPOT/PROD/COMP"

mkdir -p "$COMP"/{\
DEPOT_SERVICE,\
ESX_HOST,\
NSX_T_MANAGER,\
SDDC_MANAGER_VCF,\
TELEMETRY_ACCEPTOR,\
VCENTER,\
VCF_FLEET_LCM,\
VCF_LICENSE_SERVER,\
VCF_OPS_CLOUD_PROXY,\
VCF_SALT,\
VCF_SALT_RAAS,\
VCF_SDDC_LCM,\
VCF_SERVICE_VCD_MIGRATION_BACKEND,\
VIDB,\
VRA,\
VROPS,\
VSP}

Then I moved files from incoming-910 into their catalog component directories without renaming or extracting the OVA, ISO, TAR, TGZ, or VLCP artifacts.

mv -v \
  "$INCOMING/VMware-VCSA-all-9.1.0.0.25370922.iso" \
  "$COMP/VCENTER/"

mv -v \
  "$INCOMING/nsx-unified-appliance-9.1.0.0.25318227.ova" \
  "$COMP/NSX_T_MANAGER/"

mv -v \
  "$INCOMING/Operations-Appliance-9.1.0.0.25346025.ova" \
  "$COMP/VROPS/"

mv -v \
  "$INCOMING/Operations-Cloud-Proxy-9.1.0.0.25346033.ova" \
  "$COMP/VCF_OPS_CLOUD_PROXY/"

New 9.1 management-services packages

The 9.1 repository also required the new services-runtime and lifecycle packages rather than a monolithic Fleet Manager OVA:

mv -v \
  "$INCOMING/vcf-services-platform-template-9.1.0.0.25370367.ova" \
  "$INCOMING/vmsp-platform-9.1.0.0.25370367.tar" \
  "$INCOMING/vmsp-plugin-9.1.0.0.25370367.tgz" \
  "$INCOMING/vmsp-cli-9.1.0.0.25370367.tar.gz" \
  "$COMP/VSP/"

mv -v "$INCOMING"/vcf-fleet-lcm-9.1.0.0.25371109.tgz \
      "$INCOMING"/vcf-fleet-lcm-plugin-9.1.0.0.25371109.tgz \
      "$COMP/VCF_FLEET_LCM/"

mv -v "$INCOMING"/vcf-sddc-lcm-9.1.0.0.25371107.tgz \
      "$INCOMING"/vcf-sddc-lcm-plugin-9.1.0.0.25371107.tgz \
      "$COMP/VCF_SDDC_LCM/"

mv -v "$INCOMING"/vcf-fleet-depot-9.1.0.0.25371105.tgz \
      "$INCOMING"/vcf-fleet-depot-plugin-9.1.0.0.25371105.tgz \
      "$COMP/DEPOT_SERVICE/"

Before moving anything with a wildcard, I previewed the match with ls -lh. That prevented one typo from moving unrelated patch files.

Step 6: Normalize ownership, permissions, and capacity

chown -R vcf_lcm:vcf "$DEPOT"

find "$DEPOT/PROD" -type d -exec chmod 755 {} +
find "$DEPOT/PROD" -type f -exec chmod 644 {} +

df -h /nfs/vmware/vcf/nfs-mount
df -i /nfs/vmware/vcf/nfs-mount

I verified the completed inventory and isolated large artifacts:

find "$COMP" -type f -printf '%P\n' | sort

find "$COMP" -type f -size +10M \
  -printf '%12s  %P\n' | sort -nr

For a full VCF 9.1 binary set, I wanted generous free capacity because the workflow could temporarily require both staged and imported copies. In my lab I targeted at least 150 GB free, with 250 GB providing more breathing room.

Step 7: Match the Installer certificate name exactly

My next failure looked like a credential problem:

Validating SDDC Manager credentials.
ERROR: Failed to create token. API status code 0. API response null

The real cause was TLS hostname verification. I had supplied vcf-i.home.local, but the certificate presented by the Installer contained vcf-i.

VCF Download Tool SSL failure showing vcf-i.home.local does not match the vcf-i certificate name
The log exposed the real cause: the FQDN I supplied was not present in the certificate. The certificate subject and alternate name used the short hostname vcf-i.

I inspected the certificate directly:

openssl s_client \
  -connect vcf-i:443 \
  -servername vcf-i \
  </dev/null 2>/dev/null |
openssl x509 -noout -subject -issuer -ext subjectAltName

Then I used the hostname actually represented by the certificate:

cd /opt/vmware/vcf/lcm/vcf-download/bin

./vcf-download-tool binaries upload \
  --depot-store="$DEPOT" \
  --sddc-manager-fqdn=vcf-i \
  --sddc-manager-user=admin@local
Result: the same admin@local credentials then validated successfully. Trusting a certificate and matching its identity are separate requirements.

Step 8: Do not mistake “0 elements” for success

After fixing authentication, the metadata upload completed cleanly:

SDDC Manager credentials validated successfully.
SDDC Manager version validated successfully.
Successfully uploaded vSAN HCL file.
Successfully uploaded compatibility data.
Successfully uploaded unified release manifest file.
Successfully uploaded product version catalog file.

Then the component table displayed:

---------------------------------------------------------------------------
ID | Component | Component Full Name | Version | Release Date | Size | Type
---------------------------------------------------------------------------
0 elements
---------------------------------------------------------------------------

0 SUCCESS | 0 NOT_REQUIRED | 0 FAILED

That was not a successful binary import. It was a successful metadata upload with zero catalog-matching component binaries.

My most useful lesson: if VCFDT reports 0 elements, stop. Do not assume the binary upload succeeded just because every summary counter is zero and the metadata lines say “Successfully uploaded.”

I checked what was actually present:

du -sh "$DEPOT/PROD"

find "$COMP" -type f -printf '%P\n' | sort

find "$COMP" -type f -size +10M \
  -printf '%s %P\n' | sort -nr
What I found: the repository was still 26 GB and the large files were still present. VCFDT had ignored them because they did not match the catalog—not because the directory was empty.

Step 9: Quarantine mismatched patch files

I did not delete the newer 0200, 0300, and 0400 files. They were valid files that could be useful later with matching metadata. I moved them outside PROD so the installation depot contained one unambiguous BOM.

PATCH_HOLD=/nfs/vmware/vcf/nfs-mount/patch-hold-910
mkdir -p "$PATCH_HOLD"/{VCENTER,NSX_T_MANAGER,VROPS,VCF_OPS_CLOUD_PROXY}

find "$COMP" -type f \
  \( -name '*9.1.0.0200*' -o \
     -name '*9.1.0.0300*' -o \
     -name '*9.1.0.0400*' \) \
  -printf '%p\n'

After verifying each source, I moved the mismatched artifacts to the corresponding patch-hold directories and replaced them with the exact GA files represented by my catalog.

My condensed runbook

1
Download signed metadata first.
Extract it so PROD is directly under the depot root.
2
Read the local product catalog.
Determine the exact release families, build numbers, filenames, and component mappings it recognizes.
3
Download exact artifacts.
Do not substitute newer patch files unless the signed metadata includes those builds.
4
Stage outside PROD first.
Use an incoming directory, validate names and sizes, and then move files into component directories.
5
Validate the VCFDT runtime.
Confirm the tool version and bundled Java path before debugging the repository.
6
Match the certificate SAN.
Use the exact short name, FQDN, or IP represented in the Installer certificate.
7
Upload and inspect the component table.
Metadata success plus 0 elements means the binary work is not finished.
8
Verify in Binary Management and validation.
Make sure the required components are recognized before continuing deployment.

Lessons learned and pro tips

Metadata controls realityA binary's marketing version is not enough. The signed catalog determines whether VCFDT recognizes it.
Do not mix release familiesA repository containing GA, 0200, 0300, and 0400 files can look complete while producing zero components.
Check the certificate, not your assumptionUse the name in the certificate SAN/CN. “Always use the FQDN” was not correct for my Installer certificate.
VCF 9.1 changed Fleet ManagementUse a VCFDT version that understands VSP, Fleet LCM, SDDC LCM, and Depot Service.
Keep a master copyThe active upload tree is disposable staging space. Preserve downloads elsewhere.
Read the counters carefully0 SUCCESS / 0 FAILED does not mean success; it means nothing was evaluated.

The biggest shift in my troubleshooting was moving from “the files are present” to “the files are represented by this exact catalog.” Once I made that distinction, the behavior stopped being mysterious. VCFDT was doing exactly what it was designed to do: uploading metadata, validating identity, and ignoring artifacts that did not belong to the signed BOM it had been given.

Official references

Always compare these steps with the documentation and release-specific metadata for the exact VCF version and patch level you are deploying.