VCF 9.1 Log Management Install Stuck: NFS RPC Limits, VMSP Storage, and a Missing VRLI Fleet Depot Bundle

Environment

This happened in my two-site nested VCF 9.1 lab while deploying native Log Management from VCF Operations. The relevant pieces were:

1. The visible symptom: Log Management never really got going

The first lifecycle attempt did not provide a useful root cause. VCF Operations showed the Log Management task in progress and even threw a 504 while trying to retrieve task details.

A later retry progressed far enough to show that the deployment was sitting at Stage VCF Services runtime.

At this point, repeatedly retrying the high-level install would only create more noise. I moved down into the VCF Management Services Kubernetes layer.

2. Follow the failure into VMSP

Run Kubernetes commands from the VCF Management Services control-plane node:

export KUBECONFIG=/etc/kubernetes/admin.conf

kubectl get pods -A | egrep -v 'Running|Completed'

kubectl -n vmsp-platform get events \
  --sort-by=.lastTimestamp | tail -80

The critical component was vmsp-operator. Its temporary 100 GiB generic ephemeral PVCs were being provisioned, attached, and then failing during the mount/format path. At different points the kubelet reported errors such as:

MountVolume.MountDevice failed ...
rpc error: code = DeadlineExceeded desc = context deadline exceeded

Inside the affected worker, the backing device showed XFS and SCSI I/O failures rather than an application-level Log Management error:

XFS (sdc): log recovery write I/O error ... error -5
XFS (sdc): failed to locate log tail
XFS (sdc): log mount/recovery failed: error -5
...
SCSI device reset on scsi0:2

That changed the question from “what is wrong with Log Management?” to “why can this VCFMS worker not reliably perform I/O to its CSI disk?”

3. Move one layer down: ESXi and NFS

The affected worker VM was on esxi-04, and the VCFMS CSI disk ultimately lived on my NFS-backed datastore. ESXi reported a 1 MiB maximum NFS read/write transfer size:

[root@esxi-04:~] esxcli storage nfs param get -v NFS-DS

Volume Name  MaxQueueDepth  MaxReadTransferSize  MaxWriteTransferSize  Vmknic  Connections
-----------  -------------  -------------------  --------------------  ------  -----------
NFS-DS          4294967295              1048576               1048576  None              1

The datastore itself still appeared mounted and accessible, which is exactly why this problem was easy to miss. A basic ping also looked perfect. But the NFS TCP connection to 10.20.10.5:2049 was repeatedly closing.

On the Debian NFS server, the kernel log finally exposed the smoking gun:

svc: svc_tcp_read_marker nfsd RPC fragment too large: 1048708
svc: svc_tcp_read_marker nfsd RPC fragment too large: 1048712

The payload ESXi wanted to write was 1 MiB (1048576 bytes). The slightly larger 1048708/1048712 values in the Linux error are the RPC record with protocol overhead included.

Packet capture confirmed the behavior

A packet capture showed ESXi beginning a 1 MiB NFS WRITE and the NFS server then resetting the TCP session:

NFS request ... write ... 1048576 (1048576) bytes @ 53687116800
10.20.10.5.2049 > 10.20.10.73.708: Flags [R]

That matched the Linux kernel errors exactly.

4. Root cause #1: Linux nfsd was capped at 256 KiB

The Debian NFS server's RPC maximum block size was only:

cat /proc/fs/nfsd/max_block_size
262144

Meanwhile ESXi was issuing 1 MiB NFSv3 writes. The practical failure chain was:

Linux nfsd max_block_size = 256 KiB
          +
ESXi NFSv3 WRITE payload = 1 MiB
          ↓
nfsd rejects the RPC record as "fragment too large"
          ↓
NFS server closes/resets TCP
          ↓
ESXi retries / NFS I/O stalls
          ↓
VMDK / First Class Disk I/O stalls
          ↓
guest XFS format/mount hangs or errors
          ↓
CSI NodeStageVolume times out
          ↓
vmsp-operator never becomes available
          ↓
VCF Services Runtime staging cannot progress

The runtime fix

I stopped NFS, raised the nfsd limit to 1 MiB, and restarted the service:

systemctl stop nfs-server

echo 1048576 > /proc/fs/nfsd/max_block_size
cat /proc/fs/nfsd/max_block_size

systemctl start nfs-server

Afterward:

cat /proc/fs/nfsd/max_block_size
1048576

The new oversized-RPC errors stopped, fresh VMSP volumes could be provisioned/attached/formatted, and a new vmsp-operator pod came up normally.

5. The deployment moved forward — and exposed a second, unrelated blocker

Once storage was healthy, VCF lifecycle work started running again. That exposed a completely different error in the component-action-stage-precheck workflow:

Checking reachability of
https://vcf-fleet.home.local/depot-service/content-gateway/PROD/COMP/VRLI/operations-logs-9.1.0.0100.25435121.tgz

ERROR ... URL not found ... HTTP status: 404
No certificate issues detected
Reachability check failed not due to a certificate issue

This was good news in a strange way: VMSP was now healthy enough to execute workflows. The new blocker was simply that Fleet Depot could not serve the required Log Management binary.

6. My local depot had the big TGZ — but the VRLI bundle was incomplete

The exact required file was:

operations-logs-9.1.0.0100.25435121.tgz

I manually downloaded it from the Broadcom Support Portal and staged it in my local VCF depot:

DEPOT=/nfs/vmware/vcf/nfs-mount/depot-910-0100
VRLI=$DEPOT/PROD/COMP/VRLI

mkdir -p "$VRLI"
mv /home/vcf/operations-logs-9.1.0.0100.25435121.tgz "$VRLI/"

chown -R vcf_lcm:vcf "$VRLI"
chmod -R 750 "$VRLI"

But VCFDT still reported:

Binaries to be exported:
...
0 elements

The reason became obvious when I checked the product version catalog. The INSTALL bundle for 9.1.0.0100.25435121 consists of four files, not just the 2.1 GB payload:

operations-logs-9.1.0.0100.25435121.tgz
configuration-schema-operations-logs-9.1.0.0100.25435121.yaml
depot-manifest-operations-logs-9.1.0.0100.25435121.yaml
operations-logs-plugin-9.1.0.0100.25435121.tgz

Once I manually downloaded the three small companion files and placed all four under PROD/COMP/VRLI, VCFDT recognized the component correctly.

Validate the exact files before uploading

cd "$VRLI"

cat <<'EOF' | sha256sum -c -
8dd597633452e52c7d6d4f27d99fd2a5f3595619bd855a2d3bae038a8d78f298  operations-logs-9.1.0.0100.25435121.tgz
9e81bb199002780a6ccf1870fe930ab47fc6e61eb545270211fbbd98f9cd6054  configuration-schema-operations-logs-9.1.0.0100.25435121.yaml
d2455d7f4bacbc60d94e6190bca51f5bffcda43a2494989489efb16280f271f4  depot-manifest-operations-logs-9.1.0.0100.25435121.yaml
ff493d5eb3a94b98cfe639ddd1018e7ea884dceebfd91840391059abd413eed4  operations-logs-plugin-9.1.0.0100.25435121.tgz
EOF

7. Upload VRLI into the VCF 9.1 Fleet Depot Service

With the local bundle complete, this was the working upload:

DEPOT=/nfs/vmware/vcf/nfs-mount/depot-910-0100

$VCFDT depot binaries upload \
  --ops-fqdn=vcf-ops.home.local \
  --ops-auth-source=LOCAL \
  --ops-user=admin \
  --ops-user-password-file=/root/vcf-ops-password.txt \
  --depot-fqdn=vcf-fleet.home.local \
  --vcf-version=9.1.0.0 \
  --depot-store="$DEPOT" \
  --sku=VCF \
  --type=INSTALL \
  --component=VRLI

The important change was immediate:

Binaries to be exported:
----------------------------------------------------------------------------------------------------------------------------------
ID                                   | Component | Component Full Name | Version             | Release Date | Size       | Type
----------------------------------------------------------------------------------------------------------------------------------
342a5203-bc8e-51cb-9f64-b29509a14dc0 | VRLI      | Log management      | 9.1.0.0100.25435121 | 06/05/2026   |    2.0 GiB | INSTALL
----------------------------------------------------------------------------------------------------------------------------------
1 element

Upload Progress: 100% (2.08 GB/2.08 GB)
Depot binaries upload task completed with status SUCCESSFUL

Two upload gotchas from this lab

Routing: the Installer initially sent 10.20.10.0/24 traffic toward the wrong gateway, so VCFDT reported a certificate import failure even though the real problem was a TCP timeout. In my lab I temporarily corrected the route with:

ip route add 10.20.10.0/24 via 192.168.1.90 dev eth0

Password file: VCFDT also needs a clean VCF Operations local-admin password file. The Suite API rejects passwords outside its input limits before authentication. A direct token request is a useful way to distinguish an API validation problem from bad credentials or routing.

8. Verify Fleet is actually serving the binary

From the VCF Installer, I tested the exact URL that the VMSP stage precheck had been failing against:

curl -skI \
'https://vcf-fleet.home.local/depot-service/content-gateway/PROD/COMP/VRLI/operations-logs-9.1.0.0100.25435121.tgz'

Now it returned:

HTTP/1.1 200 OK
content-type: application/octet-stream
content-length: 2170051429

The content-length matched the product catalog exactly.

9. The existing VCF workflow recovered automatically

I did not delete failed Argo pods or restart the whole deployment. The existing lifecycle workflow retried the stage precheck on its own:

Checking reachability of https://vcf-fleet.home.local/.../operations-logs-9.1.0.0100.25435121.tgz
URL is reachable
Successfully updated precheckMessages

Subsequent precheck workflows succeeded, the stage-component workflow completed, and the queued Log Management install work advanced behind the normal lock-ops-logs mutex.

NAME                                         STATUS
stage-component-ngea52ligbfjxn7u4lwy5v6ji4   Succeeded

...
WorkflowSucceeded   Workflow completed

10. What I would check first next time

If a VCF 9.1 Log Management deployment appears stuck around VCF Services Runtime staging, I would work down the stack in this order:

VCF Operations lifecycle task
        ↓
Argo / VMSP workflow state
        ↓
vmsp-operator availability
        ↓
PVC / vSphere CSI attach + mount events
        ↓
guest disk / filesystem I/O
        ↓
ESXi datastore + NFS/SAN path
        ↓
storage server logs / packet capture

Then, once VMSP is healthy:

stage-precheck workflow
        ↓
exact Fleet Depot URL
        ↓
local depot metadata + complete component bundle
        ↓
VCFDT Fleet Depot upload
        ↓
HTTP 200 from content-gateway

References

• Broadcom KB 442857 – VCF Download Tool missing ESXi and Day-N components — lists VRLI as the Log Management component supported by VCFDT.
• William Lam – VCF 9.1 side-loading VCF binaries into VCF Installer & Fleet Depot Service — documents the additional Fleet Depot upload requirement for disconnected/local depot environments and the depot binaries upload workflow.
• Broadcom KB 442141 – How to log into the Log Management UI in VCF 9.1 — useful architectural context for native, containerized Log Management in VCF Management Services.
• Broadcom KB 441054 – VCF Management Services deployment stuck — also reinforces that old failed lifecycle tasks should not be blindly restarted once newer workflow activity has completed.

Note: Hostnames, IP addresses, routes, and the Linux NFS configuration shown here are from my lab. Validate commands and supportability against your own VCF build, storage platform, and Broadcom documentation before applying them in production.