Troubleshooting a vSphere Storage vMotion PBM Timeout

The Error I Started With

I ran into this issue while migrating a VM and moving its storage at the same time. The vSphere Client returned:

A general system error occurred:
PBM error occurred during PreMigrateCheckCallback:
Operation timed out

My first thought was that I might be dealing with a datastore or Storage vMotion problem, but the PBM reference pushed me toward Storage Policy Based Management and the vCenter Storage Policy Service, vmware-sps.

My first lesson: a PBM migration error does not necessarily mean the destination datastore is the problem. In my case, the failure started elsewhere in the management environment.

The Hidden Root Cause: My DNS VM Had Gone Offline

The reason VCSA could not resolve my ESXi hosts turned out to be straightforward once I found it: the storage backing my DNS VM had become disconnected. That took the DNS VM offline, so the VCSA suddenly lost DNS resolution for the ESXi hosts.

Because my hosts were registered and referenced by FQDN, SPS depended on DNS to resolve names such as esxi-01.home.local. Once DNS disappeared, SPS could no longer complete several host-provider and certificate-related operations.

The important distinction: there was a storage failure involved, but it was not the datastore I was migrating the workload to. The disconnected storage was underneath the DNS VM that my VMware management plane depended on.

Following the PBM Error Into SPS

I checked the SPS log on the VCSA and immediately found the first concrete clue:

Could not get host address for host esxi-02.home.local
Could not get host address for host esxi-01.home.local

java.net.UnknownHostException: esxi-01.home.local

SPS was also marking the host-side provider endpoints as offline:

https://esxi-01.home.local:9080/version.xml is offline
https://esxi-02.home.local:9080/version.xml is offline
https://esxi-04.home.local:9080/version.xml is offline

The timing was useful too. Certificate provisioning operations were taking roughly 30–40 seconds before failing, which matched the timeout behavior I was seeing in the PBM migration pre-check.

Commands I used

service-control --status vmware-sps

grep -iE "error|exception|timeout|fail|pbm|vasa" \
  /var/log/vmware/vmware-sps/sps.log | tail -100

Restoring Name Resolution on the VCSA

Since the DNS VM itself was unavailable, I needed a fast way to restore hostname resolution while troubleshooting. VCSA uses the standard Linux /etc/hosts file, so I added static mappings for the ESXi management addresses.

vi /etc/hosts

10.20.10.7   esxi-01.home.local esxi-01
10.20.10.71  esxi-02.home.local esxi-02
10.20.10.72  esxi-03.home.local esxi-03
# esxi-04 added with its management IP as well

I validated the system resolver with:

getent hosts esxi-01.home.local
getent hosts esxi-02.home.local
getent hosts esxi-03.home.local
getent hosts esxi-04.home.local

For example, esxi-01.home.local now returned:

10.20.10.7      esxi-01.home.local esxi-01

I treated /etc/hosts as a recovery and troubleshooting measure. For a permanent configuration, I want working forward and reverse DNS once the DNS VM and its storage are healthy again.

The Error Changed — Which Was Progress

After I restored local hostname resolution, the original UnknownHostException messages disappeared. SPS then progressed to a different error:

Retrieved vasa version is null.
Hostname verification could not be done.
provider: https://esxi-01.home.local:9080/version.xml is offline

I considered that progress because SPS was no longer dying at the DNS lookup stage. It was now getting further into the host provider workflow.

Checking the ESXi IOFilter Provider on TCP 9080

On esxi-01, I confirmed that TCP 9080 was actively listening and was owned by ioFilterVPServer:

0.0.0.0:9080   0.0.0.0:0   LISTEN   ioFilterVPServer

I checked the ESXi firewall as well. The iofiltervp ruleset was enabled and allowed all source IPs:

Ruleset     Allowed IP Addresses
----------  --------------------
iofiltervp  All

Most importantly, I could see multiple established TCP/9080 sessions from the VCSA IP, 10.20.10.10, to 10.20.10.7:9080. That allowed me to rule out the ESXi firewall and basic TCP reachability as the remaining cause.

Validating the ESXi Certificate and Hostname

I then inspected the ESXi machine certificate:

openssl x509 -in /etc/vmware/ssl/rui.crt \
  -noout -subject -issuer -dates

openssl x509 -in /etc/vmware/ssl/rui.crt \
  -noout -text | grep -A2 "Subject Alternative Name"

The certificate identity matched the host:

CN = esxi-01.home.local

X509v3 Subject Alternative Name:
    DNS:esxi-01.home.local

I also confirmed that ESXi itself had the same FQDN configured:

Fully Qualified Domain Name: esxi-01.home.local
Host Name: esxi-01
Domain Name: home.local

Finally, from the VCSA I connected to the port 9080 TLS endpoint with OpenSSL and received the expected VMCA-issued certificate:

openssl s_client \
  -connect 10.20.10.7:9080 \
  -servername esxi-01.home.local \
  </dev/null

Restarting SPS After Fixing the Real Problem

Restarting SPS while name resolution was still broken did not solve anything, because the service simply hit the same hostname-resolution failure again. Once I had restored resolution, I restarted SPS:

service-control --restart vmware-sps

This time the log changed completely. SPS successfully requested VMCA-signed client certificates for the ESXi hosts, installed them, and added them to the SMS store:

Requesting VMCA Signed cert for esxi-01.home.local having IP 10.20.10.7

Installed CA signed certificate at VVOLD
Added CA signed client cert to SMS Store

provisionEsxClientCertificates, Time taken: 301 ms

Previously, the workflow had spent tens of seconds before failing. After I restored name resolution, the provisioning step completed in only a few hundred milliseconds.

SPS output showing successful certificate provisioning and normal provider queries — My SPS log after the fix: host certificate provisioning completed successfully and provider queries returned normally.

The Storage Providers Came Back Online

I checked vCenter → Configure → Storage Providers. All four ESXi IOFilter providers showed Online, and the VMware vSAN provider was online as well. The IOFilter providers were reporting VASA API version 1.5.

vSphere Client showing all ESXi IOFilter Storage Providers online — The recovered state in vCenter: every ESXi IOFilter provider was online again.

The Migration Started Moving Again

The best confirmation was the migration itself. It had been sitting around 35% for a while. After I restored host resolution and restarted SPS, it started moving again and reached 37% shortly afterward.

At that point, I was confident the PBM timeout was a downstream effect of the DNS outage and stale SPS/VASA provider state, rather than a failure of the Storage vMotion destination.

My Root-Cause Chain

What failed firstThe datastore backing my DNS VM became disconnected, taking DNS offline.

What vCenter experiencedVCSA/SPS could no longer resolve the ESXi FQDNs, so provider and certificate operations failed.

What I sawThe migration failed or stalled with a PBM PreMigrateCheckCallback timeout.

The complete chain looked like this:

DNS VM datastore disconnected
        ↓
DNS VM unavailable
        ↓
VCSA could not resolve ESXi FQDNs
        ↓
SPS could not complete IOFilter/VASA provider operations
        ↓
Provider certificate provisioning failed
        ↓
PBM pre-migration validation timed out
        ↓
Storage vMotion stalled

How I Resolved It

I followed the PBM error into the SPS log and found UnknownHostException errors for the ESXi hosts.

I identified why DNS was unavailable: the storage backing my DNS VM had become disconnected.

I restored temporary host resolution on the VCSA using /etc/hosts.

I validated TCP 9080 and the IOFilter provider, including the ESXi firewall rules.

I validated the ESXi certificate and FQDN to rule out a hostname mismatch.

I restarted vmware-sps only after name resolution was working.

I confirmed all Storage Providers were Online and watched the migration resume.

Commands I Would Keep Handy

Check and restart SPS

service-control --status vmware-sps
service-control --restart vmware-sps

Search SPS for useful failures

grep -iE "error|exception|timeout|fail|pbm|vasa|hostname" \
  /var/log/vmware/vmware-sps/sps.log | tail -100

Test host resolution from VCSA

getent hosts esxi-01.home.local
getent hosts esxi-02.home.local
getent hosts esxi-03.home.local
getent hosts esxi-04.home.local

Check the IOFilter provider on ESXi

esxcli network ip connection list | grep 9080

esxcli network firewall ruleset rule list | grep -A10 -B2 iofiltervp

esxcli network firewall ruleset allowedip list -r iofiltervp

Inspect the ESXi certificate

openssl x509 -in /etc/vmware/ssl/rui.crt \
  -noout -subject -issuer -dates

openssl x509 -in /etc/vmware/ssl/rui.crt \
  -noout -text | grep -A2 "Subject Alternative Name"

What I Would Do Differently Next Time

The biggest takeaway for me is that DNS is part of the VMware management plane even when the DNS server itself is just another VM. If that VM depends on storage that can disappear, a storage incident can surface as what looks like an unrelated vCenter or PBM problem.

In a production environment, I would make sure management DNS is resilient, that forward and reverse DNS records are healthy for every ESXi host and vCenter appliance, and that infrastructure services such as DNS are not dependent on a single fragile storage path.

I would still use /etc/hosts as an emergency troubleshooting tool, but not as the long-term substitute for working DNS.

Final Takeaway

If I see this again:

PBM error occurred during PreMigrateCheckCallback:
Operation timed out

I will not jump directly to the destination datastore. I will check SPS early, look for hostname and provider errors, and confirm DNS availability before spending time on the Storage vMotion data path.

For this incident, my troubleshooting path was: PBM → SPS → ESXi provider communication → DNS → restore resolution → restart SPS → verify providers online.