Failed storage vmotion: when VMware’s log verbosity tells you Nothing At All

I recently was in the middle of migrating an entire cluster of VMs (190 in all) to a different datastore. 189 of them migrated just fine, but a single one gave me grief during the svmotion with the following error:

Relocate virtual machine
naughtyvm3161p
File /vmfs/volumes/f08ad362-ff1e8a5b/naughtyvm3161p/naughtyvm3161p-000001.vmdk was not found

This is especially curious, because when browsing the datastore, the vmdk certainly existed, albeit as naughtyvm3161p-000002.vmdk. Looking at all the config files (the vmx, the vmdk, et al) on the datastore with my favorite text editor, all of these reference the proper, existing 00002.vmdk file. The GUI, too, points to the proper file. So what on earth still thinks the VMDK was at its old name?

vpxa.log and hostd.log on the VM host were no help either – hostd.log just showed the error, and vpxa merely spat out a few thousand lines of it looking up the datastore with nothing meaningful about WHAT was referencing that file

2013-07-18T13:57:14.512Z [7B004B90 verbose 'Default' opID=AC56DD98-000446B3-3a-f0] [VpxaVmprovUtil] LocalPathToDatastoreUrl conversion: /vmfs/volumes/51e0170e-514c8539-b69a-68b599b15d64/naughtyvm3161p/naughtyvm3161p-000001.vmdk -> ds:///vmfs/volumes/51e0170e-514c8539-b69a-68b599b15d64/
2013-07-18T14:22:36.622Z [7AFA1B90 info 'DiskLib' opID=AC56DD98-00044873-1a-77] DISKLIB-DSCPTR: DescriptorDetermineType: failed to open '/vmfs/volumes/f08ad362-ff1e8a5b/naughtyvm3161p/naughtyvm3161p-000001.vmdk': Could not find the file (600000003)

Real helpful, as you can see. Peak snark was hit at around 9:57AM, when I realized I needed more coffee.

And then I noticed on a lark that this particular VM had a snapshot – based on the fact that it had a “delta” vmdk file. Why? No clue. None of these VMs should be using snapshots. After busting out the snapshot hatchet (the snatchet? the snapchet? idk) the storage vmotion was successful.

Goofy as hell, but that’s vmware for you.