VMware: Odd issue with NFS datastores

I’ve noticed something odd with a number of clusters that I administer that have NFS datastores. One of the datastores will be duplicated, and end with (1). Typically, only one host has this duplicated datastore. At first I thought it happened because the datastore was mounted twice, once by IP address, and again by hostname/FQDN.

So I painstakingly wrote this PowerCLI script that would look for all datastores with a parenthesis in its name. 😉

[powershell]
Get-Datastore * | where {$_.Name -like "*(*"}
[/powershell]

That returned this list to me.

[powershell]
Name FreeSpaceMB CapacityMB
—- ———– ———-
ESXi_CLUST_01_OS1 (1) 150422 163840
ESXi_CLUST_03_OS3 (1) 81806 81920
ESXi_CLUST_12_OS1 (1) 139271 163840
ESXi_CLUST_15_OS2 (1) 157634 163840
ESXi_CLUST_14_OS2 (1) 142356 163840
[/powershell]

From there, I went to the Datastores view within the vSphere Client (CTRL+SHIFT+D) and found the offending datastore. I switched to the Hosts tab for that specific datastore, and saw that only one host was mapped to the datastore. In addition, the duplicate datastore had a different location on the Summary tab. For example, the duplicate one was listed as having a FQDN location. Other NFS datastores in the cluster all show the IP address in the Location field.

The duplicated datastore is particularly interesting for the following reasons:

1. All of the effected clusters were configured by a PowerCLI script after initial install and connection to vCenter.
2. I know for a fact that all of the NFS datastores were specified by FQDN during scripted setup (because I wrote the script, and did a large number of the deployments).
3. By using the host->UUID->datastore script that I wrote, I’ve confirmed that the original datastore and duplicate datastore have the same UUID. This is significant, in my mind, because VMware KB1005930 specifically states that the NFS datastore hash and resulting UUID are based upon the NFS information you’ve provided. Meaning that I’d expect an NFS datastore provided by IP and one provided by FQDN would have different hashes.
4. By enabling Remote Tech Support mode, I was able to SSH into the host. Executing a ls -al /vmfs/volumes only showed the NFS datastores that were supposed to be there. There was no (1) datastore. Running the command esxcfg-nas -l didn’t show the (1) datastore, either.
5. Attempting to Unmount the datastore from vCenter throws an error. “The object has already been deleted or has not been completely created“. This error is thrown whether the host is in Maintenance Mode or not.

By now, I think you’d agree that this error/issue is perplexing and frustrating. Fortunately, there’s an easy (but potentially time-consuming) way to fix it.

1. Go back to the Datastores view and find the duplicate datastore. Click the Virtual Machines tab on the right side and perform an svMotion/cold migrate to another datastore that all the hosts in the cluster can see.
2. After all VMs on that datastore have been moved, click the Hosts tab, double-click the host, then the Virtual Machines tab and vMotion all of the VMs off the host with the duplicate datastore.
3. Put the host into Maintenance Mode and reboot. After rebooting, the (1) datastore should be gone.
4. If you’re not running DRS, be sure to re-balance the VMs as necessary across the cluster.