I’ve run into an issue lately in my VMware environment that really frustrated me because I didn’t understand why it was happening, and because there wasn’t an obvious way of determining that it wouldn’t happen again. The problem I’d been having was mismatched UUIDs on NFS datastores on clustered hosts. It’s a configuration issue that isn’t apparent until you need to vMotion a VM or are relying on HA/DRS.
If you aren’t already aware, VMware references its datastores through a generated UUID, and those UUIDs are used in .vmx and .vmdk files. If a host references a datastore using a UUID that is different from the UUID the other hosts in the cluster are using, then vMotion and HA/DRS will fail to move the effected VMs when necessary. The mismatch occurs when NFS datastores are not uniformly referenced by their IP address, host name, or fully qualified domain name throughout the cluster.
From VMware KB Article 1005930:
A UUID mismatch between two datastores occurs because the UUID is based on a hash of the NFS Server and Path, as seen by running esxcfg-nas. If you have specified the NFS server information using different methods on different hosts, then the hash value, and ultimately the UUIDs will be different.
Some other symptoms of this problem are referenced in VMware KB articles 1006052, and 1023230.
While the solutions referenced in the KB articles above work well, it can be rather klunky to have a number of SSH windows open, and visually comparing the UUIDs. I decided to write this bit of PowerCLI to assist in clearly identifying the host and datastore that is using a mismatched UUID.
[powershell]
$results = @()
$vmhosts = Get-VMHost -Location vSphereCluster
foreach ($vmhost in $vmhosts) {
$datastores = Get-Datastore -VMHost $vmhost | Where-Object {$_.Type -eq ‘NFS’}
foreach ($datastore in $datastores) {
$row = "" | Select-Object Host,Datastore,Path
for ($i = 0; $i -lt $datastore.ExtensionData.Host.Count; $i++) {
if (([string]$vmhost.Id) -contains ([string]$datastore.ExtensionData.Host[$i].Key)) {
$row.Path = $datastore.ExtensionData.Host[$i].MountInfo.Path
}
}
$row.Datastore = $datastore.Name
$row.Host = $vmhost
$results += $row
}
}
$results | Sort-Object Datastore,Host
[/powershell]
Line 1: Creates an array to hold the results
Line 2: Retrieves all of the hosts in a cluster
Line 3: Begins looping through each host in the cluster
Line 4: Retrieves all of the NFS datastores on a host
Line 5-15: Loops through each datastore and extracts the specific host’s path to that datastore. It then adds that row to the results array.
Line 17: Displays the results array, sorted by Datastore name and Host.
Notes: You will, of course, need to be connected to a vCenter server. This script was written and tested with PowerCLi 4.1 U1.
[…] Update: I had to remove the last / from the Path variable. If you do not remove the last / the script will mount the NFS share but with a new UUID. See Damian Karlson his post about this subject here. […]