Knowledge
- Recall vicfg-* commands related to listing storage configuration
- Recall vSphere 4 storage maximums
- Identify logs used to troubleshoot storage issues
- Describe the VMFS file system
Skills and Abilities
- Use vicfg-* and esxcli to troubleshoot multipathing and PSA-related issues
- Use vicfg-module to troubleshoot VMkernel storage module configurations
- Use vicfg-* and esxcli to troubleshoot iSCSI related issues
- Troubleshoot NFS mounting and permission issues
- Use esxtop/resxtop and vscsiStats to identify storage performance issues
- Configure and troubleshoot VMFS datastores using vmkfstools
- Troubleshoot snapshot and resignaturing issues
- Analyze log files to identify storage and multipathing problems
Tools
- ESX Configuration Guide
- ESXi Configuration Guide
- vSphere Command-Line Interface Installation and Scripting Guide
- Product Documentation
- vSphere Client
- vSphere CLI
- vicfg-*
- esxcli
- resxtop/esxtop
- vscsiStats
- vmkfstools
Notes
Recall vicfg-* commands related to listing storage configuration
Refer to the vSphere Command-Line Interface Reference for more information on the below commands.
vicfg-iscsi | vicfg-iscsi supports configuration and property retrieval for software or hardware iSCSI initiators. |
vicfg-mpath | The vicfg-mpath command supports listing information about Fibre Channel or iSCSI LUNs and changing a path’s state. |
vicfg-mpath35 | vicfg-mpath35 provides an interface to configure multipath settings for Fibre Channel or iSCSI LUNs on ESX/ESXi version 3.5 hosts. |
vicfg-nas | The vicfg-nas command manipulates NAS file systems associated with ESX/ESXi hosts. |
vicfg-rescan | Perform a rescan operation each time you reconfigure your storage setup |
vicfg-scsidevs | The vicfg-scsidevs command displays information about available LUNs on ESX/ESXi 4.x hosts. |
vicfg-volume | The vicfg-volume command supports resignaturing a snapshot volume and mounting and unmounting the volume. |
vicfg-module | The vicfg-module command supports setting and retrieving VMkernel module options. |
Recall vSphere 4 storage maximums
Refer to the storage section of the vSphere Configuration Maximums guide.
Use vicfg-* and esxcli to troubleshoot multipathing and PSA-related issues
Use the vicfg-mpath command to list information about FC or iSCSI luns. Refer to vSphere Command-Line Interface Reference for further detail and description of the command syntax.
You use the esxcli command to set path policy, such as Fixed, Fixed_AP, MRU, or RR
You can use the Mask_Path plugin to mask paths. Refer to VMware KB1009449 on masking a LUN from ESX and ESXi 4.0.
Use vicfg-module to troubleshoot VMkernel storage module configurations
The vicfg-module command supports setting and retrieving VMkernel module options. Refer to vSphere Command-Line Interface Reference for further detail and description of the command syntax.
Use vicfg-* and esxcli to troubleshoot iSCSI related issues
Refer to vSphere Command-Line Interface Reference for further detail and description of vicfg-isci syntax.
Troubleshoot NFS mounting and permission issues
Refer to vSphere Command-Line Interface Reference for further detail and description of the vicfg-nas syntax.
Use esxtop/resxtop and vscsiStats to identify storage performance issues
‘d’ for disk adapter view
‘u’ for disk device view
‘v’ for disk VM view
From Duncan Epping’s ESXTOP section on his blog, the following counters are critical to troubleshooting performance specifically related to storage.
DISK | GAVG (H) | 25 | Look at “DAVG” and “KAVG” as the sum of both is GAVG. |
DISK | DAVG (H) | 25 | Disk latency most likely to be caused by array. |
DISK | KAVG (H) | 2 | Disk latency caused by the VMkernel, high KAVG usually means queuing. Check “QUED”. |
DISK | QUED (F) | 1 | Queue maxed out. Possibly queue depth set to low. Check with array vendor for optimal queue depth value. |
DISK | ABRTS/s (K) | 1 | Aborts issued by guest (VM) because storage is not responding. For Windows VMs this happens after 60 seconds by default. Can be caused for instance when paths failed or array is not accepting any IO for whatever reason. |
DISK | RESETS/s (K) | 1 | The number of commands reset per second. |
Configure and troubleshoot VMFS datastores using vmkfstools
You can do a lot with this command so refer to vSphere Command-Line Interface Reference for further detail and description of the command syntax.
Troubleshoot snapshot and resignaturing issues
The vicfg-volume command supports resignaturing a snapshot volume and mounting and unmounting the volume. Refer to vSphere Command-Line Interface Reference for further detail and description of the command syntax. Check out this blog for more information and a discussion on using the vicfg-volume command.
Other relevant blogs and websites related to this section
- VMware Storage Resolution Paths
- http://www.virtualinsanity.com/index.php/2010/03/16/performance-troubleshooting-vmware-vSphere-storage/
Additional troubleshooting links:
- http://vSphere-land.com/tag/troubleshooting
- http://www.vcritical.com/2009/10/easy-recovery-from-a-full-vmware-esx-datastore/
- http://www.virtuallyghetto.com/2010/06/esxcli-part1-what-is-esxcli.html
- http://www.yellow-bricks.com/2009/03/18/iscsi-multipathing-with-esxcliexploring-the-next-version-of-esx/
- http://www.punchingclouds.com/?p=965
- http://professionalvmware.com/2010/02/manage-vmfs-file-systems-using-the-cli-vcdx-prep/