Knowledge
- Identify virtual switch entries in a Virtual Machine’s configuration file
- Identify virtual switch entries in the ESX/ESXi Host configuration file
- Identify CLI commands and tools used to troubleshoot vSphere networking configurations
- Identify logs used to troubleshoot network issues
Skills and Abilities
- Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations
- Utilize vicfg-* commands to troubleshoot ESX/ESXi network configurations
- Configure a network packet analyzer in a vSphere environment
- Troubleshoot Private VLANs
- Troubleshoot Service Console and VMkernel network configuration issues
- Troubleshoot DNS and routing related issues
- Use esxtop/resxtop to identify network performance problems
- Use CDP and/or network hints to identify connectivity issues
- Analyze troubleshooting data to determine if the root cause for a given network problem originates in the physical infrastructure or vSphere environment
Tools
- ESX Configuration Guide
- ESXi Configuration Guide
- vSphere Command-Line Interface Installation and Scripting Guide
- Product Documentation
- vSphere Client
- vSphere CLI
- o vicfg-*
- net-dvs
- resxtop/esxtop
Notes
Identify virtual switch entries in a Virtual Machine’s configuration file
- Best thing to do here is open up a vmx file and learn what is configured. Below is a trimmed down vmx from my lab with just the network setting showing.
virtualHW.version = “7?
ethernet0.present = “true”
ethernet0.wakeOnPcktRcv = “true”
ethernet0.networkName = “VM Network”
ethernet0.addressType = “vpx”
ethernet0.generatedAddress = “00:50:56:a4:52:92?ethernet1.present = “true”
ethernet1.virtualDev = “e1000?
ethernet1.networkName = “VM Network”
ethernet1.addressType = “vpx”
ethernet1.generatedAddress = “00:50:56:a4:34:04?
ethernet2.present = “true”
ethernet2.virtualDev = “e1000?
ethernet2.networkName = “VM Network”
ethernet2.addressType = “vpx”
ethernet2.generatedAddress = “00:50:56:a4:74:e9?ethernet0.startConnected = “true”
ethernet2.startConnected = “false”ethernet0.pciSlotNumber = “32?
ethernet1.pciSlotNumber = “33?
ethernet2.pciSlotNumber = “35?ethernet0.virtualDev = “e1000?
ethernet1.startConnected = “false”
Identify virtual switch entries in the ESX/ESXi Host configuration file
- load up /etc/vmware/esx.conf and check it out
Utilize net-dvs to troubleshoot vNetwork Distributed Switch configurations
- There is not a ton of information out there on using the net-dvs command. One blog that contains some relevant information can be found at http://geeksilver.wordpress.com/2010/05/21/vds-vnetwork-distributed-switch-my-understanding-part-2/
- Something I did not know, this command is listed as an unsupported command. It will not run (to my knowledge) from the vMA and I ran it when locally logged into the host. The syntax of the command can be found below
Warning: This is an unsupported command. Use at your own risk.
net-dvs -a [ -P maxPorts] switch_name
net-dvs -d switch_name
net-dvs [ -A | -D ] -p port switch_name
net-dvs [ -s name=value | -u name ] -p port switch_name
net-dvs -l [ switch_name ]
net-dvs -i (init database)
net-dvs [-S | -R | -G ]
net-dvs -T
net-dvs -v “vlanID[;t|p[0-7][;min-max,min-max…]]
net-dvs -V “primaryVID,secondaryVID,i|c|p;primaryVID,secondaryVID,i|c|p…”
net-dvs -m “sid;dname;snaplen;[oiveld];encapvlan;wildcardsIn,wildcardsOut;dstPort1,dstPort2,…;srcInPort1,srcInport2,…;srcOutPort1,srcOutPort2,…;:sid2;dname2…”
net-dvs dvswitch -k “respool1_id;respool2_id;…”
net-dvs dvswitch -p dvport -K “respool1_id:shares:limit;respool2_id:shares:limit;…”
net-dvs dvswitch -p dvport -z “respool_id”
net-dvs dvswitch -j [activate|deactivate]
net-dvs -L uplink_name1[,uplink_name2,…] -t team_policy_type -p port switch_name
net-dvs dvswitch -H “red|yellow|green:some message” switch_name
net-dvs -o “depth,param|classname;depth,param|classname;… -p port|globalPropList switch_name
net-dvs –mtu mtu_value [-p dvport] switch_name
net-dvs –x 0|1 -p dvport switch_name
net-dvs –vlan vlanID -p dvport switch_name
net-dvs –reset -p dvport switch_name
net-dvs –cap cap_value -p dvport switch_name
net-dvs –states -p dvport switch_name
net-dvs –miscInfo ;# Dumps cpu/meminfo
net-dvs –vmknicIp <vmknic> ;# Displays IPv4 address on <vmknic>
Utilize vicfg-* commands to troubleshoot ESX/ESXi network configurations
- Below are the commands I’d consider relevant for troubleshooting in this section. You can use the vSphere Command Line Reference to gain more information on each of these commands and others.
- vicfg-authconfig(4.1 only) Manages Active Directory authentication.
- vicfg-dns.pl Specifies an ESX/ESXi host’s DNS (Domain Name Server) configuration.
- vicfg-ipsec Supports setup of IPSec.
- vicfg-nics Manages the ESX/ESXi host’s NICs (uplink adapters).
- vicfg-ntp Specifies the NTP (Network Time Protocol) server.
- vicfg-route Lists or changes the ESX/ESXi host’s route entry (IP gateway).
- vicfg-snmp Manages the Simple Network Management Protocol (SNMP) agent.
- vicfg-vmknic Adds, deletes, and modifies virtual network adapters (VMkernel NICs).
- vicfg-vswitch Adds or removes virtual switches or vNetwork Distributed Switches, or modifies switch settings.
Configure a network packet analyzer in a vSphere environment
- Too much to put in words on this one. Check out the blog below for assistance. I’d recommend using Wireshark as this is what was used in the troubleshooting course offered by VMware.
- http://itknowledgeexchange.techtarget.com/it-consultant/packet-sniffing-is-your-best-friend/
- http://www.petri.co.il/wireshark-ethereal.htm
Troubleshoot Private VLANs
- Great source of PVLAN information at http://professionalvmware.com/2010/04/private-vlan-resources/
- Free video (nearly 40 minutes!) detailing PVLAN’s from Eric Sloof at http://www.ntpro.nl/blog/archives/1465-Online-Training-Configure-Private-VLAN-IDs.html
- Complete definition of what is a PVLAN from VMware KB1010691
- How to configure PVLAN’s from VMware KB1010703
Troubleshoot Service Console and vmkernel network configuration issues
- Using VMware’s Resolution Paths a good starting point is the VMware KB1007986 for troubleshooting service console issues.
Troubleshoot DNS and routing related issues
- VMware KB4309499 is probably a good start for troubleshooting DNS/routing.
- To change/update DNS use the vicfg-dns command
Use esxtop/resxtop to identify network performance problems
- Run esxtop and hit ‘n’ to enter the networking view
- Again the best resource I’ve found so far on troubleshooting using esxtop as a whole is Duncan Epping’s Blog and I’ve included the two counters for networking in the table below.
- Two key performance counters you will need to know when troubleshooting network issues are below for both received and transmitted dropped packets. This goes without saying, but you are looking for no dropped packets here.
- The default view for networking will also show current and peak transmission stats to assist in your troubleshooting.
| NETWORK | %DRPTX | 1 | Dropped packages transmitted, hardware overworked. Possible cause: very high network utilization |
| NETWORK | %DRPRX | 1 | Dropped packages received, hardware overworked. Possible cause: very high network utilization |
Use CDP and/or network hints to identify connectivity issues
- VMware KB1007069 Cisco Discovery Protocol CDP Information via the ESX Command Line and Virtual Center (note replace vmware-vim-cmd with vim-cmd)
- The command vim-cmd hostsvc/net/query_networkhintwill query and show network hints