About 3 months ago I got a visit from one of our windows server team members “umm… can we sniff a VM?”.
Although we had never done it before, about 3 hours later both teams were analyzing the packet traces. We had discussed our Nexus 1000v’s SPAN port capabilities when we were deploying the 1000v to our environment, but had yet to put together a plan to implement a virtual sniffing infrastructure.
Having the 1000v in place gave us a few options – Configure ERSPAN and send to an external source, build a virtual sniffer and collect from a SPAN port on the 1000v or sniff externally to the blade enclosure which contained our ESX hosts. As I discussed with the server team, this wouldn’t be the only or last time we would get a trace request. It would be worth the time to setup a virtual sniffer and get the process in place, and have the ability to sniff VM to VM traffic, even on the same host.
Building the Virtual Sniffer
Our team already uses an array of self-built sniffing boxes on our non-VM network. These are typically workstation class machines running a Linux server build. The thought of deploying a VM running the same O/S was sort of a joint-epiphany between our server/network teams, just as we were running through the details our plans. We already had a particular flavor of Linux that worked well for us (SLES), and a detailed sniffer build plan. Why not just apply this to a VM?
- You will want to consider the amount of traffic you plan to capture when you are sizing your VM. Consider a high I/O storage platform for your main volume to write to and perhaps a larger but slower volume for storing previous traces.
The VM sniffer would utilize a 1000v port-profile to get a vethernet port on the switch. When configuring the SPAN/Mirroring session, there is a requirement of having the source (VM to be sniffed) and destination (sniffer’s port) be on the same “line card”. This just means you need to deploy the sniffer on the same ESX host the VM to be sniffed currently resides on. You may want to watch or edit your DRS rules to keep the VM from moving during this time. Our plan was to sniff all traffic coming and going from the server in question- but you could just sniff an entire vlan or uplink port-channel.
Configuring the monitor session on the 1000v
After the sniffer was built, it was time to go through the 1000v SPAN port configuration. First was logging into the 1000v switch and finding out what Vethernet port the server to be analyzed is using. Also needed was the recently created Vethernet port the sniffer was using.
N1000v# sh int status module 3 -------------------------------------------------------------------------------- Port Name Status Vlan Duplex Speed Type -------------------------------------------------------------------------------- Veth11 ServerName, Network up 749 auto auto -- Veth22 vSniffer, Network up 749 auto auto --
Veth11 will be the Source and Veth22 will be the destination for our SPAN configuration. Enter into configuration mode to enter the commands
N1000v# conf t N1000v(config)# monitor session 1 N1000v(config-monitor)# description sniff ServerName N1000v(config-monitor)# source interface vethernet 11 both N1000v(config-monitor)# destination interface vethernet 22 N1000v(config-monitor)# no shut N1000v(config-monitor)# end
After your configuration is done, show your monitor session to confirm its configured correctly. You should do a configuration save afterwards
N1000v# show monitor session 1 session 1 --------------- description : sniff ServerName type : local state : up source intf : rx : Veth11 tx : Veth11 both : Veth11 source VLANs : rx : tx : both : source port-profile : rx : tx : both : filter VLANs : filter not specified destination ports : Veth22 destination port-profile : N1000v# copy r s [########################################] 100%
It’s time for the sniffer!
After the sniffer was deployed, it was time to begin collecting network traffic from the VM. From a terminal session, I entered a simple TCPDump command, specifying the particular NIC that is being used as the destination port in the previously configured above. Here is the example I used…
this particular command uses certain flags, which you can customize to fit what works for you – but some examples:
- -ni eth1 – which happens to be my destination port from 1000v
- -XX includes link-level header
- -s0 sets a default size of 65535 bytes for compatibility
- -C sets a 20MB limit for individual files
- -W limits the total number of files to be created
- -w writes to files in the path specified
Analyze and cleanup tasks
Once you have collected your trace data, it’s time to analyze it, and there are plenty of resources on the internet for reference. After you are done, you will want to go back and shut down and delete the monitor session on the 1000v.
N1000v# conf t N1000v(config)# monitor session 1 N1000v(config-monitor)# shut N1000v(config-monitor)# exit N1000v(config)# no monitor session 1 N1000v# show monitor session all Note: There are no sessions configured N1000v# copy r s [########################################] 100%
Final Notes
If another VM needs to be sniffed in the future, you can simply vMotion the virtual sniffer to that particular ESX host, and begin the 1000v configuration again.
If you don’t have 1000v deployed, there are some options, but it looks like it will take vSphere 5 to have a good solution in place. Thanks for reading my first post, and I am interested from hearing from you about your virtual environment sniffing experiences so far.
DataCenterHawk 