Sniffing VM traffic – using Nexus 1000v and a virtual sniffer

About 3 months ago I got a visit from one of our windows server team members “umm… can we sniff a VM?”.

Although we had never done it before, about 3 hours later both teams were analyzing the packet traces.   We had discussed our Nexus 1000v’s SPAN port capabilities when we were deploying the 1000v to our environment,  but had yet to put together a plan to implement a virtual sniffing infrastructure.

Having the 1000v in place gave us a few options – Configure ERSPAN and send to an external source, build a virtual sniffer and collect from a SPAN port on the 1000v or sniff externally to the blade enclosure which contained our ESX hosts.  As I discussed with the server team, this wouldn’t be the only or last time we would get a trace request.  It would be worth the time to setup a virtual sniffer and get the process in place, and have the ability to sniff VM to VM traffic, even on the same host.

Building the Virtual Sniffer

Our team already uses an array of self-built sniffing boxes on our non-VM network.  These are typically workstation class machines running a Linux server build.  The thought of deploying a VM running the same O/S was sort of a joint-epiphany between our server/network teams, just as we were running through the details our plans.  We already had a particular flavor of Linux that worked well for us (SLES), and a detailed sniffer build plan.  Why not just apply this to a VM?

  • You will want to consider the amount of traffic you plan to capture when you are sizing your VM.  Consider a high I/O storage platform for your main volume to write to and perhaps a larger but slower volume for storing previous traces.

The VM sniffer would utilize a 1000v port-profile to get a vethernet port on the switch.   When configuring the SPAN/Mirroring session, there is a requirement of having the source (VM to be sniffed) and destination (sniffer’s port) be on the same “line card”.  This just means you need to deploy the sniffer on the same ESX host  the VM to be sniffed currently resides on.  You may want to watch or edit your DRS rules to keep the VM from moving during this time.   Our plan was to sniff all traffic coming and going from the server in question- but you could just sniff an entire vlan or uplink port-channel.

Configuring the monitor session on the 1000v

After the sniffer was built, it was time to go through the 1000v SPAN port configuration.  First was logging into the 1000v switch and finding out what Vethernet port the server to be analyzed is using.  Also needed was the recently created Vethernet port the sniffer was using.

N1000v# sh int status module 3
Port           Name               Status    Vlan      Duplex  Speed   Type
Veth11         ServerName, Network  up       749       auto    auto    --
Veth22         vSniffer, Network    up       749       auto    auto    --

Veth11 will be the Source and Veth22 will be the destination for our SPAN configuration.  Enter into configuration mode to enter the commands

N1000v# conf t
N1000v(config)# monitor session 1
N1000v(config-monitor)# description sniff ServerName
N1000v(config-monitor)# source interface vethernet 11 both
N1000v(config-monitor)# destination interface vethernet 22
N1000v(config-monitor)# no shut
N1000v(config-monitor)# end

After your configuration is done,  show your monitor session to confirm its configured correctly.  You should do a configuration save afterwards

N1000v# show monitor session 1
   session 1
description       : sniff ServerName
type              : local
state             : up
source intf       :
    rx            : Veth11         
    tx            : Veth11         
    both          : Veth11         
source VLANs      :
    rx            :
    tx            :
    both          :
source port-profile :
    rx            :
    tx            :
    both          :
filter VLANs      : filter not specified
destination ports : Veth22
destination port-profile :
N1000v# copy r s
[########################################] 100%

It’s time for the sniffer!

After the sniffer was deployed, it was time to begin collecting network traffic from the VM.    From a terminal session, I entered a simple TCPDump command, specifying the particular NIC that is being used as the destination port in the previously configured above.   Here is the example I used…

this particular command uses certain flags, which you can customize to fit what works for you – but some examples:

  • -ni eth1 – which happens to be my destination port from 1000v
  • -XX includes link-level header
  • -s0 sets a default size of 65535 bytes for compatibility
  • -C sets a 20MB limit for individual files
  • -W limits the total number of files to be created
  • -w writes to files in the path specified

Analyze and cleanup tasks

Once you have collected your trace data, it’s time to analyze it, and there are plenty of resources on the internet for reference.   After you are done, you will want to go back and shut down and delete the monitor session on the 1000v.

N1000v# conf t
N1000v(config)# monitor session 1
N1000v(config-monitor)# shut
N1000v(config-monitor)# exit
N1000v(config)# no monitor session 1
N1000v# show monitor session all
Note: There are no sessions configured
N1000v# copy r s
[########################################] 100%

Final Notes

If another VM needs to be sniffed in the future, you can simply vMotion the virtual sniffer to that particular ESX host, and begin the 1000v configuration again.

If you don’t have 1000v deployed, there are some options, but it looks like it will take vSphere 5 to have a good solution in place.  Thanks for reading my first post, and I am interested from hearing from you about your virtual environment sniffing experiences so far.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s