Search:  
Gentoo Wiki

Pfring

Contents

Introduction

The best description of PF_RING is the one given by the folks at ntop (PF_RING's developers) and is quoted here...

PF_RING is a new type of network socket that dramatically improves the packet capture speed, and that's characterized by the following properties:

  1. Available for Linux kernels 2.4.X and 2.6.X
  2. Device driver independent (best results can be achieved using network cards that support NAPI such as the Intel cards)
  3. Kernel-based packet capture and sampling.
  4. Libpcap support (see below) for seamless integration with existing pcap-based applications.
  5. Ability to work in transparent mode (i.e. the packets are also forwarded to upperlinks so existing applications will work as usual).(1)

Abstract

The biggest problem associated with packet capture is reducing the number of dropped packets. In networks where traffic is relatively low, packet loss may be trivial or even nonexistent, but when capturing packets in high speed network environments packet loss can be substantial. To understand packet loss we first have to understand what happens to a packet when it is received by a machine.

Under normal conditions, when a network interface card (NIC) sees a packet it determines if the destination MAC address in the packet matches its own MAC address or the link layer broadcast address. If it does not match, the packet is discarded. When a packet that does match is received by the machine's NIC the packet is copied into a buffer and, the NIC informs the kernel that it has received a packet that needs to be processed. The packet is then passed from the NIC to the kernel. The kernel process the packet by performing error checking and striping off layers of the packet as it passes through the kernel. The packet is then passed to the appropriate socket based on information from the packet header and queued for use by a userland application such as ssh or Apache. The application performs a system call to get the packet and then clears the queue.

In a situation where a user wants to capture all packets, regardless of its destination or correctness, the NIC needs to be placed in promiscuous mode. In essence, this tells the NIC to accept all packets whether they are destined for that specific NIC or not. Again, the packet is copied into a buffer and, the NIC informs the kernel that it has received a packet that needs to be processed. Once again, the packet is then passed from the NIC to the kernel. At this point something different needs to happen. Since the goal is to capture all packets as they were seen on the network, it is not desirable for the kernel to discard or strip off parts of the packets during its processing, so a copy of the packet is made. One copy is sent to the upper-levels of the kernel, the same as before, where most are discarded because they are not destined for this machine. The other unaltered packet is passes to the socket of a userland packet capture application and queued for use. In most instances this socket is created by the packet capture library libpcap. Libpcap provides a common API for packet capture by creating a virtual device for applications like tcpdump and snort to read the packet from. Libpcap is responsible for performing the system call to get the packet and clearing the queue.

Packet loss can occur in a couple of places. If the buffer used by the NIC to hold packets before being passed to the kernel is full then the NIC will drop the new packet. This can occur if the NIC is receiving packets at an extremely fast rate or if the kernel is too busy performing other tasks to receive the packet. Packets can also be lost if the socket queue is full when the kernel attempts to pass the copy of the packet to libpcap's socket. This can occur if the application using libpcap is unable to process packets fast enough due to lack of resources. In both cases system utilization can play a big part in packet loss. Ironically, placing the NIC in promiscuous mode to allow packet capture greatly increases the number of packets the kernel must handle and utilizes a lot more CPU cycles then normally would be required for network operations. This increases load which in-turn can reduce the ability of the system to capturing packets.

PF_RING helps reduce packet loss by providing a kernel module and modifications to libpcap that allows an application to create a socket designed for improved packet capture through the use of a ring buffer and shared memory. It also provides a way to prevent packets from being passed to the upper-levels of the kernel, thus reducing the amount of work the kernel needs to perform for each packet which in-turn improves packet capture performance. (2)

Goal

The goal of this document is to walk the reader though the steps required to build a PF_RING enabled kernel for use with the Gentoo Linux distribution. These steps require that you have Gentoo installed successfully, have root access and be comfortable with the process of compiling and installing a kernel.

Procedure

Code: Overview

In this document uses the following versions of libpfring and kernel sources...

net-libs/libpfring-3.2.1-r1
sys-kernel/gentoo-sources-2.6.19-r5

All commands are run as root so be careful. Here is an overview of the procedure required to create a PF_RING kernel...

  1. Get PF_RING
  2. Find a place to work
  3. Edit mkpatch.sh
  4. Create tarball of kernel sources
  5. Create patch file
  6. Make a copy of kernel sources and apply the patch
  7. Configure/install new kernel
  8. Reboot
  9. Verify that the module has loaded
  10. IMPORTANT INFO (Must Read!!)

Warning: The folks at ntop publicly distribute PF_RING via CVS. It is important you use the version in portage and not the version from their CVS.

If you are trying out these steps before the libpfring ebuild has been added to portage you can get the tarball at it's bug page Bug 167108 and skip one box down to "Find a place to work". All you need for this howto is the tarball.

Code: Get PF_RING
# emerge -a libpfring

If you have never followed these instructions before than this will fail with something like...

 * Checking for CONFIG_RING enabled kernel...
 * 
 * You don't have a PF_RING enabled kernel!
 * Please follow the directions at...
 * 
 * [[Pfring]]
 * 
 * to compile a PF_RING enabled kernel.

This is OK we just want to fetch the package for now.

Code: Find a place to work

Next move to a good place to work in.

# cd /usr/local/src

Copy over the libpfring tarball, unpack it, and cd into the new directory.

# cp /usr/portage/distfiles/libpfring-3.2.1-r1.tar.bz2 .
# tar -jxvf libpfring-3.2.1-r1.tar.bz2
# cd libpfring-3.2.1-r1/
Code: Edit mkpatch.sh

If take a look at the directory you are in you'll see a file called mkpatch.sh. This script is used to create a patch for your kernel.

# ls -l
total 20
-rw-r--r-- 1 root root   275 Feb  8 13:13 README
drwxr-xr-x 4 root root   120 Feb  8 13:13 kernel
-rw-r--r-- 1 root root 14414 Feb  8 13:13 mkpatch.sh
drwxr-xr-x 5 root root   144 Feb  8 13:13 userland

Before editing this file you need some information about your current kernel so run...

# uname -r

Which in this case gives us...

2.6.19-gentoo-r5

Now open mkpatch.sh with your favorite text editor.

# vi mkpatch.sh

Scroll down a short ways and you'll see a number of variables defined. You need to change a couple of these variables to match your current kernel.
Based on the previous output of uname -r, change the following variables (and ONLY these variables) to this...

PATCH=pfring
VERSION=2
PATCHLEVEL=6
SUBLEVEL=19-gentoo-r5
EXTRAVERSION=-$PATCH

Looking at their original values might lead you to believe that -gentoo-r5 belongs in EXTRAVERSION , but it doesn't.
Don't miss the - before $PATCH in EXTRAVERSION
It is important that these are correct, so double check them before moving on.

Code: Create tarball of current kernel sources

Next, check to see if a directory called workspace exists. If not create it and then change directories to it.

# ls -l
total 20
-rw-r--r-- 1 root root   275 Feb  8 13:13 README
drwxr-xr-x 4 root root   120 Feb 15 12:05 kernel
-rwxr--r-- 1 root root 14356 Feb 15 10:27 mkpatch.sh
drwxr-xr-x 5 root root   144 Feb  8 13:13 userland

# mkdir workspace

# ls -l
total 20
-rw-r--r-- 1 root root   275 Feb  8 13:13 README
drwxr-xr-x 4 root root   120 Feb 15 12:05 kernel
-rwxr--r-- 1 root root 14356 Feb 15 10:27 mkpatch.sh
drwxr-xr-x 5 root root   144 Feb  8 13:13 userland
drwxr-xr-x 2 root root    48 Feb 15 12:07 workspace

# cd workspace

In the workspace directory, create a gziped (not bzip2) tarball of your sources. When creating the tarball, it is important to use the actual directory name and not the linux symlink.

# tar -zcpvf linux-2.6.19-gentoo-r5.tar.gz /usr/src/linux-2.6.19-gentoo-r5

You should now have a tarball of your kernel sources in the</nowiki> workspace directory.

# pwd
/usr/local/src/libpfring-3.2.1-r1/workspace

# ls -l
total 71274
-rw-r--r-- 1 root root 72912783 Feb 15 11:47 linux-2.6.19-gentoo-r5.tar.gz
Code: Create PF_RING patch for your sources

Next, change directories back to the directory containing mkpatch.sh, make mkpatch.sh executable, and run the script.

# cd ../
# chmod u+x mkpatch.sh
# ./mkpatch.sh

If you have done everything correct up to this point, you should see the following output...

Creating patch for Linux kernel linux-2.6.19-gentoo-r5 ...
Edit this file (mkpatch.sh) for a different kernel version
Kernel build area is /usr/local/src/libpfring-3.2.1-r1/workspace
rm: cannot remove `/usr/local/src/libpfring-3.2.1-r1/workspace/pfring': No such file or directory
Creating link to /usr/local/src/libpfring-3.2.1-r1 in /usr/local/src/libpfring-3.2.1-r1/workspace called pfring
Found linux-2.6.19-gentoo-r5.tar.gz in source directory /usr/local/src/libpfring-3.2.1-r1/workspace
Untarring Linux sources (read-only tree) in /usr/local/src/libpfring-3.2.1-r1/workspace/linux-2.6.19-gentoo-r5
Cloning Linux sources (read-write tree) in /usr/local/src/libpfring-3.2.1-r1/workspace
Patching Linux sources ...
  1. Install additional file include/linux/ring.h with definitions
     for packet ring.
     done
  2. Install the ring sources under the kernel tree.
     Installing kernel ring sources in
     linux-2.6.19-gentoo-r5-pfring/net/ring ... done
  3. Patch net/core/dev.c ... 
     Patch #1 (define ring_handler)
     Patch #2 (modify function netif_rx and netif_receive_skb)
     Patch #3 (modify dev_queue_xmit, found in PATCH-3-to-dev.c)
     ... done
  4. Patching file net/Makefile ... done
  5. Copy net/ring/Kconfig to linux-2.6.19-gentoo-r5-pfring/net/ring/Kconfig done
  6. Patching file net/Kconfig ... done
diff --unified --recursive --new-file linux-2.6.19-gentoo-r5 linux-2.6.19-gentoo-r5-pfring > linux-2.6.19-gentoo-r5-pfring.patch
Making Linux patch file. This could take some time, please wait ... done
Your patch file is now in /usr/local/src/libpfring-3.2.1-r1/workspace/linux-2.6.19-gentoo-r5-pfring.patch.gz

You should have a patch file for your sources in the workspace directory. If the script tries to fetch kernel sources from the internet you have done something wrong. Check the variables you defined in mkpatch.sh and try again.

Code: Make a copy of kernel sources and apply the patch

Now copy the patch file to /usr/src and change to that directory.

# cp workspace/linux-2.6.19-gentoo-r5-pfring.patch.gz /usr/src/
# cd /usr/src

Remove the linux symlink and copy your current kernel sources to a new directory with the same name adding -pfring to the end and recreate the symlink to the new kernel sources.

# rm linux
# cp -R linux-2.6.19-gentoo-r5 linux-2.6.19-gentoo-r5-pfring
# ln -s linux-2.6.19-gentoo-r5-pfring linux

Double check that the new source directory name matches the patch file name up to .patch.gz and that the linux symlink is pointing to the new kernel source directory.

# ls -l
total 46
lrwxrwxrwx  1 root root    29 Feb 15 12:27 linux -> linux-2.6.19-gentoo-r5-pfring
drwxr-xr-x 20 root root  1328 Feb  8 10:19 linux-2.6.19-gentoo-r5
drwxr-xr-x 20 root root  1328 Feb 15 12:26 linux-2.6.19-gentoo-r5-pfring
-rw-r--r--  1 root root 41220 Feb 15 12:22 linux-2.6.19-gentoo-r5-pfring.patch.gz

If everything looks good, move the patch file to the new kernel source directory, change directories to the new kernel source directory, and apply the patch first using the --dry-run option...

# mv linux-2.6.19-gentoo-r5-pfring.patch.gz linux-2.6.19-gentoo-r5-pfring/

# cd linux-2.6.19-gentoo-r5-pfring

# zcat linux-2.6.19-gentoo-r5-pfring.patch.gz | patch --dry-run -p1
patching file include/linux/ring.h
patching file net/Kconfig
patching file net/Makefile
patching file net/Makefile.ORG
patching file net/core/dev.c
patching file net/core/dev.c.ORG
patching file net/ring/Kconfig
patching file net/ring/Makefile
patching file net/ring/ring_packet.c

DO NOT TRY TO APPLY THE PATCH FROM /usr/src USING -p0 !!! This would probably patch your original source tree instead of the new source tree!
If there are no errors apply the patch for real by removing the --dry-run option...

# zcat linux-2.6.19-gentoo-r5-pfring.patch.gz | patch -p1
patching file include/linux/ring.h
patching file net/Kconfig
patching file net/Makefile
patching file net/Makefile.ORG
patching file net/core/dev.c
patching file net/core/dev.c.ORG
patching file net/ring/Kconfig
patching file net/ring/Makefile
patching file net/ring/ring_packet.c
Code: Configure and install the new kernel

First we need to edit the kernel Makefile. So open it with your favorite text editor and change...

EXTRAVERSION = -gentoo-r5

to...

EXTRAVERSION = -gentoo-r5-pfring

We do not need to copy over our .config since this is a copy of an existing configured kernel. Now run make menuconfig. Some people say you should first run make oldconfig. I have had bad experiences doing this in the past, so I do not recommend it. Set the following options using make menuconfig...

Linux Kernel Configuration: Configure the new kernel (Required)

The following options are required...

Code maturity level options  --->
  [*] Prompt for development and/or incomplete code/drivers

You must enable "Loadable module support". The PF_RING module must be loaded as a module (read the IMPORTANT INFO section below to understand why).

Loadable module support  --->
 [*] Enable loadable module support
 [*]   Module unloading

Next enable the "PF_RING sockets" support as a module. This option will be built as a module even if you try to build it directly into the kernel!

Networking  --->
  [*] Networking support
    Networking options  --->
      <M> PF_RING sockets (EXPERIMENTAL)
Linux Kernel Configuration: Configure the new kernel (optional)

The following options are optional but highly recommended to improve packet capture

Set the following based on the amount of RAM you have...

Processor type and features  --->
  High Memory Support (4GB)  --->

Set the following to use faster IO communication...

Networking  --->
  [*] Networking support
    Networking options  --->
      [*]   Packet socket: mmapped IO

If you are using a network card that supports NAPI polling you should enable it...

Device Drivers  --->
  Network device support  --->
    Ethernet (10 or 100Mbit)  --->
      [*] Ethernet (10 or 100Mbit)
      [*] EISA, VLB, PCI and on board controllers
      <*>   nForce Ethernet support
      [*]     Use Rx and Tx Polling (NAPI) (EXPERIMENTAL)
      <*>   AMD PCnet32 PCI support
      [*]     Use RX polling (NAPI) (EXPERIMENTAL)  
     
Code: Configure and install the new kernel (continued)

Now exit menuconfig saving the new .config and build the kernel...

make && make modules_install

After the kernel and modules build we need to copy the header file for the PF_RING module to the correct place.

# cp include/linux/ring.h /usr/include/linux/ring.h

Now install the new kernel. For me that means...

# mount /boot
# mv /boot/bzImage /boot/bzImage.old
# cp arch/i386/boot/bzImage /boot/bzImage-pfring

Update grub.conf...

# vi /boot/grub/grub.conf

Add an entry that points to /boot/bzImage-pfring. This is just an example. You need an entry that works for your system!.

title  Gentoo Linux - pfring
root (hd0,0)
kernel (hd0,0)/bzImage-pfring root=/dev/hda3

Do the following to ensure that the ring module loads at start up

# echo ring >> /etc/modules.autoload.d/kernel-2.6

Now reboot choosing the PF_RING enabled kernel at the grub menu.

# reboot
Code: Verify that the module has loaded

Hopefully at this point you have rebooted and are using a PF_RING enabled kernel. Let's check a couple of things...
Doing uname -r should show...

2.6.19-gentoo-r5-pfring

Doing lsmod should show a module called ring...

# lsmod
Module                  Size  Used by
ring                   11852  0 

and if you do a tail -n50 /var/log/messages somewhere in there you should see the following...

 Welcome to PF_RING 3.2.1
 (C) 2004-06 L.Deri <deri@ntop.org>
 NET: Registered protocol family 27
 PF_RING: bucket length    128 bytes
 PF_RING: ring slots       4096
 PF_RING: sample rate      1 [1=no sampling]
 PF_RING: capture TX       No [RX only]
 PF_RING: transparent mode Yes
 PF_RING initialized correctly.
 PF_RING: registered /proc/net/pf_ring/

If this all looks good, you have successfully installed a PF_RING enabled kernel. The next thing you should do is read the section below to avoid some serious pitfalls.


IMPORTANT INFO

There are a couple of important things to note when using PF_RING.

  1. Transparent Mode


Credits

  1. PF_RING Main Page
  2. Luca Deri's Improving Passive Packet Capture:Beyond Device Polling
Retrieved from "http://www.gentoo-wiki.info/Pfring"

Last modified: Fri, 05 Sep 2008 09:26:00 +0000 Hits: 2,175