Netflow analysis with SiLK - Part 1 Installation

About netsa SiLK

SiLK provides a way to capture netflow and interrogate flow data. It can be used for a variety of purposes including situational awareness, forensics and anomaly detection.

Installing SiLK


Before installing SiLK examine the perquisites for the build. I’m using CentOS 6.3 minimal install with a virtual machine disk of 100GB in total with a separate partition for /data of about 60GB. The virtual machine will be allocated 4GB of RAM and 2 virtual processors. Configure the network settings for your lab environment. In order to use YUM for installing software the the lab system will require access to the Internet. Download the following source code files from the netsa CERT project home page
  • libfixbuf-1.2.0.tar.gz
  • netsa-python-1.3.tar.gz
  • silk-2.5.0.tar.gz
  • yaf-2.3.2.tar.gz
Copy the required packages to the /usr/local/src directory. If you are using scp to copy over the packages dont’ forget ‘yum install openssh-clients’ first on the server. Start by updating the base install to the latest versions: yum update Reboot if necessary after all the updates are installed. Use YUM to install the development tools to build the SiLK software:
  • yum install gcc make
Add a couple of useful utils:
  • yum install bind-utils
  • yum install unzip
  • yum install man
Install all the prerequisites:
  • yum install python-devel
  • yum install glib2-devel
  • yum install gnutls gnutls-devel
  • yum install lzo lzo-devel
  • yum install libpcap libpcap-devel
  • yum install zlib zlib-devel
Once all the development tools and required software is installed the next step is to build the required components and SiLK.
[root@silk src]# ls
libfixbuf-1.2.0.tar.gz  netsa-python-1.3.tar.gz  silk-2.5.0.tar.gz  yaf-2.3.2.tar.gz
Uncompress and unarchive libfixbuf first:
[root@silk src]# tar zxvf libfixbuf-1.2.0.tar.gz 
[root@silk src]# cd libfixbuf-1.2.0
[root@silk libfixbuf-1.2.0]# ./configure
[root@silk libfixbuf-1.2.0]# make && make install
If all goes well fixbuf should be installed. Next install netsa-python library.
[root@silk src]# tar zxvf netsa-python-1.3.tar.gz 
[root@silk src]# cd netsa-python-1.3
[root@silk netsa-python-1.3]# python install
So far, it’s fairly easy. Now for the hard bit (it’s not too hard). Building and installing SiLK requires a few steps to get it setup correctly prior to configuration.
root@silk src]# tar zxvf silk-2.5.0.tar.gz 
[root@silk src]# cd silk-2.5.0
Setting the following environment variable tells configure where to find libraries as the default script looks in the wrong location and fails to find them. The sysconfdir parameter specifies where to look for daemon configuration files, if not specified the scripts for services will look for the configuration files in the wrong place.
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
[root@silk silk-2.5.0]# export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
[root@silk silk-2.5.0]# 
[root@silk silk-2.5.0]# ./configure --with-python --sysconfdir=/usr/local/share/silk/etc
After configure finishes it produces a summary, which should look like the following: Make sure all the options required have a ‘yes’ against them and that the paths given are correct. Then proceed to compile and install SiLK.
[root@silk silk-2.5.0]# make
If everything works install the software.
[root@silk silk-2.5.0]# make install

Configuring SiLK

At this point SiLK is ready to be configured. This part is highly dependent on where you want to collect netflow data from. There are a couple of basic config steps to support internal address spaces and add geo-location data before getting into configuring SiLK to work with Netflow sources. The SiLK Installation Handbook available from the netsa CERT website is highly recommended at this point. It provides all the detail necessary to configure SiLK. I will provide a couple of examples in this blog post but configuration depends on your environment and what devices you want to instrument. First copy the /usr/local/share/silk/addrtype-templ.txt to addresses.txt and add your internal address space to the bottom of the file. See section 3 of the SiLK Installation Handbook. SiLK needs this file compiled before it can use it.
[root@silk silk-2.5.0]# cd /usr/local/share/silk
[root@silk silk]# cp addrtype-templ.txt addresses.txt
[root@silk silk]# vi addresses.txt 
[root@silk silk]# rwpmapbuild --input addresses.txt --output address_types.pmap
Download the free version of the GeoIP Country database distributed by from MaxMind Unzip the archive and and compile the file for use by SiLK.
[root@silk silk-2.5.0]# cd /usr/local/share/silk
[root@silk silk]# unzip -p | rwgeoip2ccmap --csv-input > country_codes.pmap
Copy /usr/local/share/silk/twoway-silk.conf to /data/silk.conf and customise it for the sensors being used.
[root@silk silk]# cp twoway-silk.conf /data/silk.conf
Silk conf
For my test lab I want to collect netflow from my Internet router and two virtual routers in my vmware virtual lab server. After editing the silk.conf edit the /usr/local/share/silk/etc/sesnor.conf to provide the details for each of the sensor you want to use. Sensor conf 
The Iroute sensor is split into two sensors in this configuration, to allow easy access to DMZ traffic in my test lab. There is a lot of flexibility in the way that sensors can be configured. I would highly recommend reading the SiLK Installation Handbook and section 4 in particular for configuring sensors. The Vrouter1 sensor configuration is a simple example. Vrouter1 is a virtual router inside my vmware ESXi server. It links two networks (vswitches) together. However, before configuring the Vrouter1 netflow probe to send to the newly configured sensor, rwflopack.conf must be customised and the service started. rwflowpack is the server process that will receive the netflow data and process it into the data store. The rwflowpack.conf file is used to provide configuration information to the program. It MUST be edited before starting the rwflowpack service.
[root@silk etc]# cd /usr/local/share/silk/etc/
[root@silk etc]# vi rwflowpack.conf 
The rwflowpack.conf file needs the following changes:
  • ENABLED=yes
  • SENSOR_CONFIG=/usr/local/share/silk/etc/sensor.conf
Once rwflowpack.conf is ready copy the startup script to /etc/init.d
[root@silk etc]# pwd
[root@silk etc]# cd init.d/
[root@silk init.d]# cp rwflowpack /etc/init.d
Test starting rwflowpack using the script, it will report any errors on startup and log to /var/log/messages. If everything worked OK as above stop the rwflowpack.
[root@silk init.d]# sh rwflowpack stop
Stopping rwflowpack: [OK]
[root@silk init.d]# 
Add the firewall rules to allow the server to receive the netflow data from the probes specified.
iptables -I INPUT 2 --proto udp --dport 9998 -j ACCEPT
iptables -I INPUT 2 --proto udp --dport 9997 -j ACCEPT
service iptables save
Configure the rwflowpack service to start automatically.
[root@silk init.d]# chkconfig --add rwflowpack
[root@silk init.d]# chkconfig --level 2345 rwflowpack on
[root@silk init.d]# chkconfig --list rwflowpack
rwflowpack      0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@silk init.d]# 
Start the service
[root@silk rwflowpack]# service rwflowpack start
That completes the installation and configuration of the SiLK server. Next configure a router to send netflow data to the server. I use Vyatta community edition as a software router running as a virtual machine on a VMware ESXi host. The following diagram shows the logical configuration of the router. I really like the Vyatta community edition, it can be used for lots of things in the test lab it includes Firewall, IDS, Netflow probe and many other features.

Router Configuration

Vyatta Vrouter1 TestLab
To enable netflow set the destination server, netflow version and network interfaces to be monitored. Check the configuration on the router by running ‘run show configuration’.
system {
    config-management {
        commit-revisions 20
    console {
        device ttyS0 {
            speed 9600
    flow-accounting {
        interface eth0
        interface eth1
        netflow {
            server {
                port 9998
            timeout {
                flow-generic 1800
            version 9
Generating some traffic through the router will create some flow data on the SiLK server. As a quick check look in the /data directory on the SiLK server.
[root@silk src]# ls -l /data
total 40
drwxr-xr-x. 3 root root  4096 Oct 12 14:20 in
drwxr-xr-x. 3 root root  4096 Oct 12 14:20 innull
drwx------. 2 root root 16384 Oct 11 11:54 lost+found
drwxr-xr-x. 3 root root  4096 Oct 12 14:20 out
drwxr-xr-x. 3 root root  4096 Oct 12 14:20 outnull
drwxr-xr-x. 3 root root  4096 Oct 12 14:20 outweb
-rw-r--r--. 1 root root  1511 Oct 11 19:23 silk.conf

Try out rwfilter

The rwfilter utility (and many others) are used to query the data. For example the following shows outweb (a class of flow) flows. I used nmap from the machine to generate some traffic to test the system.
[root@silk src]# rwfilter --proto=6 --type=outweb --pass=stdout | rwuniq --fields=1,2,3,4
            sIP|            dIP|sPort|dPort|   Records|||   80|42241|         1|||   80|34144|         1|||  443|37574|         1||| 8080|39866|         1|||  443|37561|         1|||  443|39866|         1|||   80|39866|         1|||  443|39866|         1|||   80|46409|         1|||  443|37521|         1|||  443|39866|         1|||   80|39866|         1|||   80|39866|         1||| 8080|39866|         1||| 8080|39866|         1|

The power of the SiLK software lies in how flexible the query capabilities are and how they can mine useful data and allow you to draw meaningful conclusions. The query below is used to detect a scan of a set of IP’s like the one I did to generate to some flow records to test the SiLK installation. In the next post I’ll put together a few simple queries showing how to use SiLK and netflow to determine what’s going on in the network.


Popular posts from this blog

Squid Proxy with SOF-ELK Part 1

Password Cracking with CUDA 2 ways