Netflow analysis with SiLK - Part 1 Installation
About netsa SiLK
SiLK provides a way to capture netflow and interrogate flow data. It can be used for a variety of purposes including situational awareness, forensics and anomaly detection.Installing SiLK
Prerequisites
Before installing SiLK examine the perquisites for the build. I’m using CentOS 6.3 minimal install with a virtual machine disk of 100GB in total with a separate partition for /data of about 60GB. The virtual machine will be allocated 4GB of RAM and 2 virtual processors. Configure the network settings for your lab environment. In order to use YUM for installing software the the lab system will require access to the Internet. Download the following source code files from the netsa CERT project home page- libfixbuf-1.2.0.tar.gz
- netsa-python-1.3.tar.gz
- silk-2.5.0.tar.gz
- yaf-2.3.2.tar.gz
- yum install gcc make
- yum install bind-utils
- yum install unzip
- yum install man
- yum install python-devel
- yum install glib2-devel
- yum install gnutls gnutls-devel
- yum install lzo lzo-devel
- yum install libpcap libpcap-devel
- yum install zlib zlib-devel
[root@silk src]# ls libfixbuf-1.2.0.tar.gz netsa-python-1.3.tar.gz silk-2.5.0.tar.gz yaf-2.3.2.tar.gzUncompress and unarchive libfixbuf first:
[root@silk src]# tar zxvf libfixbuf-1.2.0.tar.gz [root@silk src]# cd libfixbuf-1.2.0 [root@silk libfixbuf-1.2.0]# ./configure [root@silk libfixbuf-1.2.0]# make && make installIf all goes well fixbuf should be installed. Next install netsa-python library.
[root@silk src]# tar zxvf netsa-python-1.3.tar.gz [root@silk src]# cd netsa-python-1.3 [root@silk netsa-python-1.3]# python setup.py installSo far, it’s fairly easy. Now for the hard bit (it’s not too hard). Building and installing SiLK requires a few steps to get it setup correctly prior to configuration.
root@silk src]# tar zxvf silk-2.5.0.tar.gz [root@silk src]# cd silk-2.5.0Setting the following environment variable tells configure where to find libraries as the default script looks in the wrong location and fails to find them. The sysconfdir parameter specifies where to look for daemon configuration files, if not specified the scripts for services will look for the configuration files in the wrong place.
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
[root@silk silk-2.5.0]# export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig [root@silk silk-2.5.0]# [root@silk silk-2.5.0]# ./configure --with-python --sysconfdir=/usr/local/share/silk/etcAfter configure finishes it produces a summary, which should look like the following: Make sure all the options required have a ‘yes’ against them and that the paths given are correct. Then proceed to compile and install SiLK.
[root@silk silk-2.5.0]# makeIf everything works install the software.
[root@silk silk-2.5.0]# make install
Configuring SiLK
At this point SiLK is ready to be configured. This part is highly dependent on where you want to collect netflow data from. There are a couple of basic config steps to support internal address spaces and add geo-location data before getting into configuring SiLK to work with Netflow sources. The SiLK Installation Handbook available from the netsa CERT website is highly recommended at this point. It provides all the detail necessary to configure SiLK. I will provide a couple of examples in this blog post but configuration depends on your environment and what devices you want to instrument. First copy the /usr/local/share/silk/addrtype-templ.txt to addresses.txt and add your internal address space to the bottom of the file. See section 3 of the SiLK Installation Handbook. SiLK needs this file compiled before it can use it.[root@silk silk-2.5.0]# cd /usr/local/share/silk [root@silk silk]# cp addrtype-templ.txt addresses.txt [root@silk silk]# vi addresses.txt [root@silk silk]# rwpmapbuild --input addresses.txt --output address_types.pmapDownload the free version of the GeoIP Country database distributed by from MaxMind Unzip the archive and and compile the file for use by SiLK.
[root@silk silk-2.5.0]# cd /usr/local/share/silk [root@silk silk]# unzip -p GeoIPCountryCSV.zip | rwgeoip2ccmap --csv-input > country_codes.pmapCopy /usr/local/share/silk/twoway-silk.conf to /data/silk.conf and customise it for the sensors being used.
/usr/local/share/silk [root@silk silk]# cp twoway-silk.conf /data/silk.conf
For my test lab I want to collect netflow from my Internet router and two virtual routers in my vmware virtual lab server. After editing the silk.conf edit the /usr/local/share/silk/etc/sesnor.conf to provide the details for each of the sensor you want to use.
The Iroute sensor is split into two sensors in this configuration, to allow easy access to DMZ traffic in my test lab. There is a lot of flexibility in the way that sensors can be configured. I would highly recommend reading the SiLK Installation Handbook and section 4 in particular for configuring sensors. The Vrouter1 sensor configuration is a simple example. Vrouter1 is a virtual router inside my vmware ESXi server. It links two networks (vswitches) together. However, before configuring the Vrouter1 netflow probe to send to the newly configured sensor, rwflopack.conf must be customised and the service started.
rwflowpack is the server process that will receive the netflow data and process it into the data store. The rwflowpack.conf file is used to provide configuration information to the program. It MUST be edited before starting the rwflowpack service.
To enable netflow set the destination server, netflow version and network interfaces to be monitored. Check the configuration on the router by running ‘run show configuration’.
[root@silk etc]# cd /usr/local/share/silk/etc/ [root@silk etc]# vi rwflowpack.confThe rwflowpack.conf file needs the following changes:
- ENABLED=yes
- CREATE_DIRECTORIES=yes
- SENSOR_CONFIG=/usr/local/share/silk/etc/sensor.conf
- COMPRESSION_TYPE=best
[root@silk etc]# pwd /usr/local/share/silk/etc [root@silk etc]# cd init.d/ [root@silk init.d]# cp rwflowpack /etc/init.dTest starting rwflowpack using the script, it will report any errors on startup and log to /var/log/messages. If everything worked OK as above stop the rwflowpack.
[root@silk init.d]# sh rwflowpack stop Stopping rwflowpack: [OK] [root@silk init.d]#Add the firewall rules to allow the server to receive the netflow data from the probes specified.
iptables -I INPUT 2 --proto udp --dport 9998 -j ACCEPT iptables -I INPUT 2 --proto udp --dport 9997 -j ACCEPT service iptables save Configure the rwflowpack service to start automatically. [root@silk init.d]# chkconfig --add rwflowpack [root@silk init.d]# chkconfig --level 2345 rwflowpack on [root@silk init.d]# chkconfig --list rwflowpack rwflowpack 0:off 1:off 2:on 3:on 4:on 5:on 6:off [root@silk init.d]#
Start the service [root@silk rwflowpack]# service rwflowpack startThat completes the installation and configuration of the SiLK server. Next configure a router to send netflow data to the server. I use Vyatta community edition as a software router running as a virtual machine on a VMware ESXi host. The following diagram shows the logical configuration of the router. I really like the Vyatta community edition, it can be used for lots of things in the test lab it includes Firewall, IDS, Netflow probe and many other features.
Router Configuration
To enable netflow set the destination server, netflow version and network interfaces to be monitored. Check the configuration on the router by running ‘run show configuration’.
system {
config-management {
commit-revisions 20
}
console {
device ttyS0 {
speed 9600
}
}
flow-accounting {
interface eth0
interface eth1
netflow {
server 172.31.254.117 {
port 9998
timeout {
flow-generic 1800
}
version 9
}
Generating some traffic through the router will create some flow data on the SiLK server. As a quick check look in the /data directory on the SiLK server.
[root@silk src]# ls -l /data total 40 drwxr-xr-x. 3 root root 4096 Oct 12 14:20 in drwxr-xr-x. 3 root root 4096 Oct 12 14:20 innull drwx------. 2 root root 16384 Oct 11 11:54 lost+found drwxr-xr-x. 3 root root 4096 Oct 12 14:20 out drwxr-xr-x. 3 root root 4096 Oct 12 14:20 outnull drwxr-xr-x. 3 root root 4096 Oct 12 14:20 outweb -rw-r--r--. 1 root root 1511 Oct 11 19:23 silk.conf
Try out rwfilter
The rwfilter utility (and many others) are used to query the data. For example the following shows outweb (a class of flow) flows. I used nmap from the 172.31.254.28 machine to generate some traffic to test the system.[root@silk src]# rwfilter --proto=6 --type=outweb --pass=stdout | rwuniq --fields=1,2,3,4
sIP| dIP|sPort|dPort| Records|
172.31.253.102| 172.31.254.28| 80|42241| 1|
172.31.253.109| 172.31.254.28| 80|34144| 1|
172.31.253.102| 172.31.254.28| 443|37574| 1|
172.31.253.102| 172.31.254.28| 8080|39866| 1|
172.31.253.102| 172.31.254.28| 443|37561| 1|
172.31.253.108| 172.31.254.28| 443|39866| 1|
172.31.253.102| 172.31.254.28| 80|39866| 1|
172.31.253.109| 172.31.254.28| 443|39866| 1|
172.31.253.108| 172.31.254.28| 80|46409| 1|
172.31.253.102| 172.31.254.28| 443|37521| 1|
172.31.253.102| 172.31.254.28| 443|39866| 1|
172.31.253.109| 172.31.254.28| 80|39866| 1|
172.31.253.108| 172.31.254.28| 80|39866| 1|
172.31.253.108| 172.31.254.28| 8080|39866| 1|
172.31.253.109| 172.31.254.28| 8080|39866| 1|
The power of the SiLK software lies in how flexible the query capabilities are and how they can mine useful data and allow you to draw meaningful conclusions. The query below is used to detect a scan of a set of IP’s like the one I did to generate to some flow records to test the SiLK installation.
In the next post I’ll put together a few simple queries showing how to use SiLK and netflow to determine what’s going on in the network.
Comments
Post a Comment