specter, userspace logging daemon.
specter makes use of the Linux >= 2.4.x packet filter subsystem (iptables)
and the ULOG target for iptables.
I want to provide a flexible, almost universal logging daemon for netfilter ULOG target. Although it provides wide range of functions I'm trying to keep it as simple as possible. These are my thoughts about how the architecture which is most capable of doing that:
It should be possible to add plugins / runtime modules for new protocols, etc. For example the standard logging daemon provides source-ip, dest-ip, source-port, dest-port, etc. Logging for variuos other protocols (GRE, IPsec, ...) may be implemented as modules.
... describe how and where to put the information gained by input plugins. The easiest way is to build a line per packet and fprint it to a file. Some people might want to log into a SQL database or want an output conforming to the intrusion detection systems communication draft from the IETF.
The major clue is providing a framework which is as flexible as possible. Nobody knows what strange network protocols are out there :) Flexibility depends on the communication between the output of the input plugins and input of the output plugins.
Harald, following Rusty's advise, implemented type-key-value triples,
which work quite well for that purpose. Structure used for exchanging
data between input and output plugins is defined in specter.h, and
is called specter_iret_t. Most of time, output plugins precisely know what
data they need, so there must exist good querying system, as input
keys are dynamically defined and stored. Up to ulogd 0.3 this was done
by several linked list iterations, which weren't obviously very fast.
In 0.9 Harald implemented usage of hash tables initialized during
init. The idea was good, but deep levels of data structures one
had to dig into to get simple value and somewhat obscure style
(like accessing ulogd_keyh[] from the inside of plugin) forced
me (Michal) to rewrite this again. That's when fork from ulogd happened.
Abandoning hash tables, specter implementation use only pointers
accessed by general function find_iret(). To simplify usage of
that pointers, simple data structure specter_local_ret_t and
few macros defined in plugins/lret.h were also created.
Note they're not the obligatory extension; one can create his own
implementation based on single find_iret() definition.
Important part of specter is dynamic division into execution groups.
Each group have its own set of plugins, which are invoked independently.
That also means they have separate sets of configure options and data.
This model allows you to set various iptables rules and bind different
actions for them. Currently there are two methods of grouping implemented
- one based on netlink groups, other on the netfilter marks. See the
grouping option description
for details. This functionality allows you to adjust specter strictly
to your needs - it can be small and simple substitute to netfilter LOG
target or universal utility to distribute large portions of logs.
First you will need a 2.4.x or 2.6.x kernel. If you have
a kernel >= 2.4.18-pre8, it already has the kernel suport for ULOG
(ipt_ULOG.o), only make sure that it was compiled in. If you want
to use nfmark
grouping
method, check if your kernel was compiled with CONFIG_IP_NF_MATCH_MARK,
CONFIG_IP_NF_TARGET_MARK and CONFIG_IP_NF_MANGLE options.
If you have an older kernel version (between 2.4.0 and 2.4.18-pre6), you can use the patch-o-matic system of netfilter/iptables, as described in the following section.
If you experience problems like described in my mail http://lists.netfilter.org/pipermail/netfilter-devel/2004-June/015860.html you should apply ipt_ULOG patch you can find in contrib/ subdirectory. It has been attached to 2.4.27 and 2.6.9 kernel.
You only need to read this chapter if you have a 2.4.x kernel <= 2.4.18-pre6.
In order to put the ipt_ULOG module into your kernel source, you need the latest iptables package, or even better: the latest CVS snapshot. A description how to obtain this is provided on the netfilter homepage http://www.netfilter.org/.
To run patch-o-matic, just type
make patch-o-matic
Download the specter package from http://joker.linuxstuff.pl/specter/ and untar it.
If you want to build specter with MySQL support, type './configure --with-mysql'. You may also have to specify the path of the mysql libraries using '--with-mysql=path'. To build specter without MySQL support, just use './configure'.
The same procedure apply to PostgreSQL support (use './configure --with-pgsql' with or without path to libraries).
If you have other applications using libipulog library contained with this package, you may consider building it shared. To enable this, use './configure --with-sharedlib'.
If you have sophisticated configuration and need more than default 32 execution groups, you can redefine SPECTER_GROUPS_MAX by using '--with-group-max=value' configuration option. That won't have any consequences to speed until you acctually make use of these groups. But please note that netlink grouping allows you to specify only 32 groups, and that limit is kernel-driven. Use nfmarks instead.
To compile and install the program, call 'make install'. Old configuration files won't be overwritten, so don't worry. You can also run 'make install-strip' to discard redundant symbols from specter binary.
If you're happy with defaults, there's possibility to install specter from binary package. You can download them from project's homepage: http://joker.linuxstuff.pl/specter/. Currently tgz and rpm formats are available.
There are several front-ends for viewing logs generated by specter. Although they where designed for ulogd, there should be no problem with using them with specter. Here are few links to that kind of projects:
You can also find sample php query script in contrib/.
Just add rules using the ULOG target to your firewalling chain. A very basic example:
iptables -A FORWARD -j ULOG --ulog-nlgroup 32 --ulog-prefix foo
To increase logging performance, try to use the
--ulog-qthreshold N
Of course you can combine the ULOG target with the different netfilter match modules. For a more detailed description, have a look at the netfilter HOWTO's, available on the netfilter homepage.
The number of the netlink multicast group to which ULOG'ed packets are sent. In specter, you can specify different task for different netlink groups, see specter configfile syntax reference section for more details.
Copyrange. This works like the 'snaplen' paramter of tcpdump. You can specify a number of bytes up to which the packet is copied. If you say '40', you will receive the first fourty bytes of every packet. Leave it to '0' if you want whole packet to be copied to userspace. For most tcp packets about 50 is mostly enough, but parsing higher level protocols (like in HTTP plugin) require more.
Queue threshold. If a packet is matched by the iptables rule, and already N packets are in the queue, the queue is flushed to userspace. You can use this to implement a policy like: Use a big queue in order to gain high performance, but still have certain packets logged immediately to userspace.
A string that is associated with every packet logged by this rule. You can use this option to later tell from which rule the packet was logged.
The ipt_ULOG kernel module has a couple of module loadtime parameters which can (and should) be tuned to accomodate the needs of the application:
Netlink buffer size. A buffer of the specified size N is allocated for every netlink group that is used. Please note that due to restrictions of the kernel memory allocator, we cannot have a buffer size > 128kBytes. Larger buffer sizes increase the performance, since less kernel/userspace context switches are needed for the same amount of packets. The backside of this performance gain is a potentially larger delay. The default value is 4096 bytes, which is quite small.
The flushtimeout determines, after how many clock ticks (on alpha: 1ms, on x86 and most other platforms: 10ms time units) the buffer/queue is to be flushed, even if it is not full. This can be used to have the advantage of a large buffer, but still a finite maximum delay introduced. The default value is set to 10 seconds.
modprobe ipt_ULOG nlbufsiz=65535 flushtimeout=100
If you were using ulogd before and want to keep your configuration,
check the ulogd2specter.pl script in contrib/, which will
convert your configfile. But it's still good to learn the new syntax,
as it gives you much more possibilities.
specter reads its configuration parameters from file, which is mostly
`/etc/specter.conf'. It is divided into blocks. Each block start with
a opening curly bracket { and end with closing curly bracket
}. Nesting of blocks (opening new block inside another) is
forbidden, and there's no need for that in specter configuration. In
order to distinguish between blocks, each has a name.
You can use any name for a block, except two special names:
global (which is used to specify general daemon parameters)
and plugins (that list available add-ons).
Numbers in range 1-SPECTER_GROUPS_MAX has also
special functional meaning (see
grouping option description).
You cannot define the same block twice, but don't have to define all of
them. In most configurations you'll be fine with three or four blocks.
Each block have to start in a new line, then goes its name and opening
bracket. All blocks (except for global and plugins)
are divided into logical sections, which define a configuration space
for every plugin. You start a section with a colon :
followed by its name. Within section you can finally specify your
configuration. global and plugins blocks are simpler
in that manner that they don't have any sections. Block ends with
a closing bracket. So, in general, block definition looks like this:
name {
include other_block
:section_one
some_option value
# comment
other_option "long value that needs spaces"
:section_two
# this section have no options, but it's important to specify it
:section_three
option
option value # another comment
...
}
As you can see, not every option needs a value, in that case its
presence will override a default (see below for specific options
description). A hash #
is used as a comment, as it will cause a rest of line to be ignored. Of
course you can use comments everywhere, not only inside blocks. If you
need to set an option to a string containing spaces or tabs, you can
enclose it inside double quotation marks, as shown above. And if you
ever manage to write a very long config line, you can cut it by
\ and continue your statement in the line below.
Since 1.2 version of specter you can use include statement
to attach contents of other block to current block. Length of
include chain is unlimited, but no recursion is allowed.
Each include command is performed exacly once, what
mostly does what you wanted.
Available global options are:
This options causes specter to continue running despite of errors generated by plugins. That doesn't affect initialization phase, when all errors cause an exit. This option can be useful on heavy-load systems, when you expect some malloc() to fail. It doesn't take any arguments.
Path to a file you want specter messages to get logged to. Can be set to
stdout or stderr.
The lower the value, the more information is logged. If you experience any problems, check lowest, debug loglevel=1, so that you can see all messages. The highest loglevel is 8, which cause only fatal errors to be shown. The default is 3.
Size of the netlink socket receive memory. You should set this to at least the
size of the kernel buffer (nlbufsiz parameter of the ipt_ULOG module). Please
note that there is a maximum limit in /proc/sys/net/core/rmem_max which you
cannot exceed by increasing the rmem parameter. You may need to raise the
system-wide maximum limit before. You can define this variable in kilobytes
(suffix it by 'K') or in megabytes (use 'M' suffix).
Size of the receive buffer. You should set this to at least the size rmem
option has. Like rmem can be suffixed by 'M' or 'K'.
That option sets grouping strategy. Every block which name is a number
within range 1 to SPECTER_GROUPS_MAX (default 32, use
--with-group-max build option to change it), will be treaten as
a separate execution block. Setting grouping to netlink
will cause interpreting these blocks as netlink groups (as defined with
--ulog-nlgroup iptables ULOG target option). When nfmark
value is used, groups will be compared to mark field in netfilter packet
(see iptables(8) for more details on MARK module). If you find it a bit
complicated, check
examples section.
Will set netlink group to listen to. Can't be used with grouping
set to netlink, as several nlgroups are used in that case.
plugins block structure is very simple. In each line symbolic
name and path to plugin binary have to be provided, like in a example:
BASE /lib/specter/specter_BASE.so
Please note that setting paths doesn't mean corresponding plugins will be loaded. You have to use them in blocks in order to force their load. That mean you can list all plugins you have compiled and select which to use by configuring execute blocks adequately.
Apart from the config file, there are a couple of commandline options to specter:
Print a help message about the commandline options.
Print version information about specter.
Fork off into daemon mode. Unless you are debugging, you will want to use this most of the time.
Using this commandline option, an alternate config file can be used. This is important if multiple instances of specter are to be run on a single machine.
This option tells specter to drop its privileges and run as given user.
This option tells specter to drop its privileges and run as given group.
For description of plugins and their options, see plugins section.
Say, you just want to log non-related tcp and udp packets in separate files. You must first set up your netfilter:
# iptables -A INPUT -p tcp -m state --state INVALID -j ULOG --ulog-nlgroup 1
# iptables -A INPUT -p udp -m state --state INVALID -j ULOG --ulog-nlgroup 2
plugins {
BASE /lib/specter/specter_BASE.so
LOCAL /lib/specter/specter_LOCAL.so
LOGEMU /lib/specter/specter_LOGEMU.so
}
1 {
:BASE
:LOCAL
:LOGEMU
logfile /var/log/specter.tcp
}
2 {
:BASE
:LOCAL
:LOGEMU
logfile /var/log/specter.udp
}
Maybe you want to analyze every packet that passes your HTTP server with a application that uses pcap-style files? Prepare you firewall:
# iptables -A INPUT -p tcp --dport 80 -j ULOG --ulog-nlgroup 5
# iptables -A OUTPUT -p tcp --sport 80 -j ULOG --ulog-nlgroup 5
/var/log/specter.http.
But you expect some attacks and want packets to appear immediately in
log, so you use sync option as well.
plugins {
BASE /lib/specter/specter_BASE.so
PCAP /lib/specter/specter_PCAP.so
}
5 {
:BASE
:PCAP
file /var/log/specter.http
sync
}
You're very paranoid and want to save all IPs that tried to ping you in a database, yes? Logging tcp requests are also in you concern, right? Moreover, you don't want to occupy more than one netlink group, so you decide to use mark module to divide packets into groups. Try these iptables rules:
# iptables -t mangle -A INPUT -p icmp --icmp-type echo-request -j MARK --set-mark 13
# iptables -t mangle -A INPUT -p tcp -m state --state NEW -j MARK --set-mark 15
# iptables -A INPUT -m mark --mark 13 -j ULOG --ulog-nlgroup 1
# iptables -A INPUT -m mark --mark 15 -j ULOG --ulog-nlgroup 1
global {
grouping nfmark
nlgroup 1
}
plugins {
BASE /lib/specter/specter_BASE.so
MYSQL /lib/specter/specter_MYSQL.so
}
13 {
:BASE
:MYSQL
db mydb
host localhost
user username
pass password
table pings
}
15 {
:BASE
:MYSQL
db mydb
host localhost
user username
pass password
table tcp_requests
}
You don't like fragmented packets? You can automaticaly block anyone who ever send you fragmented tcp packet. Use this single iptables rule:
# iptables -A INPUT -p tcp -f -j ULOG --ulog-nlgroup 1
plugins {
EXEC /lib/specter/specter_EXEC.so
}
1 {
:EXEC
command "/usr/sbin/iptables -A INPUT -p tcp -s %S --sport %s -j DROP"
}
specter does nearly nothing on its own, it uses plugins for all the dirty work. They are divided into two groups. Input plugins analyze a packet and create hash table concerning received data, in the form like key=value. They don't open files nor they take any input from user. Only output plugins take options. They actually use data from input plugins - save it into logs/databases or execute appropriate commands. So it's vital for you to learn about their configuration, because it's the essence of using specter.
Every input plugin analyze incoming packet in special way and generate
appropriate keys, which then can be parsed or saved by output plugins.
You can check what keys are generated by each input plugin by looking at
plugin's source - somewhere on top there's a definition of
specter_iret_t array, which lists all of them. For example,
specter_BASE generates keys like raw.mac,
ip.protocol or icmp.type, among many others of course.
Note that not all keys will be set during packet parsing, these that
won't be set will remain empty. Different output plugins handle this
case differently, please read details in their documentation.
specter comes with the following input plugins:
Basic input plugin for nfmark, timestamp, mac address, ip header, tcp header, udp header, icmp header, ah/esp header... Most output plugins need this very important plugin.
Example input plugin to log plaintext passwords as used with FTP and POP3. Don't blame me for writing this plugin! The protocols are inherently insecure, and there are a lot of other tools for sniffing passwords... it's just an example.
This is a 'virtual interpreter'. It doesn't really return any information on the packet itself, rather the local system time and hostname. Please note that the time is the time at the time of logging, not the packets receive time.
This plugin divides http message into set of keys, like protocol version or User-Agent header value. Number of supported headers is high, check the sources for full list.
specter comes with the following output plugins:
This plugin executes specified command when packet is received. By proper use of its functions you can dynamically change your firewall configuration, or even set up simple port-knocking utility.
That option defines a command that should be executed. Don't rely on your $PATH environment variable, and provide full path to an executable. Few printf-like macros can be used, which are expanded during parsing of every packet:
interface packet got received from
interface packet is going to be sent to
IP address of source host
IP address of destination host
IP protocol number (see /etc/protocols)
TCP/UDP source port
TCP/UDP destination port
ICMP type value
If you want to use literal '%' in command, write it double '%%'. You can also
use shell-like stdin/stdout/stderr redirections. > or 1>
truncates file to zero length and redirects stdout to it. >>
or 1>> will append stdout stream to destination file. In the
same manner work 2> and 2>> redirections, except
that they apply to stderr. To redirect stdout and stderr to the same file,
use &> or &>>. Redirecting input is done
by < operator, of course.
When a macros expansion is being done, and any field is empty, executing
of a given command is aborted. For example, if you have %i in your command
and specter gets a tcp packet, command won't be executed, 'cos given macro
cannot be expanded (there's no ICMP type field in a TCP packet). You can
override this behavior by setting force option. Instead of bogus data,
string "invalid" will be placed. It's up to executed application to work
with that.
If this options is set, daemon will wait until application terminates. It's probably not a good idea to actually use it. If you definitely need it, do it with caution, because it can freeze the whole daemon. Enforcing execution limits should be set in iptables rules by use of limit module, for example.
If set, child will inherit specter's environment. In other case child be be run in empty environment.
A very simple output module, dumping all packets in the format
===>PACKET BOUNDARY
key=value
key=value
...
===>PACKET BOUNDARY
...
The module defines the following configuration directives:
The filename where it should log to. The default is
/var/log/specter.oprint
An output module which tries to emulate the old syslog-based LOG targed as far as possible. Logging is done to a seperate textfile instead of syslog, though.
The module defines the following configuration directives:
The filename where it should log to. The default is /var/log/specter.logemu
Define this option if you want to have your logfile written synchronously. This may reduce performance, but makes your log-lines appear immediately.
Works the same way as ipt_LOG --log-tcp-options parameter. It enables logging of tcp options.
Log options from IP packet header (equivalent to --log-ip-options from ipt_LOG target).
Enable logging of tcp sequence numbers.
Log MAC values of incoming packets.
An output plugin for logging into a mysql database. This is only compiled if you have the mysql libraries installed, and the configure script was able to detect them. (that is: --with-mysql was specified for ./configure)
The plugin automagically inserts the data into the configured table. It connects to mysql during the startup phase of specter and obtains a list of the columns in the table. Then it tries to resolve the column names against keys of input plugins. This way you can easly select which information you want to log - just by the layout of the table.
If, for example, your table contains a field called 'ip_saddr', specter will resolve this against the key 'ip.saddr' and put the ip address as 32bit unsigned integer into the table.
You may want to have a look at the file 'doc/mysql.table' as an
example table including fields to log all keys from specter_BASE.so and
some from specter_HTTP.so. Just delete the fields you are not interested
in, and create the table.
The module defines the following configuration directives:
Name of the mysql database.
Name of the table to which specter should log.
Name of the mysql database host. If it's 'localhost' or undefined,
specter first tries to connect to local host by unix socket if possible.
Server port number for the tcp/ip connection.
Name of the mysql user, if ommited, the current user is assumed.
Password for mysql.
Size of a query buffer. You should set it only in a situation when you see "SQL buffer too small. Insert aborted." messages in your logs. Never try to lower this value below default, unless you really know what you're doing.
If this boolean option is set, MYSQL plugin will use SSL during connection to database.
Pathname to the key file.
Pathname to the certificate file.
Pathname to the certificate authority file.
Pathname to a directory that contains trusted SSL CA certificates in pem format.
List of allowable ciphers to use for SSL encryption.
An output plugin for logging into a postgresql database. This is only compiled if you have the postresql libraries installed, and the configure script was able to detect them. (that is: --with-pgsql was specified for ./configure)
The plugin automagically inserts the data into the configured table; It connects to postgresql during the startup phase of specter and obtains a list of the columns in the table. Then it tries to resolve the column names against keys of input plugins. This way you can easly select which information you want to log - just by the layout of the table.
If, for example, your table contains a field called 'ip_saddr', specter will resolve this against the key 'ip.saddr' and put the ip address as 32bit unsigned integer into the table.
You may want to have a look at the file 'doc/pgsql.table' as an
example table including fields to log all keys from specter_BASE.so and
some from specter_HTTP.so. Just delete the fields you are not interested
in, and create the table.
The module defines the following configuration directives:
Name of the postgresql database.
Name of the table to which specter should log.
Name of the postgresql database host. When undefined, specter try to connect to local database by unix socket.
Server port number for the tcp/ip connection, or socket file name extension for Unix-domain connections.
Name of the postgresql user, if ommited, the current user is assumed.
Password for postgresql.
Size of a query buffer. You should set it only in a situation when you see "SQL buffer too small. Insert aborted." messages in your logs. Never try to lower this value below default, unless you really know what you're doing.
If this boolean option is set, PGSQL plugin will use SSL during connection to database.
An output plugin that can be used to generate libpcap-style packet logfiles. This can be useful for later analysing the packet log with tools like tcpdump or ethereal.
The module defines the following configuration directives:
The filename where it should log to. The default is:
/var/log/specter.pcap
Set this option if you want to have your pcap logfile written synchronously. This may reduce performance, but makes your packets appear immediately in the file on disk.
This plugin behaves much like LOGEMU, but logs its input into syslog.
Two options are allowed:
Facility a message should be logged with. See syslog(3) manual page. specter accepts following facilities: deamon kernel, user, localx (where x is from 0 to 7).
Importance of a message. All standard syslog levels are allowed: emerg, alert, crit, err, warning, notice, info, debug.
Works the same way as ipt_LOG --log-tcp-options parameter. It enables logging of tcp options.
Log options from IP packet header (equivalent to --log-ip-options from ipt_LOG target).
Enable logging of tcp sequence numbers.
Log MAC values of incoming packets.
Comments / questions / ... are all welcomed.
Just drop me a note to ruby@joker.linuxstuff.pl.
If an error doesn't happen during compilation time, you are encoured to
get from specter as many information as you can. To do that configure it
with --enable-debug option enabled, and set loglevel (in global
options) to 1. Include information about your system (architecture,
libraries) and a description to help me in reproducing this bug.