RDBrowse 1.6
Tommy van Leeuwen

ABSTRACT

RRDBrowse is a poller daemon, templater and webinterface
for rrdtool. It works with small .nfo files which hold 
router information and optionally connection details, colors, 
min max, bandwidth settings, etc, etc. RRDBrowse uses a caching 
mechanism to store interface names. If you know MRTG and 
RRDtool, this is for you!

RRDBrowse Homepage: http://www.rrdbrowse.org/

Table of Contents:
    1. Dependencies
    2. Installation
    3. Upgrading
    4. Update Process
    5. Testing
    6. NFO Files
    7. PAGE Files
    8. Performance
    9. Cleaning Up
   10. Module List
   11. History
   12. References
   13. Copyright and License

Please see the Changes file for updates between versions.


1 DEPENDENCIES

This Perl module requires these other modules and libraries:

    RRDs (ships with rrdtool)
    Time::HiRes
    Data::Dumper
    SNMP_util
    Image::Magick
    mod_perl
    libwww-perl
    Digest::MD5
    Sys::Syslog


2 INSTALLATION

To install the Perl extension for RRDBrowse type the following:

    perl Makefile.PL

Check and fix all prerequisites which you are seeing. You can use
the CPAN module to install missing modules:

	perl -MCPAN -e'install Module::Name'

Make sure all your prerequisites are fixed and type:

    make
    make install

Copy the rrdbrowse.conf to only one of the following locations:

    /etc/rrdbrowse.conf
    /usr/local/etc/rrdbrowse.conf
    ~/.rrdbrowse.conf
    ./rrdbrowse.conf

Now, you're ready to create the directory layout.

It's preferrable to create the files in /usr/local/etc/rrd
so you don't have to update the configuration file. Any
other path is no problem ofcourse. The above directory will
be referenced to as basepath.

You must create the following directories under basepath:
rrds, nfos, pages and pngs. The basepath/pngs directory
should be writable by the user under which the webserver
runs and under the user the update daemon runs as. You must
specify the group under which your apache runs as in the
rrdbrowse.conf. The directory structure looks like:

    /usr/local/etc/rrdbrowse/
    /usr/local/etc/rrdbrowse/nfos/
    /usr/local/etc/rrdbrowse/pages/
    /usr/local/etc/rrdbrowse/rrds/
    /usr/local/etc/rrdbrowse/pngs/
    /usr/local/etc/rrdbrowse/cache/

The rrds and pngs dir mus be writable by the webserver. nfos and pages 
only have readonly access during a session.

All daemons and utilities need to have atleast one subdirectory
under which the .nfo files are stored. And example nfos
directory structure looks like:

    nfos/customers/cust1/cpu.nfo
    nfos/customers/cust2/load.nfo
    nfos/customers/cust2/bandwidth.nfo
    nfos/company/router1/bandwidth.nfo
    nfos/company/router1/load.nfo
    nfos/company/ascends/max1-users.nfo
    nfos/company/ascends/max2-users.nfo
    etc..

Now lets test if everything works.

Create a test .nfo file or copy and edit one from the examples
directory. Next execute the rbtest utility to test if all
is ok. The rbtest utility tests if it can read and parse
the nfo file, read the snmp data and create a graph.

    rbtest path/to/test.nfo

If all looks ok you are ready to put the update daemon in
cron every five minutes. Use a cron entry like:

    */5 * * * * root /usr/bin/rbupdate

You should check out the cron mail received because it can contain
errors. Probably it are snmp timeouts which can be ignored. Also if
for some reason an interface got deleted from your router 
cron mails are easy to track this kind of problems.

Setting up the Web Interface

You should set up a title and an image in the rrdbrowse.conf.
Next copy the rb.cgi to your cgi directory. Test it by executing 
rb.cgi from your website. If you want a bit more speed you can
enable mod_perl in your cgi directory. This generally gives you
a speedup of 30-50% for the website. If you don't want mod_perl,
you're finished now.

Create an apache virtualhost entry with mod_perl enabled. You can 
pick a documentroot to store it in one of your existing websites 
or you can create a virtualhost entry if you're using a dedicated 
rrd machine. An example is provided below:

    <VirtualHost rrd.chiparus.net>
       PerlFreshRestart On
       ScriptAlias /cgi-bin/ /usr/local/etc/rrd/cgi-bin/
       <Directory /usr/local/etc/rrd/cgi-bin/>
          SetHandler perl-script
          PerlHandler Apache::Registry
          Options +ExecCGI
       </Directory>
       DocumentRoot /usr/local/etc/rrd/cgi-bin/
       DirectoryIndex rb.cgi
    </Virtualhost>


3 UPGRADING

If you are upgrading from earlier versions than of 1.4 you should
make sure all options in the configuration file are ok. Especially
the 'showvars' is important. The rest will have reasonally default
values if not specified.

Most important is that since 1.4 there is only one CGI script
which does the complete website. The other single files which
build the mod_perl website in previous versions are gone. (That
includes all headers, footers and the stylesheet. Everything
is incorporated in rb.cgi). 

The original mod_perl website is not supported anymore.

Also the header/footer stuff in the configuration file is gone. 
This is replaced by 'title', which sets the pagetitle and 
'image' which in an html image tag for the image in the top 
right corner.


4 UPDATE PROCESS

The update daemon runs from cron every 5 minuts and parses every
.nfo file it finds in the nfo directory. The update daemon has a
number of threads it spawns which deal with a certain nfo file 
individually. A lot of time is spend with waiting for snmp response
however if you have a fast network and a slow server the rrd updates
may become important. On a normal network and server it should be 
able to handle at least 2500+ nfo/rrd updates every 5 minutes. 
A reasonabe recent fast PC may handle up to 10000+ updates 
every 5 minutes.


5 TESTING

A simple test script is available to test your nfo file creations.
To test a nfo name it .nf and use the test utility to test it. If
it works and no errors or bugs appear you have to rename the .nf
file to .nfo so the update process can find it. The test script can
also be used to test your modules.

Make sure you test and work on your files with .nf and rename them
later to .nfo to avoid cron emailing errors to you evey time. 


6 NFO FILES

The NFO files are text files containg one or more of the following
lines:

    Interface: The exact textual description of the interface.
    Target: community@ip.adres
    Bandwidth: does nothing
    Limit: Limit the graph to this amount of bps. 
    Description: The description, make them the same as your cisco desc!
    In: Description for the In (green) line of the default function (port.pm)
    Out: Description for the Out (blue) line of the default function (port.pm)
    Type: Which module you want to use (default: port.pm)
    Options: nowarn (don't log snmp timeouts)
    HRule: <value> prints a red line at value

Fields required for most modules are: Interface, Target and Type. Type defines
the perl module used for monitoring. See RRD::Info.pm to see a list of
allowed keywords.

Formatting: If the In: line is something like "from customer" and the Out:
line reads "to customer" it's wise to put a few extra whitespaces at the 
end to the shortest line (in this case Out:) to align the text in the graphs. 
Basically you should play a little with it to get a nice layout, but by 
all means, it's not necessary ofcourse. 

The In: and Out: keywords are only supported in port (and catalyst) modules.
See the examples directory for minimal required keywords for a module.

Please note that Bandwidth, Connection and Router are only descriptive
fields used only in the webinterface. They don't serve any purpose in
the rest of the program.


7 PAG FILES

A 'page' is simply a collection of nfo files and a title. A 'page' will show
up in RRDBrowse as a list of thumbnails linked to the detail files. You can 
place your page files in the pages directory or mix them with the nfo files
itself. The layout of a page is simple and all nfo files listed are relative 
to the basepath defined in the rrdbrowse.conf. An example page to list 
all your cpu's:

All CPU's Overview
customers/cust1/cpu.nfo
customers/cust2/cpu.nfo
customers/cust3/server1/cpu.nfo
customers/cust3/server2/cpu.nfo
etc..


8 PERFORMANCE

RRDBrowse, but especially RRDtool itself are responsible for great 
performance. Thousands of NFO files are no problem. Data which is
needed more than once during several updates is cached to disk. 
However, if you suspect a performance problem somewhere it might 
probably be due to the SNMP updates coming from a slow network or
slow server. If you want to have a look at how fast each nfo file
updates you may want to give the -p parameter to rbupdate.

    rbupdate -p

This will print the number of seconds for each update. If you plot
Apache performance statistics and have 5 nfo files for each apache
host, you can see the result of the cache too. The first update
will take much more time compared to the next updates from the 
same host.


9 CLEANING UP

If you change your nfo files, remove interfaces, routers, etc,
you can have outdated PNGs and cache data. To clear this data use
the rbclean utility.

    rbclean -p -c (-f | -d)

To clean up old PNG files use the -p option, to clean up old cache
data use the -c option. "Old" files are defined to be older than
24 hours. 

To clear all files, not just old files, use the -f (force) option. 
Warning: The force option can delete all of your cache which can 
make the next update round take a long time to complete! To see what
rbclean would do in that case, run it with the -d (dry run) parameter
so nothing gets deleted actually.

In your daily maintenance you can add "rbclean -pc" to have your old
and stalled data removed once a day. To quickly delete your png cache
in case apache or rrdtool don't refresh, use "rbclean -pf".


10 MODULES

Modules by default are installed in your $perllibdir/RRD/Function. 
A module can be referenced to by the Type: keyword in the NFO
files. The folowwing is a list of the module names together
with their description.

apabpr		Apache Bytes Per Request
apabw		Apache Bandwidth
apacpu		Apache CPU Usage
apaproc		Apache Processes
apaxs		Apache Accesses
apccurrent	APC UPS Output Current
apcload		APC UPS Output Load
apctemp		APC UPS Temperature
apctime		APC UPS Time of battery
ascinuse	Ascend ports in use
bindqrs		Bind Number of Queries
c72temp		Cisco c7xxx Temperature
carlist		Cisco CAR rate-limit
catalyst	Cisco Catalyst OS Port
catalyst64	Cisco Catalyst OS 64 Bits Port
ciscocpu	Cisco CPU Usage
memfree		Cisco Free Memory
ciscoppp	Cisco PPP Channels in use
errors		Line Errors
fsstat		Linux Openfiles (needs snmpd setup)
lincpu		Linux CPU Usage
lindiskblk	Linux Disk Blocks per Second
lindiskio	Linux Disk I/O's per Second
linload		Linux Load Average
linmem22	Linux 2.2 Memory Usage
linmem		Linux 2.4 Memory Usage
linofiles	Linux Open Files
linproc		Linux Number of Processes
linsockets	Linux Open Sockets
oidgauge	Generic OID Plotter
oidderive	Generic OID Plotter
port		Generic Port
port64		Generic 64 Bits Port
rtstats		Request Tracker Queue Statistics
tcpres		TCP Probe Response Time
telnet		Generic Telnet/TCP Interface
w2kcpu		Windows 2000 CPU Usage
w2kmem		Windows 2000 Memory Usage

Please reference to README.Modules to get a more detailed
explanation on installation of each module.

It should be noted that besides Linux, the lin* modules are 
actually net-snmp modules so most of them should also work on 
FreeBSD, Solaris, HP-UX, and any other system which runs net-snmp.

The following is a list of nice OIDs to use with the oidgauge module:

PIX Open Connections:	.1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.6
PIX Free Memory:	.1.3.6.1.4.1.9.9.48.1.1.1.6.1
PIX Used Memory:	.1.3.6.1.4.1.9.9.48.1.1.1.5.1

Basically anything can be plotted. See also the rrdtool homepage
to see things other people are doing with rrdtool.


11 HISTORY

First there was MRTG which does a nice job. But as the company
grows and grows we needed more power. Not only more power to 
handle more items, but also more configuration power. As well
for new clients, as well as things which could be graphed.

MRTG was first extended to print and plot all kind of information
we needed. It quickly evolved to a PHP interface. The NFO files
idea was born and by using a few shell scripts those NFO files
were parsed and piped to MRTG.

At some time later I discovered RRDtool and started to experiment
a little. A few Perl scripts were developed to handle SNMP requests
and put those in the RRD databases. Those Perl scripts also read
from the NFO files. Basically, this is were RRDBrowse was born. 

After a year or so I decided to rewrite and structure everything
and release it as what is now known as RRDBrowse. Nowadays it runs
a couple of hundred statistics in a few seconds at the company I
work for. Authentication was wrapped around and it is now part
of the customer portal too. 


12 REFERENCES

RRDBrowse wouldn't be possible without RRDtool, ofcourse. 
Download RRDtool from:
	http://www.rrdtool.org/

Download net-snmpd at:
	http://www.net-snmp.org/

Download windows SNMP for the public community:
	http://www.wtcs.org/snmp4tpc/

RRDBrowse homepage is located at:
	http://www.rrdbrowse.org/

RRDBrowse was made possible by Support Net B.V. who have an
excellent lineup of equipment where most of this software was
tested on initially. Visit Support Net at:
	http://www.support.nl/


13 COPYRIGHT AND LICENSE

Copyright (C) 2002 T. van Leeuwen <tommy at chiparus.org>
This software is licenced under the same terms as Perl are.

README $Id: README,v 1.24 2003/06/21 17:25:51 tommy Exp $

