Installation and Configuration
The Big Brother Systems and Network Monitor
------------------------------------------------------------------------
Installation and Configuration Manual
Version 1.5d2 - Oct 10th 2000
------------------------------------------------------------------------
1.0 Quick and Dirty Install for Big Brother
1.1 Installing Big Brother
1.2 Upgrading BB
2.0 Configuring the BB hosts file
2.1 The etc/bb-hosts file
2.2 Basic format of etc/bb-hosts
2.3 etc/bb-hosts special directives
3.0 Customizing your BB installation
3.1 The runbb.sh script
3.2 Configuring the etc/bbdef.sh file
3.3 Localizing programs paths
3.4 Creating BB clients
3.5 Configuring the Web Stuff
3.6 History
3.7 Adding your own tests
3.8 Starting BB at system boot
4.0 Configuring the Notification feature
4.1 A Little History
4.2 Setting up notification
4.3 Creating notification rules
4.4 Acknowledging notification messages
4.5 Manual notification (Web based)
4.6 SMS notification
4.7 Using kermit
4.8 Adding a custom notification
4.9 Notification/acknowledgement logs
4.10 Disabling notifications temporarely
5.0 Customizing your Big Brother display
5.1 HTMLized status logs
5.2 headers / footers
5.3 Creating skins
5.4 Inserting notes about your hosts
5.5 Extending the contents of the main HTML pages
5.6 Extending the contents of the history HTML pages
5.7 Working with historical logs
5.8 Hiding the bb.html/bb2.html page header
6.0 Miscellaneous Big Brother configuration options
6.1 Specifying filesystem specific values in the disk test
6.2 Enabling security
6.3 Using bbnet to test TCP services
6.4 Sending custom one line status with bb
6.5 Changing the 'purple' interval
6.6 Specifying processes to look for
6.7 Sending arbritary data to a BBDISPLAY(s)
6.8 Disabling the messages file(s) zero-length test
6.9 Setting a Time To Live to a status message
6.10 Disabling the connectivity test
6.11 Reducing connectivity errors
6.12 Setting up BBPAGERs and BBNETs host in failover mode
6.13 Speeding up the network tests
6.14 Enabling WML ouput (WAP browsing)
6.15 Having BB notify you to reboot a server after is has been up XXX days
7.0 Other documentation sources
------------------------------------------------------------------------
This Installation and Configuration Guide is Copyright 1997-2000
by The MacLawran Group Inc. This document may be reproduced, so
long as it is kept in its entirety and in its original format.
------------------------------------------------------------------------
------------------------------------------------------------------------
Section 1: Quick and Dirty Install for Big Brother
1.1 Installing Big Brother
WARNING: For security reasons, it is best to install and run
BB as its own user and not as root.
*** READ THE README.SECURITY FILE BEFORE PROCEEDING ***
You've extracted the BB archive and this is the directory
structure that was created:
bbnewversion/ (i.e. bb15/) referred as $BBHOME
bbvar/acks/ referred as $BBACKS
/data/ referred as $BBATA
/disabled/ referred as $BBDISABLE
/hist/ referred as $BBHIST
/histlogs/ referred as $BBHISTLOGS
/logs/ referred as $BBLOGS
1. cd bbnewversion (i.e.: cd bb15)
cd ./install
./bbconfig <OS-NAME>
where OS-NAME is bsdi sco3 sco freebsd solaris hpux9 hpux
linux sunos netbsd osf ultrix irix
unixware redhat aix dynix debian dgux
caldera mandrake
Note that <OS-NAME> is optional, BB will try to figure it out.
If you are running Linux you may have to provide the proper
distribution name:
./bbconfig readhat
./bbconfig debian
./bbconfig caldera
./bbconfig mandrake
./bbconfig linux
If you're not running one of the above, read the install/README
./bbconfig will ask you questions about your setup
be ready with:
BBHOME: directory where BB resides
(usually the choice given by BB is correct)
Should the old-style structure be kept
If you intend to use FQDN(Fully qualified domain names)
Which host(s) is(are) the BBDISPLAY(s)
Which host(s) is(are) the BBPAGER(s)
If the current host is a BBDISPLAY/BBPAGER
Default e-mail recipient for notification
URL you intend to view BB with
URL of the BB CGI scripts
The user id of your web server
2. cd ../src
type "make"
then type "make install"
If you have trouble compiling, refer to the README file
located at src/README
for those who are upgrading then go to "upgrading BB" section
3. cd ../etc
edit bb-hosts, put your hosts names in there. Refer to install/README.
This is the core of Big Brother. You must read the docs here.
edit bbdef.sh, set alarm levels and things. If you want to use
fully qualified domain name hosts then make sure you set FQDN=TRUE
in etc/bbdef.sh. If this is the first host you install and intend
to use it as the display/notification server as well as the server
that tests the network services then your bb-hosts should contain
this line:
xxx.xxx.xxx.xxx this.host.name # BBDISPLAY BBPAGER BBNET
4. ./bbchkcfg.sh
Checks the bbdef.sh/bbinc.sh/bbsys.sh source scripts for invalid entries
./bbchkhosts.sh
Checks the bb-hosts file for errors
5. cd ../..
ln -s bbdir bb where bbdir is the new version
directory (e.g. ln -s bb13a bb)
This is useful as you don't have to change the directory in your
startup script. (see section 11)
cd bb
chown -R bbuser . where bbuser is the user you defined
in the install process. This makes sure
that the bbuser can write/read into
the BB directory structure as you will
probably install it while in the root account.
cd ..
chown -R bbuser bbvar
6. ln -s /full_path_to_bb/www /WWW/bb (where /WWW is the Document Root dir).
Make sure the permissions are correct. Configure your web server
for this directory if need be.
Also make sure that your web server follows symbolic links
7. cd bbdir where bbdir is the new version directory (e.g. bb13a)
or
cd bb if you've followed section 5
./runbb.sh start
examine the BBOUT file for any errors
NB The HTML summary pages (bb.html/bb2.html) should be available
2 minutes after the startup. Don't panic the'll be there if
you are patient. In your browser, you should be able to
see the results at http://yourwebhost/bb/ or
http://yourwebhost/bb/bb.html (assuming you used /bb as BB URL root)
8. Debug, and look at all the docs.
9. Check the online documentation! It lives under:
http://bb4.com/bb/bb-help.html
10. This will have enabled you to have a display/pager server set up.
At this point no clients are running. When you have all of
your hosts defined in etc/bb-hosts then
use install/bbclient to create a tarball for BB clients of the same
OS/HW type. If you have different OS/HW platforms then
reinstall BB on each one (then use the bbclient to create a tarball
for each identical OS/HW client) and don't forget to copy your master
bb-hosts file to it. Run through the install procedure to make
sure that the clients are also installed properly(bbchkcfg.sh/bbchkhosts.cfg)
11. If you wish to start BB automatically at startup, we suggest the
following command in your startup script:
su - <bbuser> -c "cd <BBHOME>;./runbb.sh start"
<bbuser> is the user you chose at install
<BBHOME> is the directory in which you installed BB
or where you've created the bb directory link
as per section 5.
If you're new to BB, we suggest that you subscribe to the
BB mailing list. To do so, follow these instructions:
Send an e-mail to MajorDomo@bb4.com
In the body of the e-mail message (not the Subject line),
place the statement:
subscribe bb
Nick Silberstein has made an archive of the Big Brother
mailing list available. It can be found at the URL:
http://www.tpdinc.com/~bb/
When you have finished installing BB on all of your
monitored hosts, you might want to check out the FTP archive
that contains many contributed scripts. Check it out
at ftp://ftp.deadcat.net/pub/BB
Sans '98 & '99 presentations are available at
http://bb4.com/propaganda.html. These presentations
may fill in some details.
1.2 Upgrading BB
If you are upgrading from a previous version then do these steps after
the preliminary install of BB:
If your previous version of BB was pre-1.5 then follow these next steps
otherwise jump to 1.2a.
We assume that the BBVAR/BBLOGS/BBHIST/BBHISTLOGS/BBDATA/BBDISABLE
weren't modified in $BBHOME/etc/bbinc.sh.
cd BBTOP/bbvar (BBTOP is where you unpacked BB)
rm -r logs acks hist histlogs notes data disabled
mv OLDBBHOME/www/logs .
mv OLDBBHOME/www/acks .
mv OLDBBHOME/www/hist .
mv OLDBBHOME/www/histlogs .
mv OLDBBHOME/www/data .
mv OLDBBHOME/disabled .
1.2a BB upgrade steps
cd <bbdir> where <bbdir> is the directory of the new install
(where BBHOME points to)
cd www
rm -r notes
# Replace OLDBBHOME by the full path of BBHOME defined in your previous install
mv OLDBBHOME/www/notes .
cd ..
cd ext If you have external scripts
cp OLDBBHOME/ext/* .
cp OLDBBHOME/ext/hist/* hist/
cp OLDBBHOME/ext/rep/* rep/
cp OLDBBHOME/ext/mkbb/* mkbb/
cd ..
cd etc
cp OLDBBHOME/etc/bb-hosts . Depending of your current setup
cp OLDBBHOME/etc/bbwarnrules.cfg . copy and edit appropriate config files
cp OLDBBHOME/etc/bbwarnsetup.cfg .
cp OLDBBHOME/etc/bb-dftab .
cp OLDBBHOME/etc/bb-proctab .
cp OLDBBHOME/etc/security .
vi bbdef.sh Set environment variables for your setup
like FQDN, BBEXT, ...
vi bbinc.sh It is always useful to run a diff getween
your old version of bbdef.sh/bbinc.sh
to view your own changes
cd ../..
if you already have a link called
bb used to access your BB install
rm bb
ln -s <bbdir> bb Setup bb as a link to the new bbdir
directory such that it's easier
to refer rc scripts to bb/ without
having to change them every time
you upgrade.
Finally don't forget to move over the extensions you've installed
(lardd, "The State", ...)
------------------------------------------------------------------------
Section 2: Configuring the Big Brother hosts file
2.1 The etc/bb-hosts file
The etc/bb-hosts file controls where Big Brother looks for things
and the actions that are taken. The format is identical to the
standard /etc/hosts file, except with additional directives
for Big Brother.
2.2 Basic format of etc/bb-hosts
Lines are of the format:
IP-ADDR HOSTNAME # DIRECTIVES
IP-ADDR: XXX.XXX.XXX.XXX
HOSTNAME: host.domain.com if FQDN="TRUE"
host if FQDN=""
FQDN (display with Fully Qualified Domain Name) is a
variable that you set in the etc/bbdef.sh file.
Directives the Big Brother knows about are:
BBDISPLAY This host displays the HTML results
You need a Web server on this host
BBPAGER This host acts as the notification server
You can have multiple BBDISPLAYs & BBPAGERs. Just
put the BBDISPLAY/BBPAGER directive on the config line
for each host that acts as BBDISPLAY/BBPAGER.
BBNET Indicates that this host checks
the ip network services
Many hosts can act as a BBNET. If you do have more than
one BBNET, make sure that the bb-hosts file only
contains the hosts that you will test against. You must
be very careful because if a host is tested by many
BBNETs then the status display for the services on
that host will only show that last test status.
You may get errors for that host even though it shows
green, it's just that another BBNET send a status message
afterwards and that message returned OK. On the other
hand you may want to have tests from different location :)
http://www_path The host is tested for http connections
using the www_path
https://www_path The host is tested for https connections
using the www_path
(requires lynx: http://lynx.browser.org/ or curl: http://curl.haxx.nu/ )
Note that you can specify multiple URLs by joining the
URLs with '|':
http://www_path|http://www_path1
or by specifying them individually:
http://www_path http://www_path1
ftp Test for ftp service
smtp Test for smtp server
pop3 Test host for pop3 server
telnet Check telnet service
ssh Test ssh server
nntp Test nntp news server
Check /etc/services for proper service name
especially pop3 (sometimes referred as pop-3).
These directives names MUST appear in /etc/services
Any text based service can be checked by putting
the /etc/services name as the directive and adding
in the list of services in bb-network.sh
The '!' can be prefixed to a service such it will
be checked that the service is NOT running (!ftp).
The '?' can also prefix a service. This prefix
indicates that the service is considered a
dialup service: it will generate a clear button
if the service is down.
dns Checks for name resolution server
dig Same check as dns but using the dig
command if the command is available.
noping Don't do ping test for this host
noconn Don't do ping test for this host and
don't generate a clear dot
dialup If host is down then display clear button
otherwise display a green status
You can add your own directives that you access through an
external test. Known directives from user contributed
tests: oracle/fping/trap/sybase/... These contributions
can be found on the FTP archive.
This is a sample bb-hosts file:
#
# BIG BROTHER HOSTS FILE
#
192.168.110.102 bobo # BBDISPLAY BBPAGER ftp smtp pop3
192.168.110.95 admin # http://admin/
192.168.110.108 mynet # BBNET nntp!
bobo is the display and notification server.
mynet is the host that runs ip network services checks
it will check the ftp/smtp/pop3 services on bobo,
then the http service on admin and finally it makes sure
that nntp is NOT running on mynet
You can also group hosts within a seperate HTML table. There
are three group tags defined: group, group-compress and group-only.
Results can also be saved to a specific page using the 'page' directive.
It is best to have the master bb-hosts file identical on all
BB server and client hosts but it is not required (unless
you have multiple BBNETs, see previous comment)
Changes to the etc/bb-hosts file are automatic. You do not
have to restart BB to initialize the changes.
2.3 etc/bb-hosts special directives
2.3.1 Grouping on the display
The Web display may be broken into tables to create
a more aesthetic and sensible display. It's also much faster
to load small tables, than one giant table. So consider
grouping your etc/bb-host entries logically and separating
them using the "group" , "group-compress" or "group-only" directives.
group Intranet Servers
The group directive defines a block of hosts to be grouped
in the same HTML table. All hosts lines following the group
directive, until a new group/group-compress is defined, belong
to that group. The text that follows the directive is the
title given to the HTML table. Note that you can embed HTML
code in the title: Italic / H3, but use with caution.
group-compress California Servers
The group-compress is identical to the "group" directive
except it will only display services (columns) containing data
for that group.
group-only conn|cpu|disk Restricted Services
The group-only will create a table with only the colums
defined in the directive. The columns are '|' delimited.
2.3.2 DHCP hosts (no fixed IP address)
0.0.0.0 dhcphost.domain #
The 0.0.0.0 indicates that this host is a DHCP
host and that the BBNET host will not try to run the
connection test on that host.
2.3.3 Modem Banks
dialup modem-bank 204.19.116.20 4
The dialup directive (not to be confused with the other
dialup tag which displays a clear button if the host is down)
is used to specify connectivity for a bank of modems. The
2nd parameter is the name to be displayed on the display page.
The 3rd argument is the starting IP address of the modem bank.
The last argument is the number of modems on that bank.
2.3.4 Summary Lines
summary quebec.mtl 255.255.255.255 http://cafe.domain.com/bb/
This indicates to the BBDISPLAY machine that
summary information about the state of this display is to
be forwarded to the IP address noted on the line. The
summary can be sent to more than one parent machine. You
can send the generated bb.html/bb2.html or subpages created
with the 'page' directive to any other BBDISPLAY host(s).
2.3.5 Display results in an HTML subpage
page nyrouters New York Routers
BB always creates two HTML pages: bb.html/bb2.html.
When the page directive is encountered, BB will then create a new
HTML subpage until another page directive is found and will save
the output from that point on to that new subpage. It will keep a pointer
of the new sub page in bb.html. The first argument is the directive,
the second one is the name of the page (i.e. nyrouters.html)
and the remaining arguments is the caption that will appear
in bb.html when it creates the link to that subpage. Always
place your page directives after all hosts that you want to
appear in the bb.html page. When you use a page directive
then all subsequent ouput will be in a subpage.
N.B. summaries and dialup directives will always appear on
bb.html/b2.html and not in any HTML subpages
------------------------------------------------------------------------
Section 3: Customizing your BB installation
3.1 The runbb.sh script
The runbb.sh script is the master BB script. It sets up the
BB environment with the values from the configuration files.
You MUST set the BBHOME variable to the directory path where
you have installed BB. Without it, BB will not run properly.
cd $BBHOME (to your BB install directory)
To start BB:
./runbb.sh or ./runbb.sh start
To stop BB:
./runbb.sh stop
To restart BB:
./runbb.sh restart
When BB starts/restarts it checks that there is no processes
already running. If this is the case, BB will complain but
will continue to start as it cannot determine if the process
was really started by BB. BB checks for any processes that
contain the BBHOME in its definition.
If you ever have to add a script that runs as a deamon use
the following template to add it in runbb.sh:
echo "Starting snmptrapd"
echo "Starting snmptrapd" >> BBOUT
MIBFILE=$BBHOME/cmu/etc/mib.txt
export MIBFILE
$BBHOME/cmu/bin/snmptrapd -B | $BBHOME/bin/bb $BBDISP - &
set `$PS | $GREP snmptrapd | $GREP -v "$GREP"` > /dev/null
BBPID="$BBPID $1"
This is from an SNMP trapd extension available at the FTP
archive site (ftp://ftp.deadcat.net/pub/BB). You'll notice
that the PID is added to $BBPID, This way, you'll be
able to stop the extension at the same time as BB when
executing "runbb.sh stop".
3.2 Configuring the etc/bbdef.sh file
etc/bbdef.sh is where you configure how Big Brother reacts to a
variety of situations that may arise. You can establish levels
for warnings(yellow), panics(red) values.
Please note that any modifications to this file do not come
into effect until the next restart of BB. It is best to
stop/start BB immediately after this file has been modified.
Use FQDN (fully qualified domain naming)
FQDN="" - Host names rendered on the BB status pages will only
include the host name, not including the domain
name.
FQDN="TRUE" - Host names rendered on the BB status pages will
include the host name and the domain name.
Which program to use to check the HTTP connectivity:
LYNX="$BBHOME/bin/bbnet" - This indicates where bbnet (a C program
that checks the IP services) is located.
LYNX="/usr/local/bin/lynx -dump -head" - Use lynx if you need to
check password protected pages. Please use the proper
directory path as per your host installation.
NONETPAGE defines which IP services not to notify on if an error
is encountered. Include the services as they are named
in the etc/bb-hosts file. This is only valid on a host
defined as a BBNET host.
BBTMP indicates the location of the BB temporary directory. You
may want to locate it outside of the BB structure.
The DFWARN and DFPANIC variables contain the % levels at which
the disk test will fail. By default DFWARN is 90% and
DFPANIC is 95%. These values can be overriden by values
in the etc/bb-dftab file described in section 5.2.
CPUWARN and CPUPANIC are default levels based on the load
average. CPUWARN is set at 150 and CPUPANIC at 300.
These values are the load average (from uptime)
multiplied by 100.
You can define the processes for which you want to monitor if
they are up and running. You do this by setting the
PROCS and PAGEPROC variables. The processes included
in PROCS will lead to a yellow condition if one or more the
processes is down. Processes in PAGEPROC will generate
a red condition if a process is down.
MSGS and PAGEMSGS are used to monitor error messages in the
system logs. MSGS contain keywords to look for in
the system logs. If a keyword is found then this
will create a yellow condition and if the keyword
is in the PAGEMSGS variable then it will be upgraded
to a red condition. You can also discard certain
messages by defined portions of that message in the
IGNMSGS variable. You can define multiple messages
by delimiting each message by a ';'.
BBWEB="/bb"
This variable defines the path to the root page of BB.
Make sures it never ends with "/". All locations
in BB are referenced from that location.
BBWEBHTMLLOGS="${BBWEB}/html"
This variable defines the path to the HTMLized status logs.
This path is concatenated with the name of the HTMLized
status file and is appended to the e-mail notification
message for a quickpath to the error information.
The PAGELEVELS variable indicates to the 'bb' process at which
level(s) it should send a notification message. The 'bb'
program is used to send status messages and if the status
level of the message is found in the PAGELEVELS variable
then a notification message is also sent to the BBPAGER
host. This is the default value. It can be overridden
by the 'pagelevels:' token in the etc/bbwarnsetup.cfg
file. The PAGELEVELSMAIL contains the color level(s) on
which only e-mail is to be sent. The color level must also
be found in PAGELEVELS.
The WEBHISTORY defines if you want to use the web-based history
display script. This script gives you a 24hr HTML chart
and the last 50 status changes.
The SAVESTATUSLOG variable defines if the historical log of
the host.service is kept or not. The logs are saved
in the $BBHISTLOGS directory.
The BBLOGSTATUS variable determines if each log received is
rendered into a static HTML page, dynamically by CGI or
not at all (it appears as plain text). If it is
set to DYNAMIC then a CGI will create the HTMLized status
log only when requested. That's when you click on a
colored dot on the BB display pages. The dynamic CGI
is called bb-hostsvc.sh. If is set to DYNAMIC then
a static HTML status log will be generated the www/html
directory. If it is set to TEXT then no HTMLized status
logs are available. The status will appear as plain text.
NOPINGCLEAR lets you decide if you want clear dots for host that
have the noping directive. If you set to TRUE then clear
dots will be displayed for hosts with the noping as
well as hosts with IP address 0.0.0.0 (DHCP hosts or
hosts accessed only with its full name). You can use the
noconn directive in conjunction with NOPINGCLEAR to select
which hosts you don't want a clear dot.
The IPTEST_2_CLEAR_ON_FAILED_CONN variable, if set to TRUE, defines
that if a network test fails and that the connectivity (ping)
test also failed then display the test results as clear. You
can turn off this setting individually by test by using the
~ modifier.
in etc/bbdef.sh:
IPTEST_2_CLEAR_ON_FAILED_CONN=TRUE
in etc/bb-hosts
123.123.123.123.host1 # ftp
If the connectivity test to 123.123.123.123 failed and
the ftp test also failed (very likely) then display the
test with a clear dot. If you had set the directive to
~ftp then it would have been displayed as red.
3.2 Localizing programs paths
Sometimes the OS specific programs paths found in the
installation might be wrong for a particular host(s) OS.
Change the paths in the etc/bbsys.local file. The variables
contained in this file all have default values from the
bbsys.OS file. The original bbsys.local is copied from the
etc/bbsys.OS file during installation (bbconfig); so keep track
of your changes as they will automatically be overwritten if you
re-run bbconfig.
3.4 Creating BB clients
Creating BB clients is very simple:
cd $BBHOME
cd install
./bbclient <client-hostname>
<client-hostname> must be found in the etc/bb-hosts file.
This will create a tarball for the client that has the same
OS/HW platform type as the host on which the bblient command
is executed. The tarball is found in the directory above
$BBHOME.
Don't forget to configure the PROCS and PAGEPROC
variables in etc/bbdef.sh on the client after you've untarred the
archive such that the correct processes are monitored.
If you have different OS/HW platforms then install BB,
as a client, on each OS/HW platform (then use the bbclient to
create a tarball for each identical OS/HW client) and don't forget
to copy your master bb-hosts file to it. Run through the install
procedure to make sure that the clients are also installed
properly (bbchkcfg.sh/bbchkhosts.cfg).
3.5 Configuring the Web Stuff
All Web-based things live in the $BBHOME/www directory.
In order for the bb stuff to work correctly, this directory must
be linked into your Web site somewhere. I suggest something like
the following:
ln -s /home/sean/bb/www /WWW/bb (where /WWW is the Document Root dir
of your web server).
You should then be able to access BB with the URL
http://.../bb/bb.html
where ... is your webserver address.
You also have to configure the script which generates the
web-based history page if you set the WEBHISTORY variable to TRUE in
etc/bbdef.sh. This page is accessible from the "History" button
when you click on a colored dot in one of the display pages
(bb.html/bb2.html). You must copy the web/bb-hist.sh script to your
/cgi-bin directory and set the BBHOME.
Make sure the script has the proper permissions to execute: system and
web server permissions. If you want only a text based history display
then set the WEBHISTORY variable to "FALSE" in etc/bbdef.sh.
Set the BBHIST_IGNOREBLUE variable to TRUE in etc/bbdef.sh if you
don't want the blue state to be used in the calculations of daily % use.
By default the CGIBINURL (in etc/bbdef.sh) points to /cgi-bin
but if your web server cofiguration has a different CGI directory
then change it here or your history feature won't work.
If you intend to enable manual notification/acknowledgement
("PAGE/ACK"button in bb.html/bb2.html) then copy the web/bb-ack.sh
script to you /cgi-bin directory and set the BBHOME
variable in the script. Make sure the script has the proper
permissions to execute: system and web server permissions.
Acknowledgments generate a log entry in $BBACKS/acklog file.
3.6 History
Each time a status log is received it is checked against its
previous status to see if it has changed. If it has, then
the duration of the previous status is recorded and a new line
is appended to the history file with the timestamp of the
new status.
There's a history file for each host.service (status log) received.
The history files are located in the $BBHIST directory.
Here's a sample:
Sun Jan 24 18:30:45 1999 green 917220645 232068
Wed Jan 27 10:58:33 1999 red 917452713 1560
Wed Jan 27 11:24:33 1999 green 917454273
Notice that last line doesn't have a duration because it is
the current status. The duration will be appended only when
the status changes.
If you do not wish to have historical data then rename
$BBHIST to something else or remove it.
3.7 Adding your own tests
You can easily add your own tests. Start with the template
available at ext/template, add your code. Look at bb-local.sh,
bb-network.sh for an example on how to send data to BB. Then
in bbdef.sh, specify the name of your script in the BBEXT
variable. Restart BB and your test should be running. But
before you use it within BB, I suggest you test it for errors
by using the method:
cd /home/bb (or wherever your BB is located)
BBHOME=/home/bb
export BBHOME
. ./etc/bbdef.sh
cd ext
./yourexternaltest
Look for errors, fix them, rerun your test until you're
satisfied, then update bbdef.sh. Note that all temporary
files should be created in $BBTMP and make sure you remove them
after use. Also, remember that you don't have to deal with
sending notification messages, the 'bb' process send a 'page'
type message to the BBPAGER host when the status color is
found in the PAGELEVELS variable defined in bbdef.sh.
When it's ready don't forget to update the svcerrlist token
in the bbwarnsetup.cfg file on your BBPAGER host. You must
assign a numeric code to your column name.
You can also set the lifetime of the status sent by your
script by following the instructions described in
section 6.8: "Setting a Time To Live to a status message".
3.8 Starting BB at system boot
Depending on U*X operation system version (Linux/BSD/Solaris/...)
your startup procedures will vary from OS to OS. In short, you
have to create a startup script that has a start/stop
capabilities. Under a lot of OSes, you'll want to create your
script in the init.d directory and create an S89bb link in rc3.d that
links to init.d/S89bb (you may also want to create a K11bb link in
rc3.d that'll be used when the system shuts down). Use an existing
startup script has an example and substitute with these commands:
To start BB, use this command:
su - <bbuser> -c "cd <BBHOME>;./runbb.sh start"
or
su - <bbuser> -c "cd <BBHOME>;./runbb.sh restart"
To stop BB, use this command:
su - <bbuser> -c "cd <BBHOME>;./runbb.sh stop"
<bbuser> is the user that BB will execute has
make sure that bbuser has all permissions under BBHOME
<BBHOME> is the location of your BB install
e.g. su - bb -c "cd /home/bb;./runbb.sh start"
------------------------------------------------------------------------
Section 4: Configuring the Notification feature (BBWARN)
4.1 A little History
In the Beginning :), BB could only e-mail and notify by pager.
The configuration required that all BB hosts (servers and
clients) have the PAGER variable set with the recipients of the
notifications. There were no possibilities of customizing the
notification feature. This led to the:
The Pager Protection Act of 1997
================================
Use of BB has been linked to the untimely death of at least one
pager in Texas. In order to protect pagers from their users, a
notification scheme has been created to allow to specify a delay
between the notifications.
But this wasn't enough. BB netizens got restless and finally a
solution was delivered by Robert-Andre Croteau, and described in
an article in SysAdmin magazine. It was called BBWARN and it was
good. It allowed an admin to specify rules which enable him/her
to get notified based on the source of the error (host) and the
service involved and also based on the day/time. Different rules
could be set up and different recipients could be specified
depending on the characteristics of the problem
(host/service/day/time).
Numerous changes have been made and various configuration
options are available to the BB admin(s).
Notification methods
You can currently get notified by email, numeric pager and
SMS communication device. Users have reported custom
notification by alpha-numeric pagers and other PCS devices
(some are available on the FTP archive or a search of the
mailing list archives will lead you to them).
To use the email feature, your BBPAGER host (in bb-hosts)
must have an email program. This is not a problem on UNIX:
the mail (mailx) program is available.
Sending a notification message to a numeric pager requires
kermit and a modem. The path of kermit should be defined in
the KERMIT variable of bbsys.sh/bbsys.local.
4.2 Setting up notification
All of the BB notification configuration is specified in the
etc/bbwarnsetup.cfg file. Here are the various options that
can be set. Note that instructions are included in the
configuration file as comments.
All tokens in the etc/bbwarnsetup.cfg file must be defined
on the BBPAGER host. ***** The 'pagelevels:' token must be
defined on all BB hosts (servers & clients) because the 'bb'
executable sends the notification and that is done locally
on each client host. *****
bbwarn: Set to "TRUE" is you want to enable the notification
feature
svcerrlist: This contains a list of service/code pairs. The service
name is the column found in the HTML display and
the code is the value displayed on a numeric message.
If you add a custom check then the column name
must have a corresponding code.
ignforall: Is a regular expression that is used to temporarely
disable notification a host.service combinations
e.g. .*.cpu|.*.msgs|host11.*
Don't notify for any cpu/msgs errors and any errors
concerning host11.
N.B. This feature can be duplicated but the ! rule (below).
pagehelpcode: Numeric code to use when a user sends a manual
notification. See section 4.5.
ttyline: List of modems devices. You can specify more than one.
prefix: Prefix to use when dialing out
suffix: Suffix to use when dialing out (like a hangup)
pagelevels: At which color level to send a notification.
This token MUST be set on every host (clients & servers).
You must restart BB if you modify it.
pagerecovered: Set to TRUE if you want to be notified
when a problem has been fixed. This feature is only available
when 'pagetype:' is set to EVENT.
pagetype: Defines how the pager delay is handled. There are
4 choices (RCPT|EVENT|HOST|GROUP).
RCPT: the recipient is not notified until pager delay
expires.
EVENT: the recipient is not notified for a particular
host.service combination until the pager delay
expires.
HOST: the recipient is not notified for a particular
host until the pager delay expires.
GROUP: the recipient is not notified for a particular
host.service within the same etc/bb-hosts group
combination until the pager delay expires.
pagemaster: Recpient(s) to receive an e-mail notification if
a page notification could not be sent.
pageaddhtmlpath: Set to TRUE if you want the HTML path of
the status log to be appended to an e-mail
notification. If this is set to TRUE, make sure the
BBWEBHTMLLOGS variable is set correctly in etc/bbdef.sh.
cfgdelim: entry delimiter in the etc/bbwarnrules.cfga
briefrcpt: define the recipients that should receive a brief
notifications message. You can you use regular expressions:
ie. ep-* (all e-page recipients)
This expression will be expanded to this regular expresssion: ep-.*
hg-xxxxxxx: You can have multiple hg-xxxxxxx tokens that
define a group of hosts/devices. So you can
e.g. create hostgroup 'hg-routers' as an alias or
shorthand of a list your routers.
These tokens can then be specified in the bbwarnrules.cfg
file in the host-fields columns (1st/2nd column) instead
of having to type all of them individually in all rules lines.
pg-yyyyyyy: You can have multiple pg-yyyyyyy tokens that define a
group of recipients. These tokens are expanded in the rules
lines inthe recipients definition.
On the client make sure that PAGELEVELS is set (etc/bbdef.sh) and
on the server, if appropriate, that PAGELEVELSMAIL(etc/bbdef.sh) is set.
Refer to section 3.2 for more details.
4.3 Creating notification rules
You define the notification rules in the etc/bbwarnrules.cfg file.
Rules are written in the following format:
hosts;exhosts;services;exservices;day;time;recipients
hosts: match on these hosts (* is a wildcard for all hosts)
exhosts: exclude these hosts
services: match on these services (* is wildcard for all hosts)
exservices: exclude these services
day: 0-6 (sunday-saturday)
time: 0000-2359
recipients: email address, numeric pager, sms number,
qpage recipients are defined with qp-<recipient>
sendpage recipients are defined with sp-<recipient>
e-pager recipients are defined with ep-<recipient>
SNMP traps are sent with trap-<destination IP address>
(note that SNMP trap support is preliminary and may be subject to change)
There's also a special format of the rule line:
!hosts;exhosts;services;exservices;day;time;recipients
If a rule line starts with ! the event thats matches
the rule line will disable notification to any
recipient defined on that rule line.
If the recipients field is '*' then no notification
will occur for that event. Here's an example
!*;;*;;*;*;*
This will in effect disable all notifications and render
useless any other rule that you have defined :)
!*;;*;;*;*;robert@localhost 9999999
This will remove robert@localhost and 9999999 from the
list of recipients for the current event if they were
defined on another rule that matched the same event.
The value set by 'cfgdelim' (in bbwarnsetup.cfg) is your
field delimiter. In this case the value is ';'
Even though egrep regular expressions are allowed, do not use
the .* construct, just use '*'. It will be replaced with
.* in the regexp. It's just that * is more readable than .*
Also the default 'pagedelay' value (see bbwarnsetup.cfg)
which indicates how long before the next notification occurs
can be overridden on the rule line for a specific recipient
by appending the time value to the recipient:
recipient:XX where XX is the value in minutes.
For examples and complete information please refer to the
etc/bbwarnrules.cfg and the etc/bbwarnsetup.cfg files.
Escalation
To escalate a notification, you use the following format
for the recipient:
recipient:^XX[-YY]
XX is the initial wait before sending the notification
YY is the delay for each subsequent call. If it is
not specified then the 'pagedelay' value is used.
An escalation can only be acknowledged by the
recipient.
Initial delay
An initial delay can be specified when configuring a recipient.
The format is:
recipient:~XX[-YY]
XX is the initial delay before sending the notification
YY is the delay for each subsequent call. If it is
not specified then the 'pagedelay' value is used.
An initial delay can be reset if a recipient
acknowledges a notification for all recipients of
that notification (see below).
There's a simple checking program that validates the contents
of your bbwnarnrules.cfg file. It is located in $BBHOME/etc.
cd $BBHOME/etc
./bbchkwarnrules.cfg
Note that it only checks if your config file is not missing
a column.
4.4 Acknowledging a notification
When an admin is notified, the admin is always sent an
acknowlegement tag with the message. This tag number
is seven digits: the first five digits are a unique number
and the last two digits are the recipient's ID for that
notification (the recipient's ID can change, it is valid
only for the current notification).
In an e-mail notification, the ack number is in the
subject and the body of the message: in the subject
it is found at the beginning as in "!BB - XXXXXXX!",
and in the body of the message it is at the beginning of
the body delimited by [XXXXXXX].
In a numeric page, the acknowledgement is always after
the BB numeric code. It is the last 7 digits of the
numeric message. NOTE, if your pager doesn't support
more than 21 digits, then you're out of luck, you'll
be missing some digits. You could always try to
shorten the ack number but that is left as an exercise
for the reader ;-). If you get the first 5 digits
then you can always use recipient 99 to acknowledge
for all recipients but that may not fly at your site
especially if you have multiple recipients.
4.5 Manual notification (Web based "PAGE/ACK" button)
You can enable the manual notification feature and
let the user reach you by pager/e-mail if they can't get
a hold of you. If this is enabled, then you'll
receive a numeric message starting with 911 (pagecode
in etc/bbwarnsetup.cfg) followed by the phone
number to call. Copy the web/bb-ack.sh script (if
it was not copied during the installation)
to your cgi-bin directory and make sure all permissions
are correct.
You can setup who gets called when, by adding rules
to the etc/bbwarnrules.cfg file. The pseudo host to use
is notify-admin*. There's an example in the config
file.
4.6 SMS notification
This a user custom notification scheme for SMS devices.
What follows is the README file for his custom hack.
Last updated: 1997-01-11 by jaclu@ibk.se
This section describes the usage of a kermit script that
sends SMS messages.
It is used as a mail -> SMS gateway, and also for the "Big Brother"
network monitor(obviously).
The kermit script for SMS can be found in etc/sms.scr
My operator uses NOKIA server software, hopefully you can use
this also on Motorola servers, without too much hassle.
Terms
SMS Short Message Service
SMSC Short Message Service Center
CIMD Computer Interface to Message Distribution
Syntax
kermit sms.scr device file number [number [number [...]]]
device is modem device like /dev/cua0
The file should contain the message, only first line of file
will be sent, and it MUST be terminated with LF, otherwise the
kermit script won't be able to extract the message.
SMS messages can only be 160 chars, and since nobody would want
a lengthly message on a 8 character display, I haven't bothered
to handle message splitting. If message is too long, it is truncated.
If one or more SMS phonenumbers can be specified, the message will be
sent to each one of them.
Configuration
I have done this the easy way, that is I start CIMD with
faked checksums, if anybody manages to generate correct checksums,
please mail me.
Log in to SMSC
You have to set two variables in sms.scr:
\%b phonenumber to you operators SMSC
\%c login to access CIMD
For Europolitan (Swedish operator) you can use:
\%b 46708222902
\%c cimd3 # (use faked checksums)
This starts the CIMD server, and from now we must ACK
all responses.
Don't forget to include your modem prefix to \%b!!
I don't use the "prefix" token from BB, since I also use
this script as a mail->SMS gateway. If you are going to
use this, you must anyhow configure this script, so I hope
you don't mind.
Start a CIMD session
The first step in a CIMD session is to identify yourself
You have to set two variables in sms.scr:
\%d CIMD account
\%e CIMD password
Unfourtunately, I cant give you theese values ;)
Now you have configured your script and can start to use it!
Send to all recipients
BB calls the script one time for each recipient, that's
a bit of a waste, but it would just be to much work to
get around. If you call it manually or from sendmail,
you can use multiple recipients.
More info
The (english) spec of NOKIAS CIMD implementation can
be found on:
http://www.europolitan.se/europolitan/fick/tjanst/cimdspec.htm
If you happen to know swedish, you can read more on Europolitans
implementation on:
http://www.europolitan.se/europolitan/fick/tjanst/faqcimd.htm
Feedback
Please let me know if you use this with a non NOKIA server,
or if you fix real checksums!
Jacob Lundqvist <jaclu@ibk.se> (SMS:+46-708-555 456)
Thanks Jacob !!!
4.7 Using kermit
By default, BB expects kermit 5 to be installed on your
BBPAGER host. On certain installations it is V6 and up that is
installed. If that's your case then you must (might) modify the
bin/bb-page1.sh script:
change numeric.scr to numeric-k6.scr in the script.
But try with numeric.scr at first and if it doesn't work
then replace it with numeric-k6.scr. Don't forget to set KERMIT in
bbsys.sh.
4.8 Adding/modifying for a custom notification procedure
You can change the look of a notificaion message by modifying
the bin/bb-page1.sh script. Go at the last case statement at
the bottom of that script and make the appropriate changes:
${BBALPHAMSG} is the text body
${randvalsubjtag} is the security associated with the notification
$BBHOSTSVC is the hostname.service tag
$BBNUMERIC is a numeric value made up of 22+ digits
XXXYYYYYYYYYYYYZZZZZZZ
XXX - corresponding code of service defined in bbwarnsetup.cfg
YYYYYYYYYYYY - IP address
ZZZZZZZ - security code - first 5 digits: event ID code
last 2 digits: user ID code
$BBSVCNUM - equivalent of XXX in BBNUMERIC
$BBCOLORLEVEL - Color level causing the notification
$BBHOSTSVCDOTS - Name of host in the xxx.yyy.zzz format
$BBHOSTSVCCOMMAS - Name of host in the xxx,yyy,zzz format
$BBSVCNAME - Name of service causing the notification
If you want to create your own custom notification method
then you must add a case statement with a unique prefix associated
with it. Look at Sendpage (sp-) and Qpage (qp-) for examples on
how to do this.
4.9 Notification/acknowledgement logs
You'll find logs of who got notified for what reasons,
logs about recoveries, who got a recovery message and logs about
who acknowledged what notification. They are all located in the
$BBACKS directory:
acklog
notifications.log
recoveries.log
recoverymsgs.log
4.10 Disabling notifications temporarely
You can temporarely disable notifications without having to
modify the etc/bbwarnrules.cfg file.
All of you to do is send a 'disable' message to the
BBDISPLAY(s) using the following format:
MAKE SURE THAT THE BB ENVIRONMENT HAS BEEN SET PRIOR TO
USE 'bb' directly
./bb $BBDISP "disable 'host regular expression' 'duration' [reason]"
You can match multiple hosts/services by specifying a regular
expression instead of a real host name. The duration is
in minutes. You can also add an optional reason that will be
displayed in the status.
AGAIN, MAKE SURE THAT THE BB ENVIRONMENT HAS BEEN SET PRIOR TO
USE 'bb' directly
e.g.
cd $BBHOME/bin
./bb $BBDISP "disable www.bb4.com.disk 240"
Disable notifications on the disk event of www.bb4.com
for 240 minutes.
cd $BBHOME/bin
./bb $BBDISP "disable www.bb4.com* 240"
Disable notifications for all events for host
www.bb4.com for 240 minutes
cd $BBHOME/bin
./bb $BBDISP "disable www.bb4.com* 240 Taking www.bb4.com offline for a new disk"
Disable notifications for all events for host
www.bb4.com for 240 minutes and specify the reason
NOTE: duration can be expressed in seconds/minutes/hours/days
30s: 30 seconds
15m: 15 minutes
1h: 1 hour
1d: 1 day
by default, minutes are used if a qualifier is not used
To reenable a disabled host(s), send the "enable" message
./bb $BBDISP "enable 'host regular expression'"
As an example:
cd $BBHOME/bin
./bb $BBDISP "enable www.bb4.com*"
Enable notifications on all events for host www.bb4.com
Note that when you use the enable message, the colored dot
will stay blue until a new status is received by the BBDISPLAY(s)
This feature is not automatically available. It must be
explicitly compiled in. YOu must add the -DENABLE_DISABLE
directive to the CFLAGS variable of the makefile:
CFLAGS=-DENABLE_DISABLE -O
Please be warned that if you enable this feature, a hacker
could disable notifications while cracking into your systems.
------------------------------------------------------------------------
Section 5: Customizing your Big Brother display
5.1 Customizing the HTMLized status logs
If you don't like the blinking gifs, you can change them to
non-blinking by copying over the files found in www/gifs.
The non-blinking gifs start with the nb- prefix.
You can change Sean's mean face in the bb.html/bb2.html and
HTMLized status logs by replacing the bb.gif file. This only
applies to pre-1.3 versions.
You can change the header/footer for the HTMLized status pages.
The files are located in the /web directory. Note that
your customized version can contain special tags which can be
replaced by values:
&BBDATE: current date/time
&BBBACKGROUND: with the current background color
(red/green/yellow/clear/purple)
&BBRELDATE: Release date of the current version
&BBREL: Current version number
&BBCOLOR: Color of this status file
&BBHOST: Name of host of current status file
&BBSVC: Service name of current status file.
&BBIPNAME: IP address of host, if 0.0.0.0 then use hostname
&BBIP: IP address of host, always return IP address unless not found
You can add tags by modifying the bbd.c file. Every
time a status log is received by bbd, an HTML version of the
status log is created by prefixing the status with the header
file and suffixing with the footer file.
You can put tags also in the status logs:
&red &green &yellow &clear &purple
These tags are replaced with a corresponding image
source tag for the HTMLized version. Just like the bb-local.sh
script does for the disk and procs tests. So, if you create
your own custom check, enter these tags if you want to see a
colored dot in the HTMLized status.
NOTE: If you do not wish to have HTMLized status files, then
rename the www/html directory and the main BB display
pages will use the plain text status logs which are
located in the $BBLOGS directory. This will only
work if you had the install program create the
old-style directory structure.
5.2 Customizing headers / footers
bb.html/bb2.html and various HTML page generation
can be customized by modifying their respectable
headers/footers. These are located in the $BBHOME/web
directory.
NOTE: The link to bb4.com MUST be kept in the
footer page as per your agreement of the
license agreement.
5.3 Creating skins
You can create your own look and feel by customizing
various HTML aspects of Big Brother.
You can use your own gifs by creating a new directory
under $BBHOME/www. As an example, a directory 'psy'
has been created with more psychedelic colored dots
It lives in $BBHOME/www/psy. To use those images then
set the BBSKIN variable in etc/bbinc.sh to "psy".
To further modify the appearance of your display,
modify the header/footer files found in $BBHOME/web.
You also have some control over the bb.html/bb2.html
and the status logs HTML pages by modifying the
following variables in etc/bbinc.sh:
MKBBLOCAL
MKBBREMOTE
MKBBTITLE
MKBBROWFONT
MKBBCOLFONT
MKBBMANAGE
DAYS
HOURS
MINS
STATUNCHNMSG
RECVFROMMSG
DOTHEIGHT
DOTWIDTH
BBSKIN
5.4 Inserting notes about your hosts
You can setup HTML links on the host name in the
bb.html/bb2.html pages to point to an information
page for that particular host.
If a file exists in the www/notes directory that
matches the system name as displayed on the bb pages,
then they are linked into both the bb.html page and the
bb2.html summary page.
These files can end in .htm, .html, .shtml, .php3
or nothing :)
You can create these files manually or you can use the
"notes" messages type:
"notes <name of file> <data to write to the file>"
Here's an example:
cd $BBHOME/bin
./bb $BBDSIP "notes www.bb4.com.html <HTML><BODY>It's a hit!</BODY></HTML>"
This feature is not automatically available. It must be
explicitly compiled in. You must add the -DNOTESMSG
directive to the CFLAGS variable of the makefile:
CFLAGS=-DNOTESMSG -O
Please be warned that if you enable this feature, a hacker
could fill the disk with dummy notes file. Also note that
the file size is limited to the incoming buffer of BB
and by the size of the environment buffer.
Also note that if you source the BB environment prior
to sending the "notes" message and you have multiple BBDISPLAYs
then the "notes" message will be sent to all BBDISPLAYs. If you
want to send it to only one BBDISPLAY, then use that IP address
as the command line argument and set BBDISPLAYS to "" just
before sending your message.
5.5 Extending the contents of the main HTML pages
You can tack on some extra HTML code at the bottom of the
bb.html /bb2.html pages before the insertion of the footer.
You save the scripts that generate the HTML code in the
$BBHOME/ext/mkbb directory. And in etc/bbdef.sh, you specify
the name of those scripts in the BBMKBBEXT variable for output on
the bb.html page and in the BBMKBB2EXT variable for output in
bb2.html.
There's an example of a script in $BBHOME/ext/mkbb called
eventlog.sh that displays the last global events in a table
of the bb2.html page.
Make sure that the scripts are only writable by the owner
to prevent someone to execute malicious code.
5.6 Extending the contents of the history HTML pages
You can tack on some extra HTML code at the bottom of a
history display page before the insertion of the footer.
You save the scripts that generate the HTML code in the
$BBHOME/ext/hist directory. And in etc/bbdef.sh, you specify
the name of those scripts in the BBHISTEXT variable.
Make sure that the scripts are only writable by the owner
to prevent someone to execute malicious code.
5.7 Working with historical logs
There are many historical logs available with BB. These logs
contain all events that occured based on the host.service
combination, on the host itself or for all hosts. These logs
are found in the $BBHIST directory. You can use these logs to
view that last events that occured. By default, if you view
a status log, you can click the history button to view the
last 50 events for that host.service combination. You can
also use the ext/mkbb/eventlog.sh to view the last 20
events that happened for all hosts (you can view more by changing
the NUMEVENTS variable in eventlog.sh to the size you want).
You can disable the logging of events by renaming the
$BBHIST directory. You can set BBHOSTHISTLOG=FALSE
or BBALLHISTLOG=FALSE to prevent the logging of host or all
hosts events. You set those values in etc/bbdef.sh
There's a lot of interesting information available in those
logs. Check the FTP site for scripts that take advantage
of that information.
There's also a directory that keeps a copy of the status logs
when they change state such that they can be viewed later on.
The status logs are saved in the $BBHISTLOGS directory. A directory
is created with the name of the hosts and then each service has its
own directory where the status logs are saved based on the time of
the occurance.
5.8 Hiding the bb.html/bb2.html page header
You can position the bb.html/bb2.html pages at the beginning
of the first block by accessing the page as follows:
.../bb.html#begindata
You can bookmark this position to hide the header.
------------------------------------------------------------------------
Section 6: Miscellaneous Big Brother configuration options
6.1 Specifying filesystem specific values in the disk test
You can define warning and panic levels for specific
filesystems. They are defined in the etc/bb-dftab file. This file
contains definitions for all hosts all in a single file. You just need
to redistribute across your hosts. The format is as follows:
[hostname]:/filesystem:warning level:panic level
example:
/:85:95
www.maclawran.ca:/oradata:98:99
/ uses 85% for warning level and 95% for panic, this is
valid for all hosts (as no host name was define
in the definition)
Only on www.maclawran.ca /oradata uses 98% for warning
and 99% for panic
All other filesystems use the default values defined
in the etc/bbdef.sh file.
Use the etc/bb-dftab.DIST file as a starting point.
6.2 Enabling security
When BB is installed, it accepts connections from
any hosts. You can specify from which hosts only to accept
status logs using the etc/security file.
You can define individual hosts or networks in the
etc/security file. Using this format:
192.168.1.0
192.168.2.125
10.0.0.0
Accept from network 192.168.1.0, for 192.168.2.125 host
and the 10.0.0.0 network.
6.3 Using bbnet to test TCP services
You can test any text based TCP services using bbnet.
To do so add the test in bb-network.sh. If you need
just a basic test, then just add the service to
the list of other services. If you need to send a
special command to the service daemon, then add a case
statement entry like the imap test. Set change the
textmsg variable to the text to be sent to the daemom.
You can also use special options for bbnet. The -q/-Q/-s
options are available. The -q option does not return
the elasped time of test if bbnet was compiled with the
-DGETTIMEOFDAY switch (which is only valid if
the gettimeofday() function is available). The -Q option
prevents the display of error messages. Finally the -s
option only connects to the port to verify if the port
can be opened.
6.4 Sending custom one line status with bb
The bb executable used to send statuses/notifications
to the BBDISPLAY/BBPAGER can also be used by a custom
check using standard input:
custom_check | $BB $BBDISP -
The custom_check program just needs to send a single
line status to stdout (use the exact format of status
files found in $BBLOGS). This is very useful if the
custom check program is a daemon.
6.5 Changing the 'purple' interval
By default, BB will send a 'purple' status to indicate
that a status has not been received in the last 30
minutes (default). You can change this value to a more
suitable value by modifying PURPLEDELAY in etc/bbdef.sh
6.6 Specifying processes to look for
You can define processes to check for existence.
They are defined in the etc/bb-proctab file. This file
contains definitions for all hosts all in a single file.
You just need to redistribute this file across your hosts.
The format is as follows:
[hostname]:process list for yellow : process list for red
example:
localhost:smtp:httpd
www.maclawran.ca:!oracle "sleep 30":smtp
!process - check that host does not exists
"sleep 30" - use '"' when you want to check on a
process that spawns multiple words
Use the etc/bb-proctab.DIST file as a starting point.
If bb-proctab exists, then it overrides the PROCS/PAGEPROCS
values in bbdef.sh
6.7 Sending data to the BBDISPLAY to be handled by an external script
You can send data to your BBDISPLAY host to be
processed by an external script, a cron job or
a command line program/script.
The data is appended to the specified file name into the
$BBDATA directory.
To use this feature you have to send a "data" message
to the BBDISPLAY in this format:
"data <file name> <file data>"
Here's an example:
cd $BBHOME
./bb $BBDISP "data www.bb4.com.hits 952281437 10m 5000"
The "952281437 10m 5000" would be appended to the
$BBDATA/www.bb4.com.hits file.
This feature is not automatically available. It must be
explicitly compiled in. You must add the -DDATAMSG
directive to the CFLAGS variable of the makefile:
CFLAGS=-DDATAMSG -O
Please be warned that if you enable this feature, a hacker
could fill the disk with dummy data files.
Also note that if you source the BB environment prior
to sending the "data" message and you have multiple BBDISPLAYs
then the "data" message will be sent to all BBDISPLAYs. If you
want to send it to only one BBDISPLAY, then use that IP address
as the command line argument and set BBDISPLAYS to "" just
before sending your message.
6.8 Disabling the messages file(s) zero-length test
By default Big Brother verifies that none of the messages file
specified in the MSGFILE variable is of zero-length. This is
a simple test to make sure that a hacker hasn't linked your
messages file(s) to /dev/null.
Unfortunately, on some systems, when the logs are rotated
they are left empty by the procedure that rotates the logs.
If this is your case then you have 2 choices:
Immediately have the log rotation, add this command:
echo "`date`" >> <message file(s)>
or use a syslogd feature to add a line in the messages file(s)
or simply disable the zero-length check by setting the
CHKMSGLEN variable to FALSE in bbdef.sh.
Note, IMHO, the zero-length should always be there. Simply
echo the data to the messages file(s) or use syslogd features
to add an entry in the message file(s). Security is too
important to ignore. Your best bet is to install a log
scanner to analyze your messages file(s). Which one ?
It is our of BB's scope to tell which one(s) but ask
around for opinions of people who use log scanners.
6.9 Setting a Time To Live to a status message
When you create external scripts, you can now tell how
long the status is valid for. Just specify the length
in minutes of the status. This is to prevent purple status
on certain service that are not run in the regular interval
of 5 minutes.
To use this feature just follow this example
$BB $BBDISP "status+1560 www,bb4,com.backup green `date` Backup Successful"
A purple indicator will be triggered only if no status is received
within 1560 minutes (26hours).
6.10 Disabling the connectivity test
In some cases you may want to disable the connectivity test
altogether because you don't need it or because you want to
use the fping.sh external script available at ftp://ftp.deadcat.net .
To disable it, just set the CONNTEST variable in etc/bbdef.sh,
on the BBNET host, to FALSE. Also take in consideration that by
disabling this test you will lose the ability to generate clear
conditions for the other network tests because you will not have
access to then connectivity status for that host.
6.11 Reducing connectivity errors
If you are prone to a lot of red status for your connectivty
tests even though the host responds when you ping by hand.
To possibly reduce these false alarms, you may want to add/change
the PINGPAR1 or PINGPAR2 parameters in etc/bbsys.local on
the BBNET host. Adjust according to the BBNET's ping
parameters. PINGPAR1 are options that goes before the hostname/
IP address in the ping command while PINGPAR2 are options that
goes after the hostname/IP address.
Adjust the number of packets sent for the ping test. Start with
sending 2 packets and analyze the results. If it did not make
a difference, try with 3 packets. Note that when you specify
more than one packet to be sent, you inherently add a 1 second
sleep on each new packet. This will delay, in seconds, your
whole network test by (n-1) * hosts. If you test a large number
of hosts, you will see a noticable difference in how often your
hosts gets tested.
6.12 Setting up BBPAGERs and BBNETs host in failover mode
You can declare failover BBPAGER and BBNET hosts
and have them be in standby mode by checking for a failure
on the master BBPAGER or BBNET host and when that happens
they start behaving like the master BBPAGER or BBNET host
until the master host is back online.
To setup failover BBPAGER/BBNET hosts, define the primaries
BBPAGER/BBNET in the bb-hosts file with the BBPAGER or BBNET
directive and enable the failover external script on the
failover hosts by putting the failover script name into the BBEXT
variable of the bbdef.sh script for each host that should be
considered a failover host. Do not put failover in BBEXT on the
primary BBPAGER or BBNET host ! For complete details please
refer to the failover script in $BBHOME/ext.
Each failover host will generate a "fo" status log. When a host
will be in standby mode, it will generate a clear status and
when failover mode is enabled it will send a yellow status.
6.13 Speeding up the network tests
If you have a faily large number of servers to monitor there's
a very good possibility that the network tests takes a while
to complete.
You can speed up the tests by parallelizing the network tests
from a single bb-network.sh script to running multiple
copies of bb-network.sh at once.
All you have to do is to increase the value of BBNETTHREADS in
etc/bbdef.sh. Indicate how many bb-network.sh you'd like to
have working concurrently.
You can also speed up the network tests by using the fping
program to do the connectivity test. You'll have to download
fping and also get the fping.sh external script at the FTP
archive site: http://www.deadcat.net. You can disable the
regular ping test by setting CONNTEST to FALSE in etc/bbdef.sh.
6.14 Enabling WML ouput (WAP browsing)
By default, BB does not generate WML files. YOu can enable
the generation of the WML files by setting WML_OUTPUT to
TRUE in etc/bbdef.sh. The files will be saved in the www/wml
directory.
You'll be able to view the output on a WAP browser by viewing
the <BBWEB>/wml/bb.wml URL.
Make sure your web server can render wml files. In your
web server's mime.types file, make sure the following line
is defined:
text/vnd.wap.wml wml
Add it if need be and stop/start your web server.
6.15 Having BB tell you to reboot a server after is has been up XXX days
You can have BB display the "cpu" column as yellow dot when a
server has been up more than a pre-determined value. You
set the BBTOOMANYDAYSUP value in etc/bbdef.sh to the number
of days of uptime after you wish to schedule a reboot.
------------------------------------------------------------------------
Section 7: Other documentation sources
$BBHOME/README
$BBHOME/LICENSE
$BBHOME/README.INSTALL
$BBHOME/README.CHANGES
$BBHOME/install/README
$BBHOME/src/README
|
