DRBD and Heartbeat

DRBD And HeartBeat

About three months ago, I was planning a new server setup that would run our new portal as well as e-mail, databases, DNS and so forth. One of the most important goals was to create a redundant solution, so that if one of the servers failed, it wouldn't affect company operation.
I looked through a lot of the redundant solutions available for Linux at the time, and with most of them, I had trouble getting all the services we needed to run redundantly. After all, there is a very big difference in functionality between a Sendmail dæmon and a PostgreSQL dæmon.
In the end, though, I did find one solution that worked very well for our needs. It involves setting up a disk mirror between machines using the software DRBD and a high-availability monitor on those machines using Heartbeat.
DRBD mirrors a partition between two machines allowing only one of them to mount it at a time. Heartbeat then monitors the machines, and if it detects that one of the machines has died, it takes control by mounting the mirrored disk and starting all the services the other machine is running.
In this tutorial, I show you how to set up a redundant Sendmail system, because once you do that, you will be able to set up almost any service you need. We assume that your master server is called server1 and has an IP address of 192.168.1.1, and your slave server is called server2 and has an IP address of 192.168.1.2.
And, because you don't want to have to access your mail server on any of these addresses in case they are down, we will give it a virtual address of 192.168.1.5. You can, of course, change this to whatever address you want in the Heartbeat configuration.

How It Works

-------------------------

This high-availability solution works by replicating a disk partition in a master/slave mode. The server that is running as a master has full read/write access to that partition; whereas the server running as slave has absolutely no access to the partition but silently replicates all changes made by the master server.
Because of this, all the processes that need to access the replicated partition must be running on the master server. If the master server fails, the Heartbeat dæmon running on the slave server will tell DRBD that it is now the master, mount the replicated partition, and then start all the processes that have data stored on the replicated partition.

How to Get It Running

-------------------------------------
The first step for running a redundant system is having two machines ready to try it out. They don't need to have identical specs, but they should meet the following requirements:

• Enough free space on both machines to create an equal-sized partition on each of them.
• The same versions of the dæmons you want to run across both machines.
• A network card with crossover cable or a hub/switch.
• An optional serial port and serial port crossover cable for additional monitoring.

You also should think carefully about which services you want running on both machines, as this will affect the amount of hard disk you will need to dedicate to replication across them and how you will store the configuration and data files of these services.
It's very important that you have enough space on this shared partition, because it will be the main data storage location for all of these services. So, if you are going to be storing a large Sendmail spool or a database, you should make sure it has more than enough space to run for a long time before having to repartition and reconfigure DRBD for a larger disk size.

Setting Up the Basics on Your Servers

----------------------------------------------------------------
Once you've made sure your machines are ready, you can go ahead and create an equal-sized partition on both machines. At this stage, you do not need to create a filesystem on that partition, because you will do that only once it is running mirrored over DRBD.
For my servers, I have one DRBD replicated drive that looks like this on my partition tables:

/dev/sda5      7916    8853   7534453+  83  Linux

Note: type fdisk -l at your command prompt to view a listing of your partitions in a format similar to that shown here. Also, in my case, the partition table is identical on both redundant machines.
The next step after partitioning is getting the packages for Heartbeat version 1.2+ and DRBD version 0.8+ installed and the DRBD kernel module compiled. If you can get these prepackaged for your distribution, it will probably be easier, but if not, you can download them from www.linux-ha.org/DownloadSoftware and www.drbd.org/download.html.
Now, go to your /etc/hosts file and add a couple lines, one for your primary and another for your secondary redundant server. Call one server1, the other server2, and finally, call one mail, and set the IP addresses appropriately. It should look something like this:

192.168.1.1    server1
192.168.1.2    server2
192.168.1.5    mail

Finally, on both your master and slave server, make a folder called /replicated, and add the following line to the /etc/fstab file:

/dev/drbd0    /replicated   ext3   noauto    0   0

Configuring DRBD

--------------------------------

After you've done that, you have to set up DRBD before moving forward with Heartbeat. In my setup, the configuration file is /etc/drbd.conf, but that can change depending on distribution and compile time options, so try to find the file and open it now so you can follow along. If you can't find it, simply create one called /etc/drbd.conf.
Listing 1 is my configuration file. I go over it line by line and add explanations as comments that begin with the # character.

Listing 1. /etc/drbd.conf

# Each resource is a configuration section for a
# mirrored disk.
# The drbd0 is the name we will use to refer
# to this disk when starting or stopping it.

resource drbd0 {
  protocol C;
  handlers {
    pri-on-incon-degr "echo 'DRBD: primary requested but inconsistent!'
    ↪| wall; /etc/init.d/heartbeat stop"; #"halt -f";
    pri-lost-after-sb "echo 'DRBD: primary requested but lost!'
    ↪| wall; /etc/init.d/heartbeat stop"; #"halt -f";
  }

  startup {
    degr-wfc-timeout 120;    # 2 minutes.
  }

  disk {
    on-io-error   detach;
  }
# These are the network settings that worked best for me.
# If you want to play around with them, go
# ahead, but take a look in the man pages of drbd.conf
# and drbdadm to see what each does.

  net {
    timeout 120;
    connect-int 20;
    ping-int 20;
    max-buffers     2048;
    max-epoch-size  2048;
    ko-count 30;

# Remember to change this shared-secret on both the master
# and slave machines.

    cram-hmac-alg "sha1";
    shared-secret "FooFunFactory";
  }

  syncer {
    rate 10M;
    al-extents 257;
  }

# This next block defines the settings for the server
# labeled as server1.  This label should be in your
# /etc/hosts file and point to a valid host.

  on server1 {

# The following device will be created automatically by
# the drbd kernel module when the DRBD
# partition is in master mode and ready to write.
# If you have more than one DRBD resource, name
# this device drbd1, drbd2 and so forth.

    device     /dev/drbd0

# Put the partition device name you've prepared here.

    disk      /dev/sda5;

# Now put the IP address of the primary server here.
# Note: you will need to use a unique port number for
# each resource.

    address   192.168.1.3:7788;
    meta-disk  internal;
  }

# This next block is identical to that of server1 but with
# the appropriate settings of the server called
# server2 in our /etc/hosts file.

  on server2 {
    device    /dev/drbd0;
    disk      /dev/sda5;
    address   192.168.1.2:7788;
    meta-disk internal;
  }
}

Now, let's test it by starting the DRBD driver to see if everything works as it should. On your command line on both servers type:

drbdadm create-md drbd0; /etc/init.d/drbd restart; cat /proc/drbd

If all goes well, the output of the last command should look something like this:

0: cs:Connected st:Secondary/Secondary ds:Inconsistent/Inconsistent r---
   ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0
       resync: used:0/7 hits:0 misses:0 starving:0 dirty:0 changed:0
       act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0

Note: you always can find information about the DRBD status by typing:

cat /proc/drbd

Now, type the following command on the master system:

drbdadm -- --overwrite-data-of-peer primary drbd0; cat /proc/drbd

The output should look something like this:

0: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent r---
   ns:65216 nr:0 dw:0 dr:65408 al:0 bm:3 lo:0 pe:7 ua:6 ap:0
       [>...................] sync'ed:  2.3% (3083548/3148572)K
       finish: 0:04:43 speed: 10,836 (10,836) K/sec
       resync: used:1/7 hits:4072 misses:4 starving:0 dirty:0 changed:4
       act_log: used:0/257 hits:0 misses:0 starving:0 dirty:0 changed:0

This means it is syncing your disks from the master computer that is set as the primary one to the slave computer that is set as secondary.
Next, create the filesystem by typing the following on the master system:

mkfs.ext3 /dev/drbd0

Once that is done, on the master computer, go ahead and mount the drive /dev/drbd0 on the /replicated directory we created for it. We'll have to mount it manually for now until we set up Heartbeat.

Preparing Your Services

------------------------------------------
An important part of any redundant solution is properly preparing your services so that when the master machine fails, the slave machine can take over and run those services seamlessly. To do that, you have to move not only the data to the replicated DRBD disk, but also move the configuration files.
Let me show you how I've got Sendmail set up to handle the mail and store it on the replicated drives. I use Sendmail for this example as it is one step more complicated than the other services, because even if the machine is running in slave mode, it may need to send e-mail notifications from internal applications, and if Sendmail can't access the configuration files, it won't be able to do this.
On the master machine, first make sure Sendmail is installed but stopped. Then create an etc directory on your /replicated drive. After that, copy your /etc/mail directory into the /replicated/etc and create a symlink from /replicated/etc/mail to /etc/mail.
Next, make a var directory on the /replicated drive, and copy /var/mail, /var/spool/mqueue and any other mail data folders into that directory. Then, of course, create the appropriate symlinks so that the new folders are accessible from their previous locations.
Your /replicated directory structure should now look something like:

/replicated/etc/mail
/replicated/var/mail
/replicated/var/spool/mqueue
/replicated/var/spool/mqueue-client
/replicated/var/spool/mail

And, on your main drive, those folders should be symlinks and look something like:

/etc/mail -> /replicated/etc/mail
/var/mail -> /replicated/var/mail
/var/spool/mqueue -> /replicated/var/spool/mqueue
/var/spool/mqueue-client -> /replicated/var/spool/mqueue-client
/var/spool/mail -> /replicated/var/spool/mail

Now, start Sendmail again and give it a try. If all is working well, you've successfully finished the first part of the setup.
The next part is to make sure it runs, even on the slave. The trick we use is copying the Sendmail binary onto the mounted /replicated drive and putting a symlink to the binary ssmtp on the unmounted /replicated folder.
First, make sure you have ssmtp installed and configured on your system. Next, make a directory /replicated/usr/sbin, and copy /usr/sbin/sendmail to that directory. Then, symlink from /usr/sbin/sendmail back to /replicated/usr/sbin/sendmail.
Once that's done, shut down Sendmail and unmount the /replicated drive. Then, on both the master and slave computers, create a folder /replicated/usr/sbin and a symlink from /usr/sbin/ssmtp to /replicated/usr/sbin/sendmail.
After setting up Sendmail, setting up other services like Apache and PostgreSQL will seem like a breeze. Just remember to put all their data and configuration files on the /replicated drive and to create the appropriate symlinks.

Heartbeat is designed to monitor your servers, and if your master server fails, it will start up all the services on the slave server, turning it into the master. To configure it, we need to specify which servers it should monitor and which services it should start when one fails.
Let's configure the services first. We'll take a look at the Sendmail we configured previously, because the other services are configured the same way. First, go to the directory /etc/heartbeat/resource.d. This directory holds all the startup scripts for the services Heartbeat will start up.
Now add a symlink from /etc/init.d/sendmail to /etc/heartbeat/resource.d.
Note: keep in mind that these paths may vary depending on your Linux distribution.
With that done, set up Heartbeat to start up services automatically on the master computer, and turn the slave to the master if it fails. Listing 2 shows the file that does that, and in it, you can see we have only one line, which has different resources to be started on the given server, separated by spaces.

Listing 2. /etc/heartbeat/haresources

server1 IPaddr::192.168.1.5/24 datadisk::drbd0 sendmail

The first command, server1, defines which server should be the default master of these services; the second one, IPaddr::192.168.1.5/24, tells Heartbeat to configure this as an additional IP address on the master server with the given netmask. Next, with datadisk::drbd0 we tell Heartbeat to mount this drive automatically on the master, and after this, we can enter the names of all the services we want to start up—in this case, we put sendmail.
Note: these names should be the same as the filename for their startup script in /etc/heartbeat/resource.d.
Next, let's configure the /etc/heartbeat/ha.cf file (Listing 3). The main things you would want to change in it are the hostnames of the master/slave machine at the bottom, and the deadtime and initdead. These specify how many seconds of silence should be allowed from the other machine before assuming it's dead and taking over.
If you set this too low, you might have false positives, and unless you've got a system called STONITH in place, which will kill the other machine if it thinks it's already dead, you can have all kinds of problems. I set mine at two minutes; it's what has worked best for me, but feel free to experiment.
Also keep in mind the following two points: for the serial connection to work, you need to plug in a crossover serial cable between the machines, and if you don't use a crossover network cable between the machines but instead go through a hub where you have other Heartbeat nodes, you have to change the udpport for each master/slave node set, or your log file will get filled with warning messages.

Listing 3. /etc/heartbeat/ha.cf

debugfile /var/log/ha-debug
logfile /var/log/ha-log
logfacility     local0
keepalive 2
deadtime 120
initdead 120
serial  /dev/ttyS1
baud 9600
udpport 694
udp     eth0
nice_failback on
node server1
node server2

Now, all that's left to do is start your Heartbeat on both the master and slave server by typing:

/etc/init.d/heartbeat start

Once you've got that up and running, it's time to test it. You can do that by stopping Heartbeat on the master server and watching to see whether the slave server becomes the master. Then, of course, you might want to try it by completely powering down the master server or any other disconnection tests.

Congratulations on setting up your redundant server system! And, remember, Heartbeat and DRBD are fairly flexible, and you can put together some complex solutions, including having one server being a master of one DRBD partition and a slave of another. Take some time, play around with them and see what you can discover.

-->

By: Unknown On 2:23 PM

Continue Reading

Centralized Syslog Server

A centralized syslog server was one of the first true SysAdmin tasks that I was given as a Linux Administrator way back. My boss at the time wanted to pull in log files from various appliances and have me use regexp to search them for certain key words. At the time Linux was still in its infancy, and I had just been dabbling with it in my free time. So, I jumped at the chance to introduce Linux to the company that I had worked for. Did it work? You bet it did! What this post is going to cover is not only how to setup a centralized syslog-ng server, but why you would go about setting one up in the first place.
So what is syslog? Syslog is used in Linux to log system messages (huh, another easy to guess name). Syslog-ng is just a rewrite of the original syslog, that was developed in 1998. Syslog-ng is still being actively developed (as of 2010) by BalaBit IT Security and comes with many more features, including better TCP handling, TLS encryption of messages, and sending messages to a database among other things. Some distributions allow you to install either syslog, rsyslog or syslog-ng. For this article, I'll be focusing on syslog-ng as this is more up to date, and if the reader wishes, can be 'supported' via the company that owns the syslog-ng software by going with their enterprise edition version at a later date.
Now that you've got an overview of syslog-ng, let's talk about just why you would use a centralized syslog-ng server. I am sure there are more than the two reasons that I will bring up, but I can think of at least 2 of them off the top of my head. The first is for security purposes. If you have your routers, firewalls, switches, Linux servers and/or other hardware pointing to a SECURED centralized syslog-ng server, when someone does attempt to attack one of the above devices log files can be safely off-site in a secure location. If syslog files are kept on the device this gives an attacker the ability to clean up their tracks. Granted, they can disable the ability to send log files to an external syslog-ng server, but any and all connections prior to that will be located on the centralized syslog server. The other reason is for convenience. For instance, if you have a server that crashed and is unresponsive, you can check the kernel error logs on your centralized syslog server. If you want to check syslog patterns between various dates over an extended time, regex the log files from the centralized syslog server.
So what do I do? I actually use both approaches at home. Not only do my devices and servers forward all their syslog files to a centralized location, but that location is locked down. The machine in question is a virtual machine with only 1 port open (syslog) and accessible only from the local machine, the syslog files are kept on an external drive. Is it paranoia? Probably a wee bit. But I do know that in my home environment, if my external drive fills up from too many syslog files it won't crash my virtual machine. If somehow something happens to my virtual machine, my host OS won't be affected, if someone does gain access to one of my devices then they can't gain access to my syslog server. Granted if something happened to my host OS then I would have issues with my guest VM's, but we can't always prepare for everything. Okay, I admit it's paranoia in the highest of levels, and for most people this is probably too far.
Before we get started, here's a quick disclaimer. First off, as with all of my previous posts, I do all of my blogpost testing in Debian. In this case I had a virtual machine setup for Debian 6.0.1, thus your mileage may vary. Also I won't be getting into how to properly secure your server, best practices on where to place syslog files, or how to setup anything other than syslog-ng. I leave that up to the reader. This blog post just covers the basics of a centralized syslog-ng server.

Installing and Configuring - Server Side

Installing syslog-ng isn't as hard as it looks especially if you're installing from packages. For Debian: apt-get install syslog-ng, for Redhat: yum install syslog-ng. For those of you that enjoy a good source install: http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.2.4/source/syslog-ng_3.2.4.tar.gz Download, unpackage, configure, make & make install. Once you have syslog-ng installed, we can get to configuring the server side.

Global Options

First thing you need to do is locate your syslog-ng configuration file. The default install (for Debian variants) is '/etc/syslog-ng/syslog-ng.conf'. Before editing any configuration files it is best practice to make a copy of the original configuration file prior to any changes. This is just in case something happens and you need to go back to the original configuration file. I tend to label my original configuration files with .orig (in this case: syslog-ng.conf.orig). Now that you have made a copy of your configuration file, let's open it up with your editor of choice and get started.
long_hostnames(default: off ) - For this post I'm using syslong-ng OSE version 3.1, and I actually can't find long_hostnames in the global configuration guide online. I'll go with long hostnames as a default of off, being fully qualified domain names.
flush_lines(default: 0 ) - Sets the number of lines flushed to a destination at a time. Setting to 0 sends messages as they are received, but keep in mind setting this number higher may increase message latency. This is useful on the client side of syslog-ng. You would keep xx messages on the client before flushing to the destination so that you are not flooding the main syslog-ng server if you have alot of traffic coming from a server.
use_dns(default: no ) - Options: yes, no, persist_only. This one is up to you and your environment. If your syslog-ng is behind a firewall and not accessible to the outside of the world then 'yes' would be appropriate. If accessible to the outside of the world, set to 'no' in order to stop possible DoS attacks. I set mine to 'persist_only' which checks my /etc/hosts file on my syslog-ng server to resolve hostnames, without relying on dns servers.
use_fqdn(default: no ) - Set the Fully Qualified Domain Name, your choice. As a home network I only have one internal domain name. So mine defaults to 'no'. Setting to 'yes' would have your clients hostname show up as: 'hostA.domain.com' instead of 'hostA'
owner(default: root ) - Owner of output files
group(default: adm ) - Group of output files
perm(default: 0640 ) - Permission of output files. Defaults to 640 - Owner Read-Write, Group Read, Other none.
stats_freq(default: 0 ) - Time (in seconds) between two STATS (statistics messages about dropped log messages) Messages. 0 disables STATS messaging.
bad_hostname(default: ^gconfd$ ) - Regex containing hostnames that should not be handled as hostnames..in this case gconfd. If you have more than a handful of servers than I woudl recommend hostnames, unless of course you remember every ip address in your domain..if you, I applaud you.
Now that's it for the 'Default' Global configuration options, but there are many more that you can use. I also use the following:
normalize_hostnames(yes) - This converts all hostnames to lowercase. Some of my devices have uppercase hostnames, and sometimes I get carried away with a new host and Uppercase the first letter of the hostname. This will just lowercase all characters for easier readability.
keep_hostname(yes) - This keeps the hostname if running through a relay or an external server, so that when the host finally reaches the central server the hostname comes with it instead of relying on DNS (or /etc/hosts). If you're using $HOST macro, this should be enabled.
In a bigger and more important environment (read: not soho) I would be setting stats_freq(600) and stats_level(2) in order to retrieve statistics messages from the server. In most soho environments you might be gathering syslog data from 3-5 devices, at which point the odds of actually losing data are pretty slim. In a larger enterprise environment of several hundred devices reporting to centeral syslog servers, enabling statistics allows the sys admin the ability to check on stats and possibly lost messages.
Your global configuration options (if you want it to mirror mine) would look like the following:

options {(off); 
flush_lines(0); 
use_dns(persist_only);
use_fqdn(no); 
owner("root"); 
group("adm"); 
perm(0640); 
stats_freq(0);
bad_hostname("^gconfd$"); 
normalize_hostnames(yes);
keep_hostname(yes); 
};

Setting up Listener

Setting up the listener for syslog-ng is actually only a few lines in the configuration file. A typical listener line looks like this:

source s_net { tcp((ip(127.0.0.1) port(1000) max-connections 5000)); udp (); };

source s_net = Network listener
tcp(ip(127.0.0.1) = Listen on localhost. If you have multiple NIC's, or want to specify an ip to bind this to, change 127.0.0.1 to the ip address of that specific network card
port (1000) = Listen to TCP port 1000
max connections = Allow 5000 simultaneous connections (stops the dreaded 'run away server' syndrome)
udp () = Some devices send their syslog messages via udp, so enable udp if you can't specify tcp and port number.
encrypt(allow) = This could be an entire blog post in itself. Syslog-ng allows for encrypted (TLS, certificate based) syslog messages
Mine for example looks like this:

# Listen on TCP Port 1000 and UDP Port 514, Max 500 Connections source s_net {
tcp(port(1000) max-connections(500)); udp(););

Destination - What goes up must come down. In this case what gets sent out must get put somewhere. Once a message is received from the syslog-ng server it's got to go somewhere. Thus the destination section of the syslog-ng.conf file. As you can see, the default covers your *nix destination for server messages on the local machine. But what about incoming messages? Where do they go? Good question, by default they will send their syslog messages to the subsystem specified in syslog-ng. For instance if it's a message that would be classified as an authentication message (/var/log/auth) then it will dump the message into the syslog-ng's /var/log/auth.log file with the appended information (hostname, date/time, etc).
If that's actually what you want to accomplish, a bunch of servers dumping to the same file as your main server, then I guess the task is complete. But syslog-ng can do so much more than that. If I do much more on server side configuration though I fear this will end up being a chapter in a book. Destinations can be flat files, pipes into other applications, SQL Databases (mysql, MS SQL, Oracle, etc), Remote Log servers, and Terminal Windows. I'll be focusing on flat files and assume you are doing the same for now.
Now the way I setup my centralized syslog server might be different then the way you setup yours. In my case I have a folder that has each hostname and the syslogs from the hostname are located in the folder. For Example: /mount/syslog/macha, /mount/syslog/beag, and so on and so forth. Logrotate takes care of zipping, removing (old files are backed up to a remote server just in case) and cleaning up log files.
My Destination directive looks like this:

destination d_net_auth {
file("/var/log/syslog/remote/$HOSTNAME/auth.log"); }; destination d_net_cron {
file("/var/log/syslog/remote/$HOSTNAME/cron.log"); }; destination d_net_daemon
{ file("/var/log/syslog/remote/$HOSTNAME/daemon.log"); }; destination
d_net_kern { file("/var/log/syslog/remote/$HOSTNAME/kern.log"); }; destination
d_net_lpr { file("/var/log/syslog/remote/$HOSTNAME/lpr.log"); }; destination
d_net_mail { file("/var/log/syslog/remote/$HOSTNAME/mail.log"); }; destination
d_net_syslog { file("/var/log/syslog/remote/$HOSTNAME/syslog.log"); };
destination d_net_user { file("/var/log/syslog/remote/$HOSTNAME/user.log"); };
destination d_net_user { file("/var/log/syslog/remote/$HOSTNAME/uucp.log"); };
destination d_net_debug { file("/var/log/syslog/remote/$HOSTNAME/debug"); };
destination d_net_error { file("/var/log/syslog/remote/$HOSTNAME/error"); };
destination d_net_messages { file("/var/log/syslog/remote/$HOSTNAME/messages");
}; destination d_net_mailinfo {
file("/var/log/syslog/remote/$HOSTNAME/mail/mail.info"); }; destination
d_net_mailwarn { file("/var/log/syslog/remote/$HOSTNAME/mail/mail.warn"); };
destination d_net_mailerr {
file("/var/log/syslog/remote/$HOSTNAME/mail/mail.err"); }; 

Now in theory, the syslog-ng server is supposed to create the directories necessary for the files to drop into (as specified in the global policies) but sometimes I run into problems where the directories were not created properly and the errors in syslog-ng are reported in /var/log/errors. To alleviate future pain and suffering I tend to create the host and log files as I go, anything I'm missing will end up in /var/log/errors and I can create them later.
For those of you that are veteran syslog-ng users, you might wonder why I split my localhost destination and my remote(off-site clients) destinations when in theory I could have created a d_auth and had my regular localhost filter into a folder as well. The reason behind that was that I wanted to separate my localhost syslog traffic from remote traffic - more configuration lines, but easier on me. Also, I'm not messing with the Linux subsystem when it's out looking for where to put regular log files.
Filtering - The ability for Syslog-NG to filter its messages is what really seperates the 'men from the boys' in the syslog battle. The filtering is what really sets syslog-ng apart. Granted I separate my hosts in folders defined in $HOST variable, but filtering is the real meat and potatoes. With filtering I can (and do) the following: Filter Firewall logs looking for certain key words such as port scans, that get dumped into 1 folder, DDOS attacks that get filtered into another folder. My voip adaptor sends syslog events and I filter based on those messages into individual files instead of a single file. Filtering also allows you to specify multiple hosts to filter based on, and into multiple destinations. Not only that, but you can use regular expressions in filtering.
Filtering expressions are created like: filter { expression; };
is the name you give your filter. contains the function, and boolean operators (and,or,not).
An example for my firewall would be:

filter firewall_ddos_filter { host("10.1.1.1") and match("Denial of Service"
value("MESSAGE")); };

This filter is called 'firewall_ddos_filter, it listens for incoming syslog messages from 10.1.1.1 with a message of 'Denial of Service'. To complete the filter you need a log statement:
log firewall_ddos_filter { source(s_net); filter(firewall_ddos_filter); destination(d_net_firewall_ddos); };
In my above destination I would add a destination for firewall DDOS Attacks, port scanning, etc. This makes it easier to separate log files from servers/devices that do not use the standard *nix logging facilities, or easier for a system admin to filter logs coming out of a firewall (or many firewalls filtered into one log).
If you want to use multiple 'firewall' hosts (as an example) do NOT use just add them in and create a log/filter rule using a boolean operator of 'and'. It will not work, and you beat your head on the desk for many hours to come. Instead, use the 'or' boolean operator as such:
filter firewall_ddos_filter { host("10.1.1.1") or host ("10.1.1.2") and match("Denial of Service" value("MESSAGE")((; };
My 'Default' Filtering directive looks like this (Beautified for this post but they call fit in 'paragraph' form as long as there is a semi-colon seperating each case):

filter f_dbg { level(debug); }; 
filter f_info { level(info); }; 
filter f_notice{ level(notice); }; 
filter f_warn { level(warn); }; 
filter f_err { level(err);  }; 
filter f_crit { level(crit .. emerg); };

filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };
filter f_error { level(err .. emerg) ; }; 
filter f_messages { level(info,notice,warn) and not facility(auth,authpriv,cron,daemon,mail,news);
};

filter f_auth { facility(auth, authpriv) and not filter(f_debug); }; 
filter f_cron { facility(cron) and not filter(f_debug); }; 
filter f_daemon { facility(daemon) and not filter(f_debug); }; 
filter f_kern { facility(kern) and not filter(f_debug); }; 
filter f_lpr { facility(lpr) and not filter(f_debug);}; 
filter f_local { facility(local0, local1, local3, local4, local5, local6, local7) and not filter(f_debug); }; 
filter f_mail { facility(mail) and not filter(f_debug); }; 
filter f_news { facility(news) and not filter(f_debug); };
filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; 
filter f_user { facility(user) and not filter(f_debug); }; filter f_uucp { facility(uucp) and not filter(f_debug); };

filter f_cnews { level(notice, err, crit) and facility(news); }; 
filter f_cother { level(debug, info, notice, warn) or facility(daemon, mail); };

filter f_ppp { facility(local2) and not filter(f_debug); }; 
filter f_console { level(warn .. emerg); };

Statistics

There's nothing more I enjoy better than some good statistics. When I run any server or service, be it at the house or at work I want to see what my server has processed over time. Beginning with version 3.1, syslog-ng now has a syslog-ng-ctl stats utility which has greatly simplified grabbing log files. Prior to 3.1 to fetch statistic files you would run: echo STATS | nc -U /var/run/syslog-ng.ctl.
Because I'm a regex geek I'm not thrilled with the semi-colons in the output of syslog-ng-ctl stats thus I run: syslog-ng-ctl stats | sed 's|;|\t|g' to clean up the output.
What you have when you type the above command is 6 columns: SourceName, SourceID, SourceInstance, State, Type and Number.
SourceName - The name of the Source, for instance: destination, source, global, center
SourceID - The ID you gave the source (a previous example was firewall_ddos_filter, other examples would be: d_mail, d_net_user, etc)
SourceInstance - The destination of the Source Instance such as a filename, or the name of an application for a program source (sql) or destination
State: - Status of the object: a (Active - Currently active and receiving data), d (Dynamic - Not continuously available) o (Once active but stopped receiving messages such as an orphaned object)
Type - Type of Statistic such as: Processed: Number of Messages that reached their destination Dropped: Number of dropped messages Stored: Number of messaged stored in message Queue waiting to be sent to destination Suppressed (not sent): Number of Suppressed Messaged Stamp: Timestamp of Last message sent. These statistics are reset when the syslog-ng service is reset.
Number: Number of Messages

Log Rotate, Log Rotate, LOG ROTATE

Was that a clear enough message for you? Rotating your message logs will save your butt in the log run. Without rotating your logs your log disk space will just continue to grow and grow eventually filling up your hard drive. Not only will log rotate save space, but it will make searching for log files on specific dates easier than pulling up a 50MB log file that you didn't set into log rotate and searching for a specific date. Depending on your distro, logrotate is located in /etc/logrotate.conf. As this isn't a blogpost on logrotate, I'll leave your configuration up to your imagination and give you an example on how I rotate my log files:
/var/log/remote/*/ { rotate 5 weekly missingok create }
This goes through /var/log/remote/*/ every week and rotates my logs. Logs are rotated for 1 month at which point I have a cronjob that tar-zips my old logs and they are moved off to a backup location where they are kept for another month before being rotated off. In a business environment of course logs would be kept for however long management and legal dictates, but for a home environment I feel 2 months of logs is good enough to troubleshoot any problems that might have come up in that time.

Syslog Client

As each server and device is different in their setup, I won't get too in-depth into this. Syslog communicates on UDP port 514, but as I stated earlier above, I also set the main syslog server to communicate on TCP port 1000 for other devices. This allows the syslog-ng server to listen on two ports, 514 UDP for devices that can't change their ports, and TCP 1000 for servers that you can specify port numbers. Why did I put TCP 1000 and not TCP 514? Because Linux uses tcp 514 for rsh (remote shell) which would have caused some problems with my (and other's) host system. If you plan on running syslog-ng on the outside of the world (and I would assume your setting authentication, and using TLS encryption) then setting a TCP port that's not typical would be your best bet.
1. For devices all you should need to do is tell the device to point to the hostname and make sure either UDP 514 or TCP 1000 is the destination
2. For rsyslog clients add the following line:

For TCP:  *.* @@ipaddress:1000
For UDP:  *.* @ipaddress:514

3. For syslog-ng clients add the following line:

*New syslog Protocol* syslog(host tranport [options];

*old syslog protocol* destination d_tcp { syslog(ip("remoteip")
transport("tcp") port(1000) localport(999)}; };

destination d_udp { syslog(ip("remoteip") transport("udp") port(514)
localport(999)}; };

Conclusion

Well there you have it, a birds eye view of syslog-ng. There is plenty more that you can learn about syslog-ng, as I just went into the basics of getting started. From here you can get into macros, increased filtering, and TLS/Certificate based encryption of syslog messages (which I might cover in a later blog post). By sending your syslog messages to a centralized syslog server, and backing up said syslog server, you can rest assured that your system messages are secure and easy to get to when you need them.

By: Unknown On 2:21 PM

Continue Reading

Creating Jailkit With Chroot Sftp

A few months back I was given an assignment to create some chroot jails for a group of customers so that they could securely upload files with sftp. The requirement was that the customers needed to be able to upload file, but in a secure and private way. Customer One should not be able to see Customer Two's files, for example. And neither customer should be able to browse the filesystem of the server. I was also asked to define a process whereby our support staff could add new jails as needed.
I've used the chroot command many times, it's a very handy way of fixing servers that can't otherwise be fixed --- like when no one can remember the root password on some box that no one has touched for years. Simply boot the server with a live CD (Knoppix is a personal favorite, I have it loaded on a usb key that I carry with me everywhere), mount the main drive and then issue a command along the lines of:

chroot /mnt/hda1


Suddenly it's like you are root on the local filesystem and not on the live CD. I can then change passwords, run programs, and generally get myself into trouble. :-)

You can do a lot more with chroot than simply use it as part of your system recovery toolkit though. Chroot is _very_ useful on a day-to-day basis for jailing applications and users that should (or need) to be isolated from the rest of the system (and from each other).

To create a jail you create a folder that has a replication of the directory structure of a normal Linux box. The difference is that you only copy in the bare minimum of what you need into the directory structure. The way you find out what libraries and files an application needs is by using ldd or strace. If I can get away with it, I prefer using ldd since its output is much more readable, but there are times when both are needed to track down elusive libraries.
As an example, lets say I want to have a jail that contains the bash shell, just that and nothing more. It's a stupid example, but very easy.
I first need to create the directory that will house the jail:

Akhil@groundzero:~$ mkdir /opt/jail


It's just an empty directory now, so we now need to add some apps to it. Bash lives in the /bin directory so I first recreate the /bin directory and then copy the bash binary into it like so:

Akhil@groundzero:~$ mkdir /opt/jail/bin
Akhil@groundzero:~$ cp /bin/bash /opt/jail/bin/

Then I run ldd on bash to find out what libraries it uses like so:

Akhil@groundzero:~$ ldd /bin/bash
linux-gate.so.1 => (0xffffe000)
libncurses.so.5 => /lib/libncurses.so.5 (0xb7ec2000)
libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0xb7ebe000)
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7d7d000)
/lib/ld-linux.so.2 (0xb7f1f000)
Akhil@groundzero:~$

I can ignore the linux-gate.so.1 file, it's a virtual object created by the Linux Kernel, but I need the others so I create the directories and then copy them over:

Akhil@groundzero:~$ mkdir -p /opt/jail/lib/tls/i686/cmov
Akhil@groundzero:~$ cp /lib/libncurses.so.5 /opt/jail/lib/
Akhil@groundzero:~$ cp /lib/tls/i686/cmov/libdl.so.2 /opt/jail/lib/tls/i686/cmov/
Akhil@groundzero:~$ cp /lib/tls/i686/cmov/libc.so.6 /opt/jail/lib/tls/i686/cmov/
Akhil@groundzero:~$ cp /lib/ld-linux.so.2 /opt/jail/lib/

Now I can test the jail like so:

Akhil@groundzero:~$ sudo chroot /opt/jail /bin/bash
bash-3.2#

The presence of the bash-3.2# prompt tells me I was successful.
This is a pretty useless jail right now, with only bash. I can 'cd' and 'pwd' and 'exit' and do other bash built-in commands, but I can't 'ls' or 'less' or 'cp' or 'mv' or anything else that is external to bash. Adding more binaries is the same as adding bash: copy the binary in and then ldd it to find its libraries and then copy them in. A lot of programs use the exact same libraries, so often you'll find that copying the binary in to an existing jail is all you need to do since the libraries will already be there.
Once I copy in a few binaries this could turn into a very nice sandbox. However, I had some specific needs when I started out building custom jails, and there are a few gotchas that the above jail doesn't come close to meeting:
1. Users assigned to a jail need to be automatically chrooted into their jail when they login

2. There should be no shell access --- we don't want them to be able to ssh in, they should only be able to use sftp and scp

3. This needs to be done yesterday, and it needs to be repeatable.

Number one is easy enough in theory, just create a false shell whose sole purpose is to chroot you into your jail. Yeah. Easy. Oh, and do it securely with no shortcuts that might open up security holes. Yup, definitely easy.
Number two actually is easy in theory and practice, just copy in the sftp and scp binaries and leave out the ssh binary and any shell binaries.
Number three is the real kicker. I needed a quick, repeatable process. Especially since once things are up and running the plan was to turn everything over to the support group so that they could create new jails as needed.
So what did I do? I went looking for something someone had already built and I found Jailkit.
There are other tools out there for automatically creating jails, so please do some searching and explore all options when choosing your own solution. Some of the things I like about jailkit are:
1. It has many pre-configured collections that you can mix and match to build the perfect jail. I call them jail-sets.
2. If none of the existing jail-sets meets your needs you can customize them or create new ones.
3. Creating multiple jails is easy and each jail can have multiple users assigned to it --- users can't belong to more than one jail though, but that's a good thing IMO.
4. Adding users to jails is very easy, almost to the point of "that's all there is to it?"
5. The fake shell that Jailkit uses to chroot users when they log in checks for potential security risks every time it is run and will disconnect the user if it notices anything bad (like the jail directory being setuid root, for example).
6. The Jailkit website has clear, easy to follow directions that walk you through several potential jail setups and it has a good troubleshooting section to help you when things don't work for some reason or other.
Here's what I did to create the jails on my server:
As far as the actual server is concerned, I didn't do much to it apart from installing a base ubuntu-server OS with the ssh-server option. Oh, and I also installed the build-essential meta-package so that I could compile and install Jailkit.
After downloading the tarball from the Jailkit website, Jailkit is installed from source like so:

Akhil@groundzero:~$ wget http://olivier.sessink.nl/jailkit/jailkit-2.5.tar.gz
Akhil@groundzero:~$ tar -zxvf jailkit-2.5.tar.gz
Akhil@groundzero:~$ cd jailkit-2.5
Akhil@groundzero:~/jailkit-2.5$ ./configure
Akhil@groundzero:~/jailkit-2.5$ make
Akhil@groundzero:~/jailkit-2.5$ sudo make install

The whole unpack-compile-install process is really quick. By default, Jailkit installs its binaries into /usr/sbin/ and the configuration and template files into /etc/jailkit/.
Once installed, creating jails is easy. As I said above, Jailkit comes with several default jail-sets. These are defined in /etc/jailkit/jk_init.ini and they include things like scp, sftp, apache, basicshell, and so on. As far as learning about what jail-sets are available and what each jail-set includes, the best thing I can say is to look through the /etc/jailkit/jk_init.ini file and familiarize yourself with them. This file is also where you can define your own custom sets.
You create a jail using the jk_init script and specifying where you want the jail to be located and what jail-sets you want included in the jail. The example below is what I use to create a jail that has sftp and scp functionality:

Akhil@groundzero:~$ sudo jk_init -v /path/to/jail sftp scp jk_lsh

The jk_lsh jail-set provides a special shell that limits the binaries it will execute via a config file. That way, even if a user uploads their own binary file, they won't be able to execute it.
The jail at this point (assuming nothing went wrong) is technically a valid jail, but there aren't any users in it yet and because this jail has no 'shell', it is pretty useless to try and chroot into it manually to test it. The next order of business therefore is to create some users and add them to the jail. Once a user has been created and added to a jail using the method below, they will always be put in that shell whenever they log in. So you absolutely need to create special jail-only users. Do not put regular users into jails!

Akhil@groundzero:~$ sudo groupadd companyname
Akhil@groundzero:~$ sudo useradd -g companyname -m username
Akhil@groundzero:~$ sudo passwd username
Akhil@groundzero:~$ sudo jk_jailuser -m -j /path/to/jail username

The reason I create a group is because I am creating jails per client company on this server and it made sense to group (no pun intended) all the users that way. I also threw in the passwd command into the example above since the useradd command (on Ubuntu at least) doesn't set a password by default. Things may be different on your Linux distro of choice.
The jk_jailuser utility, when run as in the example above, changes the shell of an existing user to jk_chrootsh in the /etc/passwd file and it changes the home directory to /path/to/jail/./home/username. It also adds the user to a stripped down passwd file located at /path/to/jail/etc/passwd and adds the user's groups to a stripped down group file located at /path/to/jail/etc/group.
If you want to edit the two passwd files yourself you can do so. Just change the user's line in /etc/passwd to something resembling this:

username:x:1010:1010::/path/to/jail/./home/username:/usr/sbin/jk_chrootsh

and change the user's line in /path/to/jail/etc/passwd to something resembling this:

username:x:1010:1010::/home/username:/usr/sbin/jk_lsh

(I'm afraid of messing things up if I do it manually, so I use the jk_jailuser utility whenever possible.)
Now we need to allow the user to execute the scp and sftp programs. Edit the /path/to/jail/etc/jailkit/jk_lsh.ini file and add in some lines like so:

[group companyname]
paths= /usr/bin, /usr/lib/
executables= /usr/bin/scp, /usr/lib/sftp-server, /usr/lib/openssh/sftp-server, /usr/libexec/sftp-server

That's all there is to it. I now have a fully functioning jail and I can test it by connecting to the server using sftp or scp. When I sftp in and look around, I only see the directories below the jail directory, exactly the behavior I wanted.
Should you find later on that the jail needs to have some other binary in it, Jailkit also includes a handy copy command that you can use to copy additional files into jails like so:

Akhil@groundzero:~$ sudo jk_cp /path/to/jail /path/to/command

Very simple, and it handles all of the locating of dependencies, libraries, and whatnot. The only gripe I have about the jk_cp command is that you have to specify jk_cp
instead of doing it the way the cp command works, which is cp . On occasion I have run into issues where Jailkit doesn't copy over everything properly, or it simply misses some libraries. In those cases I had to experiment with copying over additional libraries. Rather than reinvent the wheel, I'll just point you to the Jailkit website, which has a very nice page on troubleshooting jails. (it's in the links)
Once I had jailkit working on my jail server and I had a repeatable process for adding users, my next step was to create two _very_ simple shell scripts for creating jails and creating jailed users. These scripts are used by our support staff, who have limited sudo access to run only these scripts as root along with the passwd command (so that they can change jailed users' passwords, if need be).
The two scripts are called mkjail and jailuser. They're not all that pretty, and they could be considered dangerous if used by the wrong people. But for internal use by a select group of specifically authorized and trained people, they work fine.
With the server set up and the scripts in place and the staff trained my job was done. The server is under constant use and new jails are created whenever they are needed with no intervention needed on my part. Of course, the minute the project was completed I was immediately assigned three more . . . but that's what's great about working in technology, right? It's never boring, that's for sure. :-)
Admittedly, creating an sftp jail is probably not something that everyone will need to do, but jailkit has found its way into my sysadmin bag of tricks. Whenever I need to isolate a user or a process into a sandbox, and a Xen virtual machine is way too much overkill, the first thing I reach for is jailkit.

By: Unknown On 3:31 PM

Continue Reading

Svn backup

Whatever your reason for migrating repository history, using the svnadmin dump and svnadmin load subcommands is straightforward.svnadmin dump will output a range of repository revisions that are formatted using Subversion's custom filesystem dump format. The dump format is printed to the standard output stream, while informative messages are printed to the standard error stream. This allows you to redirect the output stream to a file while watching the status output in your terminal window. For example:
$ svnlook youngest myrepos
26
$ svnadmin dump myrepos > dumpfile
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
…*
Dumped revision 25.
* Dumped revision 26.At the end of the process, you will have a single file (dumpfile in the previous example) that contains all the data stored in your repository in the requested range of revisions. Note that svnadmin dump is reading revision trees from the repository just like any other “reader” process would (e.g., svn checkout), so it's safe to run this command at any time.The other subcommand in the pair, svnadmin load, parses the standard input stream as a Subversion repository dump file and effectively
replays those dumped revisions into the target repository for that operation. It also gives informative feedback, this time
using the standard output stream:
$ svnadmin load newrepos <><<<>>>
<<<>>>
<<<>------- Committed new rev 25 (loaded from original rev 25) >>>
<<<>------- Committed new rev 26 (loaded from original rev 26) >>>The result of a load is new revisions added to a repository—the same thing you get by making commits against that repository from a regular Subversion client. Just as in a commit, you can use hook programs to perform actions before and after each of the commits made during a load process. By passing the --use-pre-commit-hook and --use-post-commit-hook options to svnadmin load, you can instruct Subversion to execute the pre-commit and post-commit hook programs, respectively, for each
loaded revision. You might use these, for example, to ensure that loaded revisions pass through the same validation steps that regular commits pass through. Of course, you should use these options with care—if your post-commit hook sends emails to a mailing list for each new commit, you might not want to spew hundreds or thousands of commit emails in rapid succession at that list! You can read more about the use of hook scripts in the section called “Implementing Repository Hooks”.
Note that because svnadmin uses standard input and output streams for the repository dump and load processes, people who are feeling especially saucy can try things such as this (perhaps even using different versions of svnadmin on each side of the pipe):
$ svnadmin create newrepos
$ svnadmin dump oldrepos svnadmin load newreposBy default, the dump file will be quite large—much larger than the repository itself. That's because by default every version of every file is expressed as a full text in the dump file. This is the fastest and simplest behavior, and it's nice if you're piping the dump data directly into some other process (such as a compression program, filtering program, or loading process). But if you're creating a dump file for longer-term storage, you'll likely want to save disk space by using the --deltas option. With this option, successive revisions of files will be output as compressed, binary differences—just as file revisions are stored in a repository.
This option is slower, but it results in a dump file much closer in size to the original repository.
We mentioned previously that svnadmin dump outputs a range of revisions. Use the --revision (-r) option to specify a single revision, or a range of revisions, to dump. If you omit this option, all the existing repository revisions will be dumped.
$ svnadmin dump myrepos -r 23 > rev-23.dumpfile
$ svnadmin dump myrepos -r 100:200 > revs-100-200.dumpfile
As Subversion dumps each new revision, it outputs only enough information to allow a future loader to re-create that revision based on the previous one. In other words, for any given revision in the dump file, only the items that were changed in that revision will appear in the dump. The only exception to this rule is the first revision that is dumped with the current svnadmin dump command.
By default, Subversion will not express the first dumped revision as merely differences to be applied to the previous revision. For one thing, there is no previous revision in the dump file! And second, Subversion cannot know the state of the repository into which the dump data will be loaded (if it ever is). To ensure that the output of each execution of svnadmin dump is self-sufficient,
the first dumped revision is, by default, a full representation of every directory, file, and property in that revision of the repository. However, you can change this default behavior. If you add the --incremental option when you dump your repository, svnadmin will compare the first dumped revision against the previous revision in the repository—the same way it treats every other revision that gets dumped. It will then output the first revision exactly as it does the rest of the revisions in the dump range—mentioning only the changes that occurred in that revision. The benefit of this is that you can create several small dump files that can be loaded in succession, instead of one large one, like so:$ svnadmin dump myrepos -r 0:1000 > dumpfile1
$ svnadmin dump myrepos -r 1001:2000 --incremental > dumpfile2
$ svnadmin dump myrepos -r 2001:3000 --incremental > dumpfile3
These dump files could be loaded into a new repository with the following command sequence:
$ svnadmin load newrepos <>

$ svnadmin load newrepos <>Another neat trick you can perform with this --incremental option involves appending to an existing dump file a new range of dumped revisions. For example, you might have a post-commit hook that simply appends the repository dump of the single revision that triggered the hook. Or you might have a script that runs nightly to append dump file data for all the revisions that were added to the repository since the last time the script ran. Used like this, svnadmin dump can be one way to back up changes to your repository over time in case of a system crash or some other catastrophic event.
The dump format can also be used to merge the contents of several different repositories into a single repository. By using the --parent-dir option of svnadmin load, you can specify a new virtual root directory for the load process. That means if you have dump files for three repositories—say calc-dumpfile, cal-dumpfile, and ss-dumpfile—you can first create a new repository to hold them all:
$ svnadmin create /var/svn/projects
$Then, make new directories in the repository that will encapsulate the contents of each of the three previous repositories:
$ svn mkdir -m "Initial project roots" \
file:///var/svn/projects/calc \
file:///var/svn/projects/calendar \
file:///var/svn/projects/spreadsheet
Committed revision 1.
$Lastly, load the individual dump files into their respective locations in the new repository:
$ svnadmin load /var/svn/projects --parent-dir calc <>

…$ svnadmin load /var/svn/projects --parent-dir calendar <>

…$ svnadmin load /var/svn/projects --parent-dir spreadsheet <> We'll mention one final way to use the Subversion repository dump format—conversion from a different storage mechanism or version control system altogether. Because the dump file format is, for the most part, human-readable, it should be relatively easy to describe generic sets of changes—each of which should be treated as a new revision—using this file format. In fact, the cvs2svn utility (see the section called “Converting a Repository from CVS to Subversion”) uses the dump format to represent the contents of a CVS repository so that those contents can be copied into a Subversion repository.

By: Unknown On 2:54 PM

Continue Reading