Purpose

The purpose of this utility is to regularly scan and new entries in the mailwatch database an build up a profile of all the machines which connect and send email. Then based upon a few rules it can decide whether a particular IP address should be temporarily blocked. A database table of all IP addresses which should be blocked is maintained (which can be used directly by Postfix) and optionally two text files exported which are in rbldnsd and user defined format.

How it works

The main application is run from a crontab file at least once per hour and typically every few minutes. It maintains an hourly history gojng back 23 hours about each IP address that sends mail through Mailscanner. The information which is recorded about each IP address is the number of mails sent, the number of definite ham messages and the number of definite spam messages. The threshold for definite spam and ham messages are defined in terms of their spamassassin score. Defaults are ham is a score of less than 5 (the spamassassin default) and spam is a score of more than 20. Messages with a score of between the ham and spam threshold are therefore for all practical purposes ignored (a spammer will still get blocked if they send some high scoring spams even if a lot more spam just goes over the spamassassin spam threshold).

An IP address is decided to be blocked when it has sent more than a configured number of definite spam messages in the previous 23 hours and have not sent and definite ham messages. The IP address is then put into a table of blocked IP addresses together with information when the record should expire (configurable in x number of hours) and a customisable error message which supports some variables. Then if enabled the text files are written.

More detailed information

Configurable variables

For a client to be identified as a spammer a number of conditions must apply :-

  1. They must have sent at least minspamcount definite spams in the last 23 hours.
  2. They must not have sent any definite hams.
spamscore Threshold for definite spam. Default is 10. Any spam score above this is classed as a definite spam.
hamscore Threshold for definite ham. Default is 5. Any spam score below this is definite ham.
Anything with a score of between hamscore and spamscore is a grey area and not used. Therefore you can set spamscore fairly high and not have to worry about a lower scored spam mail coming from the client causing them not to be blocked.
minspamcount The minimum number of definite spams that must be received within a 23 hour period for the client to be considered for blocking.
blocktime The number of hours a client ip should be blocked for. Note that once blocked they will no longer be appearing in the mailwatch table so having a short block time could result in the same IP address regularly being blocked and unblocked. The default is 24 hours.
errtemplate

The error message which is written into the database of blocked IP addresses and which is also reported to the connecting IP address when the mail is refused. If you are using Postfix to query the block table directory you will want to make sure that this error template starts with the 'REJECT' keyword so that postfix rejects the message. The following variables are supported and will be replaced by their corresponding values :-

[ip] - the ip address of the connecting client
[count] - the number of messages sent
[spam] - the number of definite spams sent
[expires] - date and time when the block will expire e.g 2007-09-22 21:53:28
[expires2] - date and time when the block will expire in rfc2822 format e.g Sat, 22 Sep 2007 21:53:28 +0100
[expires3] - date and time when the block will expire in rfc2822 but timezone by name e.g Sat, 22 Sep 2007 21:53:28 (BST)
[hamscore] - the threshold for a definite ham
[spamscore] - the threshold for a definite spam
[minspamcount] - the required number of definite spams in order to be blocked

rbltempfile

The name of the temporary rbl file to be generated. Since rbldnsd monitors for changes you dont want to erase the contents of a file and then start populating otherwise rbldnsd could read it while it is still being written.

The default is /tmp/mailwatch2rbl.rbl and should be a safe default. If the final rbl file is stored on a different mount point then you should specify a different temp file so that the file move operation does not have to perform a copy across differnt disks which is much slower.

rblfile The rbl file to be written. Used by rbldnsd. Leave blank to disable this feature.
rblsoa SOA record to put in the rbldnsd file. This configuration item has been added so that negative lookups can be cached which reduces the queries to the nameserver.
rblns NS record to put in the rbldnsd file
rblmode

'simple' is where the complete error text (obtained from the template) is written into the rbl file. This does however cause the rbl file to become quite large and consume more memory so it is only recomended for small sites.

The new default is 'advanced'. This makes the rbl file much smaller but you have a more limited amount of information it can contain as it relies on its own template. For example any manual blacklist added has the rbl file entry set according to the template and not the custom error message that was defned.

rbls1
rbls2
rbltemplate

rbls1 and rbls2 are two sets of strings which can be referenced in the rbl template as $1 and $2. The rbltemplate is defined in the rbl file itself as rbldnsd has its own string expanding function. The rbltemplate supports the following vaiables :-

$1 - the string defined in rbls1
$2 - the string defined in rbls2
$ - the IP address of the connecting client
[expires] - date and time when the block will expire e.g 2007-09-22 21:53:28
[expires2] - date and time when the block will expire in rfc2822 format e.g Sat, 22 Sep 2007 21:53:28 +0100
[expires3] - date and time when the block will expire in rfc2822 but timezone by name e.g Sat, 22 Sep 2007 21:53:28 (BST)

customtempfile The name of the custom temporary file to be written.

customfile

The name of the custom file to be written. Leave blank to disable this feature.
customtemplate

A template for the custom text file which can create files compatible with other MTAs such as sendmail. The following variables are supported :-

[ip] - the ip address of the connecting client
[errormsg] - the error messages which in turn is generated from the errtemplate configuration item.

customcmd Custom command to be run if the new custom file is different to the old one. Note a new custom file is always written (if enabled) but the command is only run if the file has changed.

Download

mailwatch2rbl version 1.1

Tools

mw2rbltool <option(s)>

whitelist <ipaddress> Whitelists the given IP address so that it will never be blacklisted
blacklist <ipaddress> <expires> <message>

Blacklists the given IP address until <expires> while giving the error message <message> to any connections.
<expires> is any valid php strtotime string. Some examples are :-

+1day
"20 december 2009"
2008-01-01
"next tuesday"

clearip <ipaddress> Clears all history of the IP address including a whitelist or blacklist entry.
show whitelist Shows a list of all whitelisted IP addresses.
show blacklist Shows a list of all blacklisted IP addresses, their expiry date and the country of origin (using GeoIP).
show ip <ipaddress> Shows all information about the specified IP address including the history over the past 23 hours and full details if it is currently blacklisted.
show top spam Shows the details of the top 25 IP addresses which have sent the most spam.
show top ham Shows the details of the top 25 IP addresses which have sent the most ham .
show top messages Shows the details of the top 25 IP addresses which have sent the most messages .

Bugs

Mailwatch stores the date and time in the local timezone with no timezone indication. Therefore mailwatch2rbl also has to work in the local timezone and this causes some minor timezone related issues :-

  1. When summer time starts the clock effectivly jumps forward by typically 1 hour.Therefore at 2am the clock will jump forward to 3am and because there was no processing done during the 02 hour the data for the previous days 03 hour has not been erased and new data will append to it. Therefore when the time jumps forward some data may fail to be expired correctly.
  2. When summer time ends the clock jumps back from say 2am to 1am. The result of this is that the previous hours data is effectivly lost.
  3. All expiry times contained in the block table are local and therefore if they are referenced in the error template care should be taken to also mention the timezone it refers to.