Monday, November 23, 2009

Understanding DNSBL Filtering

SkyHi @ Monday, November 23, 2009
A DNSBL (commonly known as a 'Blocklist") is a database that is queried in realtime by Internet mail servers for the purpose of obtaining an opinion on the origin of incoming email. The role of a DNSBL such as Spamhaus' SBL/XBL/PBL Advisory system is to provide an opinion, to anyone who asks, on whether a particular IP Address meets Spamhaus' own policy for acceptance of inbound email.

Basic DNSBL flow:



The policy of the Receiver governs whether a message is blocked or not

Every Internet network that chooses to implement spam filtering is, by doing so, making a policy decision governing acceptance and handling of inbound email. The Receiver unilaterally makes the choices on whether to use DNSBLs, which DNSBLs to use, and what to do with an incoming email if the email message's originating IP Address is "listed" on the DNSBL. The DNSBL itself, like all spam filters, can only answer whether a condition has been met or not.




Points to note are:

1. The Receiver makes a policy decision to accept, tag or reject the message based on, amongst other things, whether the Sender's IP Address is listed on a Spamhaus DNSBL.

2. The Receiver is asking Spamhaus a question ("is this IP listed on a Spamhaus database?"). Spamhaus replies automatically to the Receiver's question.

3. Spamhaus has no control over how the Receiver chooses to handle incoming email from IPs which are listed on a Spamhaus database.

4. Instead of rejecting messages that do not pass the Spamhaus test, many Receivers tag such messages. Tagged messages are then either placed in a 'Junk' mailbox or are put through further, more processor-intensive spam filters (such as SpamAssassin, etc.).

5. Spamhaus does however recommend that networks using the Spamhaus DNSBLs reject (bounce) rather than simply trash incoming emails from IP Addresses listed on a Spamhaus database. The reason for this is that rejecting email (with a hard '5**' during the SMTP phase and before the email body is accepted) maintains the fundemental rule of email deliverability: If an email can not be delivered *always let the Sender know*.


The Rights of a Sender -versus- the Rights of a Receiver

The Internet is a network of private networks. Each network sets its own policy for what email it will or will not accept. In the following diagram, the end of the Sender's private network and the beginning of the Receiver's private network are marked (A) and (B). This diagram demonstrates that no 'blocking' of email occurs either on the exit from the Sender's network, the Sender's connection into the Public Internet, or even at the entrance to the Receiving network. DNSBLs are used by the Receiver's private mail server and from within the Receiver's private network.



A common misconception is that an email Sender whose IP address is listed on a DNSBL is 'blocked' from sending out email. In fact Senders are in no way prevented by DNSBLs from sending email. The Spamhaus DNSBLs are used only by receiving mail systems on private networks and are used voluntarily.

Spamhaus does not tell a 3rd-party mail system what to do with an item of email, the 3rd-party mail system asks Spamhaus for an opinion and Spamhaus responds to that request with its opinion. In effect the receiving mail server asks the Spamhaus DNSBL "Does this Sender's IP Address exist on the Spamhaus database?", the Spamhaus DNSBL simply responds with a "Yes" if present or, if not present does not respond at all (no response means "we have no opinion on that IP Address").

Other DNSBLs?

SkyHi @ Monday, November 23, 2009
We believe that an effective spam filtering system is a hybrid of a number of techniques, you should never put all your eggs in one basket. See Effective Spam Filtering for an excellent discussion of modern spam fighting techniques along with other tools.

In addition to the excellent SpamHaus SBL, XBL and PBL subzones, here are a few other DNSBLs that you may wish to consider. It is extremely important that you evaluate them according to your needs. Some of these lists are NOT appropriate for certain environments.

Before using DNSBLs, we recommend becoming familiar with the DNSBL lookup tools on dnsstuff.com.

Jeff Makey provides a useful Blacklists Compared page.

Only those DNSBLs we have personal experience with are listed here. While reading these, consider your options - they can either be used in a full blocking mode (a DNSBL hit means the email is blocked), or, as part of a scoring system (a DNSBL hit plus other "scores" are required for a block).

1. NJABL. Like Spamhaus, NJABL is a reliable and responsible DNSBL has a number of subzones. The NJABL proxy DNSBL is incorporated in the Spamhaus XBL (along with CBL). The NJABL Dynablock list has been decommissioned in favor of the Spamhaus PBL, and is now just a mirror of the PBL. If you're using NJABL Dynablock, replace it with PBL lookup/naming conventions, some time in the not too distant future, the "NJABL named" version of the PBL will probably disappear. The NJABL open relay DNSBL is also good, but does not yield many hits these days.
2. WPBL. This is a good, reliable and responsible DNSBL, however, as it has very low thresholds (and somewhat limited coverage) it is strongly recommended that it not be used as a single reason for email rejection - this is discussed on their web page. It should be used in a scoring system such as SpamAssassin.
3. Spamcop. SpamCop is a good, solid, professionally operated DNSBL. Due to the way it's implemented, it used to occasionally "throw" undesirable false positives, and it was best used in a scoring system. Since then, changes have been made, and using it as an outright blocking mechanism is a reasonable choice.
4. Invaluement DNSBL [Note: Commercial] ivmURI and ivmSIP are good solid and professionally operated lists. ivmURI is a URI (domain) DNSBL like SURBL or URIBL, with high effectiveness (comparable with URIBL/SURBL), extremely low false positives, and quick to list. ivmSIP is a IP-based DNSBL which is particularly good at catching "new" emitters. Its FP rate is quite low. Both of which shouldn't be considered substitutes for Zen/Spamcop, but do complement them well.
5. SORBS The SORBS open relay, open socks and open proxy lists are good (noting that listing expiration is extremely long), but the other lists should not be used (especially dynamic), except in a scoring system with "moderate" scores.
6. SPEWS The SPEWS list is dead - DO NOT USE. The SPEWS downloadable zone hasn't been updated since August of 2006, and most mirrors have emptied the zone. In its heydey (years ago), SPEWS was reasonably reliable, but generally not useable except in hobbyist mail server situations because of false positives and the difficulty in getting delistings, and otherwise only useable as part of a scoring system.
7. APEWS When it became apparent that SPEWS was no longer being maintained, someone, or a group of someones, copied the SPEWS web pages and presumably the SPEWS list of the time, and operated it as a new DNSBL "APEWS". The new operators are far more aggressive than SPEWS ever was, and will list large chunks of net space over a single third party incident report that may not have had anything to do with spam. Eg: APEWS has been known to list entire netblocks because of a single out of date CERT report of a single IP acting as a bot C&C.

APEWS is reportedly blocking 2/3rds of all useable Internet IP space.

APEWS false positives in most situations are extremely high, and it should not be used except in some very specific circumstances (eg: single user systems via scoring). The main reason we mention APEWS is that DNSSTUFF queries APEWS listings, and it tends to alarm listees and cause long flamewars on the only places that people can find to discuss them (eg: news.admin.net-abuse.email), with no useful result. APEWS provides no mechanism for appealing listings, and we believe that is not best practise for DNSBL operation.

As far as we can determine, few (if any) mail servers actually use APEWS, so, an APEWS listing is largely meaningless. Getting out of APEWS is very difficult, and APEWS can just about be completely ignored as being irrelevant.

Most recent news: APEWS may no longer be queriable - most, if not all, of it's "mirror/publishers" have withdrawn offering it (eg: APEWS listed at least one of its own mirrors).
8. TQMcube lists (it has several) appeared popular and reasonably effective, however, the admin has completely vanished (we're rather worried about him), and the list appears to be on autopilot now.
9. Regional DNSBLs. In some cases it may be desirable to use a DNSBL that lists certain regions of the world - for example, if you don't need or want to correspond in email with anyone in China, you can use a DNSBL specifically designed to list all IPs in China.

There are a number of these lists, the best known is korea.services.net,

WARNING: blackholes.us has gone out of service, and briefly it was blacklisting the world.

BE AWARE that if you use such a service, you will get very little if any email from these regions. These list IPs in those regions, not IPs in those regions known to spam. Use them at your own risk. Or in a scoring system.
10. ORDB, OSIRUS, MONKEYS, DSBL: just in case: these DNSBLs are defunct and should NOT be used.

Blacklists Compared

SkyHi @ Monday, November 23, 2009
Latest Report

* 14 November 2009

Background
There are two types of DNS blacklists. The older, more common type lists IPv4 addresses as DNS domains, so if a hypothetical DNS list blacklist.example.org were to list the loopback address 127.0.0.1 it could be found by looking for a DNS "A" (address) record for 1.0.0.127.blacklist.example.org. "A" records exist for listed IP addresses, and do not exist for those that are unlisted.

The less common type of DNS blacklist lists domains by name. In this case, a listing of the domain example.com could be found in the hypothetical list by looking for a DNS "A" record for example.com.blacklist.example.org. Again, "A" records exist for listed domains and do not exist for those that are unlisted.

Methodology
Public DNS blacklists are compared here using a sampling method. Each week's sample is extracted from the sendmail logs generated by SDSC's primary mail server, and the data covers a 1-week period beginning and ending each Saturday approximately 4:05 AM US/Pacific time.

The sample of IP addresses used in a comparison report is the entire set of IP addresses that are logged as having made SMTP connections to the mail server during the week. Reverse DNS lookups are done on each of the IP addresses to get a list of domain names, and lookups of those domain names are done in each of the domain name blacklists. The sample of domains used in the last comparison report is the entire set of SMTP envelope sender domains that are logged as having been presented to the mail server during the week. To maintain SDSC e-mail privacy the lists of specific IP addresses and domains are not available to the public.

For each of the IP addresses and domains in the sample lists, lookups are done in each of the DNS blacklists. There are millions of DNS lookups required to complete the survey, and to complete it within a week the work is done using multiple parallel threads.

Conclusions
The results of this survey are not necessarily helpful for choosing a blacklist for the purpose of blocking spam. Such an evaluation would require good data on which of the thousands of e-mail messages handled by SDSC every week are truly spam and which are not, but if it were easy to automatically tell the difference we would just block the spam and not worry about blacklists. Without that data there can be no objective measure of a blacklist's effectiveness for blocking spam.

This survey also does not attempt to measure the quality of blacklists in terms of erroneous listings (false positives) nor in terms of missing entries (false negatives) with respect to each blacklist's policy. To do so would require maintaining my own lists for comparison with the other blacklists, which would take far more effort than I am willing to expend.

That said, it is my subjective opinion that most of the blacklists surveyed here are at least somewhat useful for blocking spam. There is a tendency for those with the highest number of hits to list many places that send substantial quantities of non-spam, and at the other end of the scale some blacklists (such as those listing specific ISPs) are so narrowly focused that they are good for blocking a relatively small fraction of spam with a correspondingly small probability of false positives. To the best of my knowledge all of the blacklist operators make an honest effort to maintain their listings according to their published policies, but the blacklists that cover less than 1% of the survey space seem to be too ineffective to be worthwhile. Your mileage may vary.

For the record, SDSC uses these blacklists:

* dsn.rfc-ignorant.org
* zen.spamhaus.org
* dul.dnsbl.sorbs.net
* bl.spamcop.net

to great effect with only a few mitigating whitelist entries. A local blacklist with several thousand entries and some custom sendmail rules round out our automated spam blocking efforts.

Blacklists Compared

SkyHi @ Monday, November 23, 2009
Survey results for all known public IP-based DNS blacklists. Lookups were done on connecting IP addresses. The "union of most IP zones" line excludes the hostkarma.junkemailfilter.com (all results but 127.0.0.2, 127.0.0.3, and 127.0.0.4), exemptions.ahbl.org, query.bondedsender.org, list.dnswl.org, accredit.habeas.com, iadb.isipp.com, and iadb2.isipp.com zones because they are not blacklists. Because it is too aggressive to be widely useful the l2.apews.org zone is also excluded from the union.

Hits DNS Zone
170583 (total number of IP addresses tested, including 182 at SDSC)
167027 (union of most IP zones)
154048 b.barracudacentral.org
151539 zen.spamhaus.org (union of all results)
143380 l2.apews.org
116790 cbl.abuseat.org
116704 zen.spamhaus.org (result 127.0.0.4 = Composite Blocking List)
110585 dnsbl-3.uceprotect.net
109629 dnsbl-1.uceprotect.net
108983 hostkarma.junkemailfilter.com (result 127.0.0.2 = spam source)
108857 dnsbl-2.uceprotect.net
106530 dnsbl.sorbs.net (union of all results)
98651 zen.spamhaus.org (result 127.0.0.11 = Spamhaus PBL, Spamhaus entry)
98430 psbl.surriel.com
81072 dnsbl.sorbs.net (result 127.0.0.7 = hacked/vulnerable)
80507 ubl.unsubscore.com
64818 bl.spameatingmonkey.net
49453 blackholes.five-ten-sg.com (union of all results)
45645 blackholes.five-ten-sg.com (result 127.0.0.2 = spam source)
44228 hostkarma.junkemailfilter.com (result 127.0.1.1 = MTA sends QUIT)
40680 no-more-funn.moensted.dk (union of all results)
38401 db.wpbl.info
38043 dnsbl.sorbs.net (result 127.0.0.10 = dialup)
33895 bl.spamcop.net
31653 bl.score.senderscore.com
25521 no-more-funn.moensted.dk (result 127.0.0.3 = dialup)
24715 zen.spamhaus.org (result 127.0.0.10 = Spamhaus PBL, ISP contributed)
20635 spam.dnsbl.sorbs.net
18567 hostkarma.junkemailfilter.com (result 127.0.1.2 = MTA does not send QUIT)
18243 ix.dnsbl.manitu.net
13565 mail-abuse.blacklist.jippg.org
11004 no-more-funn.moensted.dk (result 127.0.0.2 = spam source)
10832 dnsbl.inps.de
10276 korea.services.net
9766 bl.spamcannibal.org
7878 ips.backscatterer.org
4963 spamsources.fabel.dk
3838 list.dnswl.org (not a blacklist!)
2880 hostkarma.junkemailfilter.com (result 127.0.0.1 = whitelisted)
2815 blackholes.five-ten-sg.com (result 127.0.0.9 = misc)
2565 no-more-funn.moensted.dk (result 127.0.0.10 = open proxy)
2266 dnsbl.njabl.org (union of all results)
2166 hostkarma.junkemailfilter.com (result 127.0.0.5 = do not blacklist)
2028 hostkarma.junkemailfilter.com (result 127.0.0.3 = mix of spam and nonspam)
1853 dnsbl.njabl.org (result 127.0.0.9 = open proxy)
1420 no-more-funn.moensted.dk (result 127.0.0.9 = misc)
1378 tr.countries.nerd.dk
1134 zen.spamhaus.org (union of SBL and SBLCSS results)
976 blackholes.five-ten-sg.com (result 127.0.0.4 = unconfirmed opt-in)
801 zen.spamhaus.org (result 127.0.0.2 = Spamhaus SBL)
574 aspews.ext.sorbs.net
512 accredit.habeas.com (not a blacklist!)
462 l2.bbfh.ext.sorbs.net
459 dnsbl.sorbs.net (result 127.0.0.6 = spam source)
404 query.bondedsender.org (not a blacklist!)
333 zen.spamhaus.org (result 127.0.0.3 = Spamhaus SBLCSS)
240 no-more-funn.moensted.dk (result 127.0.0.7 = spam haven)
213 dnsbl.njabl.org (result 127.0.0.4 = spam source)
200 dnsbl.njabl.org (result 127.0.0.2 = open relay)
180 l1.bbfh.ext.sorbs.net
167 dnsbl.ahbl.org (union of all results)
161 iadb2.isipp.com (not a blacklist!)
139 dnsbl.ahbl.org (result 127.0.0.3 = open proxy)
130 iadb.isipp.com (not a blacklist!)
96 hostkarma.junkemailfilter.com (result 127.0.0.4 = spam source aspirant)
95 tor.dnsbl.sectoor.de (result 127.0.0.2 = /24 contains a Tor server)
95 tor.dnsbl.sectoor.de (union of all results)
53 bl.deadbeef.com
32 dnsbl.sorbs.net (result 127.0.0.3 = open socks proxy)
31 dnsbl.sorbs.net (result 127.0.0.2 = open http proxy)
28 dnsbl-0.uceprotect.net
28 dnsbl.ahbl.org (result 127.0.0.4 = spam source)
18 multi.uribl.com (union of all results)
17 multi.uribl.com (result 127.0.0.2 = spam resource)
14 blackholes.five-ten-sg.com (result 127.0.0.12 = spam-friendly freemail provider)
10 blackholes.brainerd.net
9 zen.spamhaus.org (result 127.0.0.5 = time-expired NJABL open proxy)
4 hostkarma.junkemailfilter.com (result 127.0.2.3 = domain first seen more than 7 days ago)
3 multi.surbl.org
3 spamguard.leadmon.net (union of all results)
2 dnsbl.sorbs.net (result 127.0.0.4 = open proxy)
2 spamguard.leadmon.net (result 127.0.0.7 = spam source netblock)
2 blackholes.five-ten-sg.com (result 127.0.0.11 = TCPA violator)
1 multi.uribl.com (result 127.0.0.8 = new domain or concealed contact)
1 dnsbl.sorbs.net (result 127.0.0.9 = zombie network)
1 dnsbl.sorbs.net (result 127.0.0.5 = open relay)
1 spamguard.leadmon.net (result 127.0.0.8 = open proxy)
1 no-more-funn.moensted.dk (result 127.0.0.4 = unconfirmed opt-in)
1 blackholes.five-ten-sg.com (result 127.0.0.6 = open relay)
Survey results for all known public domain-name-based DNS blacklists. Lookups were done on the domain names of connecting IP addresses. The "union of most domain zones" line includes data from the hostkarma.junkemailfilter.com zone only when the results are 127.0.0.2, 127.0.0.3, or 127.0.0.4 because other results are not blacklists.

Hits DNS Zone
170583 (total number of IP addresses whose names were tested, including 182 at SDSC)
99463 hostkarma.junkemailfilter.com (result 127.0.2.3 = domain first seen more than 7 days ago)
87202 hostkarma.junkemailfilter.com (result 127.0.1.1 = MTA sends QUIT)
81646 (union of most domain zones)
70614 l1.apews.org
66144 abuse.rfc-ignorant.org
31279 dynamic.rhs.mailpolice.com
27920 whois.rfc-ignorant.org (union of all results)
20975 whois.rfc-ignorant.org (result 127.0.0.7 = no whois data at all)
10228 hostkarma.junkemailfilter.com (result 127.0.0.5 = do not blacklist)
6946 whois.rfc-ignorant.org (result 127.0.0.5 = bad whois data)
3726 webmail.rhs.mailpolice.com
3049 hostkarma.junkemailfilter.com (result 127.0.0.1 = whitelisted)
846 adv.rhs.mailpolice.com
752 hostkarma.junkemailfilter.com (result 127.0.1.4 = no verify host)
643 hostkarma.junkemailfilter.com (result 127.0.0.2 = spam source)
571 urired.spameatingmonkey.net
344 hostkarma.junkemailfilter.com (result 127.0.0.3 = mix of spam and nonspam)
340 hostkarma.junkemailfilter.com (result 127.0.1.2 = MTA does not send QUIT)
294 bl.deadbeef.com
206 hostkarma.junkemailfilter.com (result 127.0.2.2 = domain first seen in last 7 days)
149 rhsbl.sorbs.net (result 127.0.0.11 = domain uses bad address space)
149 rhsbl.sorbs.net (union of all results)
72 postmaster.rfc-ignorant.org
43 bogusmx.rfc-ignorant.org
41 bulk.rhs.mailpolice.com
40 dsn.rfc-ignorant.org (zone not intended for this use)
39 multi.surbl.org
36 multi.uribl.com (result 127.0.0.2 = spam resource)
36 multi.uribl.com (union of all results)
34 rhsbl.ahbl.org
19 fresh.spameatingmonkey.net
16 dob.sibl.support-intelligence.net
9 hostkarma.junkemailfilter.com (result 127.0.2.1 = domain first seen today)
5 porn.rhs.mailpolice.com
Survey results for all known public domain-name-based DNS blacklists. Lookups were done on SMTP sender domains. The "union of most domain zones" line includes data from the hostkarma.junkemailfilter.com zone only when the results are 127.0.0.2, 127.0.0.3, or 127.0.0.4 because other results are not blacklists.

Hits DNS Zone
74060 (total number of domains tested, including 164 at SDSC)
68987 hostkarma.junkemailfilter.com (result 127.0.2.3 = domain first seen more than 7 days ago)
19002 (union of most domain zones)
8577 hostkarma.junkemailfilter.com (result 127.0.1.1 = MTA sends QUIT)
7303 whois.rfc-ignorant.org (union of all results)
6826 whois.rfc-ignorant.org (result 127.0.0.7 = no whois data at all)
6567 hostkarma.junkemailfilter.com (result 127.0.0.5 = do not blacklist)
4738 urired.spameatingmonkey.net
4271 abuse.rfc-ignorant.org
3073 hostkarma.junkemailfilter.com (result 127.0.0.1 = whitelisted)
2348 l1.apews.org
1710 postmaster.rfc-ignorant.org
1614 hostkarma.junkemailfilter.com (result 127.0.0.2 = spam source)
1361 bogusmx.rfc-ignorant.org
1276 multi.surbl.org
1035 multi.uribl.com (union of all results)
1023 multi.uribl.com (result 127.0.0.2 = spam resource)
1007 webmail.rhs.mailpolice.com
807 hostkarma.junkemailfilter.com (result 127.0.2.2 = domain first seen in last 7 days)
654 hostkarma.junkemailfilter.com (result 127.0.0.3 = mix of spam and nonspam)
568 dsn.rfc-ignorant.org
477 whois.rfc-ignorant.org (result 127.0.0.5 = bad whois data)
336 adv.rhs.mailpolice.com
283 dynamic.rhs.mailpolice.com
252 porn.rhs.mailpolice.com
100 bulk.rhs.mailpolice.com
51 hostkarma.junkemailfilter.com (result 127.0.1.2 = MTA does not send QUIT)
42 hostkarma.junkemailfilter.com (result 127.0.1.4 = no verify host)
37 fresh.spameatingmonkey.net
34 rhsbl.sorbs.net (result 127.0.0.11 = domain uses bad address space)
34 rhsbl.sorbs.net (union of all results)
28 hostkarma.junkemailfilter.com (result 127.0.2.1 = domain first seen today)
28 dob.sibl.support-intelligence.net
20 ex.dnsbl.org
20 rhsbl.ahbl.org
12 multi.uribl.com (result 127.0.0.4 = opt-out spam resource)
2 fraud.rhs.mailpolice.com
2 bl.deadbeef.com
1 hostkarma.junkemailfilter.com (result 127.0.1.3 = MTA somtimes sends QUIT)

DNSBL: Configuring Sendmail for DNS-Based Blacklisting

SkyHi @ Monday, November 23, 2009
Contents

* Introduction
* Step 1: Choose Which Blacklist(s) to Use
* Step 2: Modify sendmail.cf for Blacklisting
o Optional: Enable DNSBL for Designated Users Only
o Alternate Disabling Option: Exempt Specified Local Users from Blacklisting
o Optional: Customize the Error Message
o Optional: Change Default Behavior When Lookups Fail Temporarily
* Appendices
o Appendix A: Notes on Creating Your Own DNS Blacklist
o Appendix B: Enhanced DNS BlackLists
o Appendix C: Subscriber-Only Blacklists--mail-abuse.org
o Appendix D: Name Server Issues
* Afterword

Introduction

These instructions describe how to modify sendmail's configuration file to enable the dnsbl (DNS Blacklist) feature to block incoming e-mail from IP addresses that are listed on one or more blacklists. These particular blacklists are checked by sendmail during the SMTP conversation, avoiding the generation of the bounce messages that are generated (for example) when SpamAssassin--called from procmail--consults DNS Blacklists. (The configuration of SpamAssassin to consult DNS Blacklists is outside the scope of this document.)

Here is how DNSBLs work:

1. A mail server that wants to send mail to your mail server establishes a connection to your server.
2. Your server examines the socket information to find the IP address of the server at the other end of the connection.
3. Your server creates a special "pseudo-hostname" by reversing the order of the components of the IP address, then concatenating them with the domain name of the blacklist:

Example:
If the incoming IP address is 166.70.98.32 and the name of the blacklist is dnsbl.sorbs.net, then the resulting "pseudo-hostname" is 32.98.70.166.dnsbl.sorbs.net

4. Your server does a regular DNS lookup of the pseudo-hostname 32.98.70.166.dnsbl.sorbs.net. If it resolves, then the owner of the blacklist considers the host 166.70.98.32 to be a spammer. If the lookup doesn't resolve, then the IP address isn't blacklisted.

Step 1: Choose Which Blacklist(s) to Use

At the time of this update (9/10/2009), my own sendmail configuration uses only the following blacklist:

* zen.spamhaus.org

I offer it only as a starting point. Jeff Makey's Blacklists Compared lists many other DNS blacklists. You will probably want to check Google or Yahoo to find out what others think of particular blacklists.

Add new lists very carefully! Some are overly aggressive in their blacklisting criteria: they might block mail from some of your best friends! Others list not only single IP addresses, but an entire network block that includes the offending IP address. (I generally try to avoid those blacklists ...)

After adding a new blacklist to your configuration, check the mail log, searching for "check_". Make sure that the blacklist isn't too agressive.

The location of mail logs varies from OS to OS. Try looking in /var/log/maillog or /var/log/messages for starters..

Step 2: Modify sendmail.cf for Blacklisting

Modern sendmail.cf files are generated from macro configuration (mc) files.

If you use a standard sendmail.org distribution, follow the instructions in the cf/README file that ships with sendmail. The procedure for generating the sendmail.cf on FreeBSD is described in a FreeBSD sendmail FAQ. Check the documentation for your OS if neither of these applies.

To add a DNS Blacklist to the macro configuration file, add a dnsbl FEATURE line for each blacklist (in the section of the mc file that has other FEATURE lines):


FEATURE(`dnsbl',`name-of-blacklist')dnl

Note that the single quotation marks are not all the same: the first quotation mark in each pair is a backquote; the second is an apostrophe.

The following illustrates how to add the blacklist zen.spamhaus.org to your sendmail mc file.

Example:


FEATURE(`dnsbl',`zen.spamhaus.org')dnl

Optional: Enable DNSBL for Designated Users Only

There are a variety of reasons to enable DNS Blacklisting for only a subset of your users: You might be skeptical of the DNSBL feature; you might want to charge extra for blacklisting; perhaps most of your users like receiving SPAM. Whatever the reason, you can configure sendmail to enable DNS Blacklisting for designated recipients only. Here is how to do it.

1. Insert a special delay_checks feature line in your sendmail mc file--somewhere after the feature that enables the access database:


FEATURE(`dnsbl',`zen.spamhaus.org')dnl
FEATURE(`access_db')dnl
FEATURE(`delay_checks',`hater')dnl

2. Add a line to the access db for each local recipient that wants DNS Blacklisting:


Spam:legolas@whipple.org HATER
Spam:bilbo@whipple.org HATER

3. Regenerate the sendmail.cf and access.db

The following commands might generate access.db on your platform:


# cd /etc/mail
# makemap hash access < access

Alternate Disabling Option: Exempt Specified Local Users from Blacklisting

In the option described above, we enabled DNS Blacklisting for a list of enumerated local recipient e-mail addresses. You might want to (instead) enable DNS Blacklisting for all local users except for a handful of local users who want to receive all incoming mail. (Maybe they are plastic surgeons that specialize in the enhancement of gender-specific body parts, for example, and don't want to miss the latest developments in their field.:-) If you are in this situation, you can configure sendmail to exempt specified local users from DNS Blacklisting. Here is how to do it.

1. Insert a special (but different from the above) delay_checks feature line in your sendmail mc file--somewhere after the feature that enables the access database:


FEATURE(`dnsbl',`zen.spamhaus.org')dnl
FEATURE(`access_db')dnl
FEATURE(`delay_checks',`friend')dnl

2. Add a line to the access db for each local recipient who does not want DNS Blacklisting:


Spam:legolas@whipple.org FRIEND
Spam:bilbo@whipple.org FRIEND

3. Regenerate the sendmail.cf and access.db

Note: The "hater" and "friend" [of spam] arguments to the delay_checks feature are mutually exclusive--you must choose one or the other (if you choose to use the feature at all). For more information, see my document that gives steps for configuring per user spam control, or the cf/README or the Bat Book.

Optional: Customize the Error Message

If you aren't satisfied with the default error message sendmail returns, you may specify one that you like better as a a third argument to the FEATURE macro. Here are two examples:


FEATURE(`dnsbl',`relays.ordb.org', `"550 5.7.1 Access denied(O): Unsolicited e-mail from " $&{client_addr} " refused. Request access at http://www.whipple.org/mailrequest.html"')dnl
FEATURE(`dnsbl',`dnsbl.sorbs.net',`"554 Rejected " $&{client_addr} " found in dnsbl.sorbs.net"')dnl

Optional: Change Default Behavior When Lookups Fail Temporarily

There will undoubtedly be occasions when attempts to look up the pseudo-hostname timeout or name resolution attempts otherwise fail. Sendmail's default behavior in this case is not to reject the incoming mail. If you would prefer to return a temporary error to the incoming server, you can do so by specifying a fourth argument, the value `t', to the FEATURE macro:


FEATURE(`dnsbl',`dnsbl.sorbs.net', , `t')dnl

If you also customize the error message, the above will look something like:


FEATURE(`dnsbl',`dnsbl.sorbs.net',`"554 Rejected " $&{client_addr} " found in dnsbl.sorbs.net"',`t')dnl

Now when the lookup fails, your sendmail will return the following error to the sending server:


451 Temporary lookup failure of IP at name-of-blacklist

Appendix A: Notes on Creating Your Own DNS Blacklist

If you want to set up your own blacklisting service and offer the service to others, all you need is a domain name (a third-level subdomain of your main second-level domain, for example), a name server (well, two--or more--name servers), and a way of updating your name servers to add or delete subdomains. Oh yes, you probably need lots of bandwidth and computing power, if you plan on being a public blacklist.

I could, for example, designate the subdomain spammers.whipple.org as the name of a DNS blacklist, then set up two authoritative name servers for spammers.whipple.org (or just use the whipple.org name servers, if they are powerful enough and have sufficient bandwidth). Whenever I want to add an IP address to the spammers.whipple.org blacklist, I add an "A" resource record for 7th-level domain consisting of the IP address (reversed) followed by spammers.whipple.org.

Then when other e-mail servers want to check my blacklist, they construct a 7th-level pseudo-hostname that ends with spammers.whipple.org and see if it resolves.

What should be on the right-hand-side of the "A" records that our blacklisting name server returns? For testing purposes (I guess?), most blacklists include a localhost-like sample entry (such as 127.0.0.2 or 127.0.0.3). Let's try looking up what kinds of "A" records the dnsbl.sorbs.net list returns when we chedk 127.0.0.2, using the dig command:


% dig 2.0.0.127.dnsbl.sorbs.net

The day I tried this, dig returned the following:


; <<>> DiG 8.3 <<>> 2.0.0.127.dnsbl.sorbs.net
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49598
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 7, ADDITIONAL:
7
;; QUERY SECTION:
;; 2.0.0.127.dnsbl.sorbs.net, type = A, class = IN

;; ANSWER SECTION:
2.0.0.127.dnsbl.sorbs.net. 2D IN A 127.0.0.6
2.0.0.127.dnsbl.sorbs.net. 2D IN A 127.0.0.7
2.0.0.127.dnsbl.sorbs.net. 2D IN A 127.0.0.9
2.0.0.127.dnsbl.sorbs.net. 2D IN A 127.0.0.10
2.0.0.127.dnsbl.sorbs.net. 2D IN A 127.0.0.2
2.0.0.127.dnsbl.sorbs.net. 2D IN A 127.0.0.5
2.0.0.127.dnsbl.sorbs.net. 2D IN A 127.0.0.3
2.0.0.127.dnsbl.sorbs.net. 2D IN A 127.0.0.4

;; AUTHORITY SECTION:
dnsbl.sorbs.net. 1d18h34m20s IN NS rbl1.oregonstate.edu.
dnsbl.sorbs.net. 1d18h34m20s IN NS rbl2.oregonstate.edu.
dnsbl.sorbs.net. 1d18h34m20s IN NS sorbs.bl.xs4all.nl.
dnsbl.sorbs.net. 1d18h34m20s IN NS sorbs-sql1.vix.com.
dnsbl.sorbs.net. 1d18h34m20s IN NS rbldns0.sorbs.net.
dnsbl.sorbs.net. 1d18h34m20s IN NS rbldns2.sorbs.net.
dnsbl.sorbs.net. 1d18h34m20s IN NS rbldns3.sorbs.net.

;; ADDITIONAL SECTION:
rbl1.oregonstate.edu. 18h54m31s IN A 128.193.0.30
rbl2.oregonstate.edu. 18h54m31s IN A 128.193.0.130
sorbs.bl.xs4all.nl. 18h54m19s IN A 194.109.9.11
sorbs-sql1.vix.com. 47m6s IN A 204.152.186.189
rbldns0.sorbs.net. 3m3s IN A 203.15.51.34
rbldns2.sorbs.net. 3m3s IN A 209.209.1.20
rbldns3.sorbs.net. 3m3s IN A 209.142.2.10

;; Total query time: 244 msec
;; FROM: gabriel.whipple.org to SERVER: 166.70.98.35
;; WHEN: Mon May 17 14:02:05 2004
;; MSG SIZE sent: 43 rcvd: 491

Notice in the "answer section" that the right-hand-side of the eight "A" resource records have values:

* 127.0.0.6
* 127.0.0.7
* 127.0.0.9
* 127.0.0.10
* 127.0.0.2
* 127.0.0.5
* 127.0.0.3
* 127.0.0.4

Using multiple resource records for a single (each with different right-hand-sides) lets you assign significance to the value that is returned on the right-hand-side. (Visit the blacklist's home page to find an explanation of the meaning of the possible right-hand-sides.)

Sendmail's dnsbl feature (described above), doesn't check the value of the right-hand-side. (All it cares about is whether something--anything--is returned.) If you want sendmail to check the right-hand-side value returned by DNS, see the enhdnsbl (Enhanced DNS Blacklist) feature below.

Because 2.0.0.127.dnsbl.sorbs.net resolves (returns at least one "A" record, above), we know that the test IP address 127.0.0.2 is blacklisted by dnsbl.sorbs.net.
Appendix B: Enhanced DNS BlackLists

Sendmail's enhanced DNS Blacklist feature (enhdnsbl) is identical to the dnsbl feature described earlier in this document, except that it allows you to specify an optional fifth argument. The fifth argument is for matching against the value returned in the right-hand-side of the blacklist's DNS resource record. (See the previous appendix for that discussion.)

Since all but the first argument are optional for both the dnsbl and enhdnsbl features, you could (if you prefer) substitute "enhdnsbl" for "dnsbl" in all the FEATURE statements in this document.

If (after visiting the dnsbl.sorbs.net web site and finding the meaning of the different values that can be returned on the right-hand-side of the blacklist's "A" resource records) you determine that you are interested in blocking e-mail only if the right-hand-side is 127.0.0.9 (for example), try one of the following:


FEATURE(`enhdnsbl',`dnsbl.sorbs.net', , , `127.0.0.9')dnl
FEATURE(`enhdnsbl',`dnsbl.sorbs.net',`"554 Rejected " $&{client_addr} " found in dnsbl.sorbs.net"', , `127.0.0.9')dnl
FEATURE(`enhdnsbl',`dnsbl.sorbs.net',`"554 Rejected " $&{client_addr} " found in dnsbl.sorbs.net"',`t', `127.0.0.9')dnl

Appendix C: Subscriber-Only Blacklists at mail-abuse.org

Not all DNS blacklists are open to queries from all servers on the Internet. Probably the most well known blacklists are those of mail-abuse.org: blackholes.mail-abuse.org, relays.mail-abuse.org and dialups.mail-abuse.org. Those three became "subscriber-only" lists in 2001. (When you subscribe to their service, they ask for the IP address of your mail server. Then they add your server's IP address to an access control list (ACL). See the discussion about caching only name servers below if you decide to subscribe to mail-abuse.org. If you are a non-profit hobbyist, you might qualify for a free subscription.)

Although mail-abuse.org's lists require a subscription, most other blacklists can be queried by anyone.
Appendix D: Name Server Issues

It might be to your advantage to run a caching name server on your mail server. (If you have a subscription from mail-abuse.org, it is probably a requirement to run a caching name server: mail-abuse.org will list your mail server's IP address in its access control list; your ISP's name servers [for example] won't be able to resolve the pseudo-hostnames in mail-abuse.org's blacklists.) If you run a caching name server, you should limit the hosts that can query your name server to your mail server (and probably backup MX servers).

Here is an example of a possible BIND 9 named.conf configuration file for a caching name server. Note that the allow-query line restricts who can query the name server.


/*
* A simple BIND 9 configuration
*/

options {
directory "/etc/namedb";
allow-query { 166.70.98.35; 166.70.98.36; };
};

zone "0.0.127.in-addr.arpa" {
type master;
file "localhost.rev";
};

zone "." {
type hint;
file "named.root";
};


Afterword

This document is a work in progress. Please send corrections or suggestions to me!

Protecting your phpinfo(), sort of.

SkyHi @ Monday, November 23, 2009
404 Not Found
//PHPMYADMIN/config/config.inc.php?p=phpinfo();: 1 Time(s)
//admin/config/config.inc.php?p=phpinfo();: 1 Time(s)
//dbadmin/config/config.inc.php?p=phpinfo();: 1 Time(s)
//myadmin/config/config.inc.php?p=phpinfo();: 1 Time(s)
//mysql/config/config.inc.php?p=phpinfo();: 1 Time(s)
//p/m/a/config/config.inc.php?p=phpinfo();: 1 Time(s)
//php-my-admin/config/config.inc.php?p=phpinfo();: 1 Time(s)
//phpMyAdmin/config/config.inc.php?p=phpinfo();: 1 Time(s)
//phpmyadmin/config/config.inc.php?p=phpinfo();: 1 Time(s)
//pma/config/config.inc.php?p=phpinfo();: 1 Time(s)


Server config: /etc/php.ini

disable_functions = "phpinfo"

Programmer:

The truly security-conscious developer should write their application with security in mind from the beginning. It will relieve headaches and help to plug potential security holes down the road.

To that end, when replacing Slackware with Ubuntu Server on my 64-bit server, I needed some information about the configuration of PHP 5 on Ubuntu, because it differed from my old Slackware box.

To add to my server configuration’s complexity, I use an FHS /srv layout, which stores my web sides on a separate RAID array from the system’s root partition. Since I will never use Apache 2’s “default” configuration for a site, I needed to use it to serve phpinfo(), but in a way that wouldn’t leave me open in the future if I somehow forgot to remove it.

I came up with this simple script that should work in most cases (nearly all if your development machine is on the same LAN or VPN as your server).

if ($_SERVER['REMOTE_ADDR'] == '192.168.2.89')
{
phpinfo();
} else {
echo "

You're like, not authorized to view this.

";
}
?>

Not something to be relied upon, but it’s a quick way to cover your butt if you forget to clean up after yourself.

Sunday, November 22, 2009

The Secret to Learning Anything

SkyHi @ Sunday, November 22, 2009
Some of the most important lessons I learned in college came from one professor, Michael Mitzenmacher. Now, this was a guy with a lot of papers to his name, tenured at Harvard, working on some pretty darn complicated computer science theory (I took his algorithms class). So you'd expect that I'd learn something important. But as it turned out, the biggest lessons I learned from him weren't on the topics he taught. I learned a secret that helped me learn much more effectively.

At one point when describing the problem sets in the class, he gave some advice that stuck with me:

Don't suffer from Math Major syndrome

So, what you ask, is Math Major syndrome? Well, think of it this way--if you are studying a hard problem, what's your first tendency? To get lost in thought? To start at it, mulling over ideas in your head, waiting for a flash of insight? If so, that's math major syndrome. It's the idea that you can (or have to) solve problems entirely through a brilliant flash of insight, without doing any actual work.





The alternative is quite simple:

Hard problems become easier by working through them with diagrams, effort and patience

In other words, don't just wait for answers to come to you--go out and find them. Don't just use what's in your head--use paper, or the computer, or a whiteboard, to draw out the ideas, try experiments, make the patterns visible rather than waiting for a flash of insight.

I remember really learning this lesson during the first problem set--I'd started it when it was handed out, but the night before it was due I still had one problem left. I spent hours on that problem, but I spent it drawing out equations, working through possiblities. Each one ended in failure, until I had a flash of insight in the middle of writing out an equation. If I hadn't worked through (and discarded) all those possibilities, I'd have had no hope of solving that problem.

So how do you apply this principle in practice?
Program Program Program
If you're learning to program, you should be programming. Work practice problems while you read tutorials--in fact, you're probably better off in most cases working practice problems than you are reading about programming. (You will need to read, but without practice, that reading means nothing.)

In fact, if you're just learning to program, then I suggest that you stop reading this article right now and go write some code (or get yourself set up with a compiler and then go write some code). But not something you know how to solve--try to write a program that is just a little bit scary or hard. You should feel just a bit scared that you won't be able to do it.
Draw it Out
A second corollary to the principle is that you're better off using paper than trying to imagine everything. In fact, I can't say enough about the importance of drawing diagrams. A lot of things in programming sound very abstract or difficult to describe in words, but they can be quickly shown with a picture. Computer memory, for example, can be thought of as a long block of cells (almost like in an Excel spreadsheet). Drawing out each piece of memory in your program (when dealing with pointers or data structures) can really help visualize what is going on. You can also write out recursion manually by showing each recursive call with its arguments and return value.

But even simpler concepts become clear when you draw them or work through them step by step--having trouble understanding a complex piece of code? Write out the different variables and how they are modified. For example, the boolean expression

! a || b

(in other words, NOT a OR b)

has a very specific meaning in English: "if a is true, then b must be true"; this could be useful for checking user inputs ("if the user pressed exit, then the game must be over--otherwise, print out an error"). But it's not at all obvious what ! a || b means unless you actually work through it. What do I mean?

The statement "if a is true, then b must be true" requires that:

if a = true, b must be true
if a = false, b can be either true or false

How does that work for the expression "! a || b? You can draw out the table by writing a in one column, b in a second, and whether the condition is satisfied result in a third:
a b ! a || b
0 0 1
0 1 1
1 1 1
1 0 0
Now you can more quickly see that the table expresses the English condition. The condition is satisfied in all the cases except when a is true and b is false (clearly, that can't satisfy an expression that says, "if as is true, then b is true"!)

Being able to look at the data really helps!