Blacklist via Request Method
[ #1 ] This first blacklisting method evaluates the client’s request method. Every time a client attempts to connect to your server, it sends a message indicating the type of connection it wishes to make. There are many different types of request methods recognized by Apache. The two most common methods are GET and POST requests, which are required for “getting” and “posting” data to and from the server. In most cases, these are the only request methods required to operate a dynamic website. Allowing more request methods than are necessary increases your site’s vulnerability. Thus, to restrict the types of request methods available to clients, we use this block of Apache directives:
RewriteEngine On
ServerSignature Off
Options +FollowSymLinks
RewriteCond %{REQUEST_METHOD} ^(delete|head|trace|track) [NC]
RewriteRule ^(.*)$ - [F,L]
The key to this rewrite method is the REQUEST_METHOD in the rewrite condition. First we invoke some precautionary security measures, and then we evaluate the request method against our list of prohibited types. Apache will then compare each client request method against the blacklisted expressions and subsequently deny access to any forbidden requests. Here we are blocking delete and head because they are unecessary, and also blocking trace and track because they violate the same-origin rules for clients. Of course, I encourage you to do your own research and establish your own request-method security policy.
Blacklist via the Request
[ #2 ] The next blacklisting method is based on the client’s request. When a client attempts to connect to the server, it sends a full HTTP request string that specifies the request method, request URI, and transfer-protocol version. Note that additional headers sent by the browser are not included in the request string. Here is a typical example:
GET blog/index.html HTTP/1.1
This long request string may be checked against a list of prohibited characters to protect against malicious requests and other exploitative behavior. Here is an example of sanitizing client requests by way of Apache’s THE_REQUEST variable:
RewriteEngine On
RewriteCond %{THE_REQUEST} ^.*(\\r|\\n|%0A|%0D).* [NC]
RewriteRule ^(.*)$ - [F,L]
Here we are evaluating the entire client-request string against a list of prohibited entities. While there are many character strings common to malicious requests, this example focuses on the prevention of HTTP response splitting, cross-site scripting attacks, cache poisoning, and similar dual-header exploits. Although these are some of the most common types of attacks, there are many others. I encourage you to check your server logs, do some research, and sanitize accordingly.
Blacklist via the Referrer
[ #3 ] Blacklisting via the HTTP referrer is an excellent way to block referrer spam, defend against penetration tests, and protect against other malicious activity. The HTTP referrer is identified as the source of an incoming link to a web page. For example, if a visitor arrives at your site through a link they found in the Google search results, the referrer would be the Google page from whence the visitor came. Sounds straightforward, and it is.
Unfortantely, one of the biggest spam problems on the Web involves the abuse of HTTP referrer data. In order to improve search-engine rank, spambots will repeatedly visit your site using their spam domain as the referrer. The referrer is generally faked, and the bots frequently visit via HEAD requests for the sake of efficiency. If the target site publicizes their access logs, the spam sites will receive a search-engine boost from links in the referrer statistics.
Fortunately, by taking advantage of mod_rewrite’s HTTP_REFERER variable, we can forge a powerful, customized referrer blacklist. Here’s our example:
RewriteEngine On
RewriteCond %{HTTP_REFERER} ^(.*)(<|>|'|%0A|%0D|%27|%3C|%3E|).* [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?adult(-|.).*$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?poker(-|.).*$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?.*(-|.)?drugs(-|.).*$ [NC]
RewriteRule ^(.*)$ - [F,L]
Same basic pattern as before: check for the availability of the rewrite module, enable the rewrite engine, and then specify the prohibited character strings using the HTTP_REFERER variable and as many rewrite conditions as necessary. In this case, we are blocking a series of potentially malicious characters in the first condition, and then blacklisting any referrer containing the terms “adult”, “poker”, or “drugs”. Of course, we may blacklist as many referrer strings as needed by simply emulating the exisiting rewrite conditions. Just don’t get carried away — I have seen some referrer blacklists that are over 4000 lines long!
Blacklist via Cookies
[ #4 ] Protecting your site against malicious cookie exploits is greatly facilitated by using Apache’s HTTP_COOKIE variable. HTTP cookies are chunks of data sent by the server to the web client upon initialization. The browser then sends the cookie information back to the server for each subsequent visit. This enables the server to authenticate users, track sessions, and store preferences. A common example of the type of functionality enabled by cookies is the shopping cart. Information about the items placed in a user’s shopping cart may be stored in a cookie, thereby enabling server scripts to respond accordingly.
Generally, a cookie consists of a unique string of alphanumeric text and persists for the duration of a user’s session. Apache’s mod_cookie module generates cookie values randomly and upon request. Once a cookie has been set, it may be used as a database key for further processing, behavior logging, session tracking, and much more. Unfortunately, this useful technology may be abused by attackers to penetrate and infiltrate your server’s defenses. Cookie-based protocols are vulnerable to a variety of exploits, including cookie poisoning, cross-site scripting, and cross-site cooking. By adding malicious characters, scripts, and other content to cookies, attackers may find and exploit sensitive vulnerabilities.
The good news is that we may defend against most of this nonsense by using Apache’s HTTP_COOKIE variable to blacklist characters known to be associated with malicious cookie exploits. Here is an example that does the job:
RewriteEngine On
RewriteCond %{HTTP_COOKIE} ^.*(<|>|'|%0A|%0D|%27|%3C|%3E|).* [NC]
RewriteRule ^(.*)$ - [F,L]
This is as straightforward as it looks. Check for the required rewrite module, enable the rewrite engine, and deny requests for any HTTP_COOKIEs containing the specified list of prohibited characters. In this list you will see characters generally required to execute any sort of scripted attack: opening and closing angle brackets, single quotation marks, and a variety of hexadecimal equivalents. Feel free to expand this list with additional characters as you see fit. As always, recommendations are most welcome.
Blacklist via Request URI
[ #5 ] Use of Apache’s REQUEST_URI variable is frequently seen in conjunction with URL canonicalization. The REQUEST_URI variable targets the requested resource specified in the full HTTP request string. Thus, we may use Apache’s THE_REQUEST variable to target the entire request string (as discussed above), while using the REQUEST_URI variable to target the actual request URI. For example, the REQUEST_URI variable refers to the “blog/index.html” portion of the following, full HTTP request line:
GET blog/index.html HTTP/1.1
For canonicalization purposes, this is exactly the type of information that must be focused on and manipulated in order to achieve precise, uniform URLs. Likewise, for blacklisting malicious request activity such as the kind of nonsense usually exposed in your server’s access and error logs, targeting, evaluating, and denying malicious URL requests is easily accomplished by taking advantage of Apache’s REQUEST_URI variable.
As you can imagine, blacklisting via REQUEST_URI is an excellent way to eliminate scores of malicious behavior. Here is an example that includes some of the same characters and strings that are blocked in the upcoming 4G Blacklist:
RewriteEngine On
RewriteCond %{REQUEST_URI} ^.*(,|;|:|<|>|">|"<|/|\\\.\.\\).* [NC,OR]
RewriteCond %{REQUEST_URI} ^.*(\=|\@|\[|\]|\^|\`|\{|\}|\~).* [NC,OR]
RewriteCond %{REQUEST_URI} ^.*(\'|%0A|%0D|%27|%3C|%3E|).* [NC]
RewriteRule ^(.*)$ - [F,L]
Again, same general pattern of directives as before, only this time we are specifying forbidden characters via the REQUEST_URI variable. Here we are denying any URL requests containing invalid characters, including different types of brackets, various punctuational characters, and some key hexadecimal equivalents. Of course, the possibilities are endless, and the blacklist should be customized according to your specific security strategy and unfolding blacklisting needs.
Blacklist via the User Agent
[ #6 ] Blacklisting via user-agent is a commonly seen strategy that yields questionable results. The concept of blacklisting user-agents revolves around the idea that every browser, bot, and spider that visits your server identifies itself with a specific user-agent character string. Thus, user-agents associated with malicious, unfriendly, or otherwise unwanted behavior may be identified and blacklisted in order to prevent against future access. This is a well-known blacklisting strategy that has resulted in some extensive and effective user-agent blacklists.
Of course, the downside to this method involves the fact that user-agent information is easily forged, making it difficult to know for certain the true identity of blacklisted clients. By simply changing their user-agent to an unknown identity, malicious bots may bypass every blacklist on the Internet. Many evil “scumbots” indeed do this very thing, which explains the incredibly vast number of blacklisted user-agents. Even so, there are certain limits to the extent to which certain user-agent strings may be changed. For example, GNU’s Wget and the cURL command-line tool are difficult to forge, and many other clients have hard-coded user-agent strings that are difficult to change.
On Apache servers, user-agents are easily identified and blacklisted via the HTTP_USER_AGENT variable. Here is an example:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(<|>|'|%0A|%0D|%27|%3C|%3E|).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(HTTrack|clshttp|archiver|loader|email|nikto|miner|python).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|libwww\-perl|curl|wget|harvest|scan|grab|extract).* [NC]
RewriteRule ^(.*)$ - [F,L]
This method works just like the others: check for the mod_rewrite module, enable the rewrite engine, and proceed to deny access to any user-agent that includes any of the blacklisted character strings in its name. As with our previous blacklisting techniques, here we are prohibiting angle brackets, single quotation marks, and various hexadecimal equivalents. Additionally, we include a handful of user-agent strings commonly associated with server attacks and other malicious behavior. We certainly don’t need anything associated with libwww-perl hitting our server, and many of the others are included in just about every user-agent blacklist that you can find. There are tons of other nasty user-agent scumbots out there, so feel free to beef things up with a few of your own.
Blacklist via the Query String
[ #7 ] Protecting your server against malicious query-string activity is extremely important. Whereas static URLs summon pages, their appended query strings transmit data and pass variables throughout the domain. Query-string information interacts with scripts and databases, influencing behavior and determining results. This relatively open channel of communication is easily accessible and prone to external manipulation. By altering data and inserting malicious code, attackers may penetrate and exploit your sever directly through the query string.
Fortunately, we can protect our server against malicious query-string exploits with the help of Apache’s invaluable QUERY_STRING variable. By taking advantage of this variable, we can ensure the legitimacy and quality of query-string input by screening out and denying access to a known collection of potentially harmful character strings. Here is an example that will keep our query strings squeaky clean:
RewriteEngine On
RewriteCond %{QUERY_STRING} ^.*(localhost|loopback|127\.0\.0\.1).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(\.|\*|;|<|>|'|"|\)|%0A|%0D|%22|%27|%3C|%3E|).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(md5|benchmark|union|select|insert|cast|set|declare|drop|update).* [NC]
RewriteRule ^(.*)$ - [F,L]
As you can see, here we are using the QUERY_STRING variable to check all query-string input against a list of prohibited alphanumeric characters strings. This strategy will deny access to any URL-request that includes a query-string containing localhost references, invalid punctuation, hexadecimal equivalents, and various SQL commands. Blacklisting these enitities protects us from common cross-site scripting (XSS), remote shell attacks, and SQL injection. And, while this a good start, it pales in comparison to the new query-string directives of the upcoming 4G Blacklist. ;)
Blacklist via IP Address
[ #8 ] Last but certainly not least, we can blacklist according to IP address. Blacklisting sites based on IP is probably the oldest method in the book and works great for denying site access to stalkers, scrapers, spammers, trolls, and many other types of troublesome morons. The catch is that the method only works when the perpetrators are coming from the same location. An easy way to bypass any IP blacklist is to simply use a different ISP or visit via proxy server. Even so, there is no lack of mindless creeps out there roaming the Internet, who sit there, using the same machine, day after day, relentlessly harassing innocent websites. For these types of lazy, no-life losers, blacklisting via IP address is the perfect solution. Here is a hypothetical example demonstrating several ways to blacklist IPs:
# block individual IPs
RewriteEngine On
RewriteCond %{REMOTE_ADDR} ^123\.456\.789\.1$ [OR]
RewriteCond %{REMOTE_ADDR} ^456\.789\.123\.2$ [OR]
RewriteCond %{REMOTE_ADDR} ^789\.123\.456\.3$ [OR]
RewriteRule ^(.*)$ - [F,L]
# block ranges of IPs
RewriteEngine On
RewriteCond %{REMOTE_ADDR} ^123\.$ [OR]
RewriteCond %{REMOTE_ADDR} ^456\.789\.$ [OR]
RewriteCond %{REMOTE_ADDR} ^789\.123\.456\.$ [OR]
RewriteRule ^(.*)$ - [F,L]
# alt block IP method
order allow,deny
allow from all
deny from 123.
deny from 123.456.
deny from 123.456.789.0
In the first block, we are blacklisting three specific IP addresses using Apache’s mod_rewrite and its associated REMOTE_ADDR variable. Each of the hypothetical IPs listed represent a specific, individual address. Then, in the next code block, we are blocking three different ranges of IPs by omitting numerical data from the targeted IP string. In the first line we target any IP beginning with “123.”, which is an enormous number of addresses. In the second line, we block a different, more restrictive range by including the second portion of the address. Finally, in the third line, we block a different, much smaller range of IPs by including a third portion of the address.
Then, just for kicks, I threw in an alternate method of blocking IPs. This is an equally effective method that enables you to block IP addresses and ranges as specifically as necessary. Each deny line pattern-matches according to the specified IP string.
Dealing with Blacklisted Visitors
In each of these eight blacklisting techniques, we respond to all blacklisted visitors with the server’s default “403 Forbidden” error. This page serves its purpose and requires very little to deliver in terms of system resources, however there is much more that you can do with blacklisted traffic. Here are a few ideas:
Redirect to home page
More subtle than the 403 error, this redirect strategy routes blocked traffic directly to the home page. To use, replace the RewriteRule directive (i.e., the entire line) with the following code:
RewriteRule ^(.*)$ http://your-domain.tld/ [F,L]
Redirect to external site
The possibilities here are endless. Just make sure you think twice about the destination, as any scum that you redirect to another site will be seen as coming from your own. Even so, here is the code that you would use to replace the RewriteRule directive in any of the examples above:
RewriteRule ^(.*)$ http://external-domain.tld/some-target/page.html [F,L]
Redirect them back to their own site
This is one of my favorites. It’s like having a magic shield that reflects attacks back at the attacker. Send a clear message by using this code as the RewriteRule directive in any of our blacklisting methods:
RewriteRule ^(.*)$ http://%{REMOTE_ADDR}/ [F,L]
Custom processing
For those of you with a little skill, it is possible to redirect your unwelcome guests to a fail-safe page that explains the situation to the client while logging all of the information behind the scenes. This is perhaps the most useful approach for understanding your traffic and developing an optimal security strategy. The code would look something like this, depending on your file name and its location:
RewriteRule ^(.*)$ /home/path/blacklisting-script.php [F,L]
Closure
This article presents eight effective techniques for protecting your server and preventing malicious behavior. While each of these methods may be used individually, they are designed to secure different aspects of your environment and thus provide a more complete type of firewall protection when combined into a synergized whole. Even when combining these techniques, however, keep in mind that blacklisting various protocols serves to complement a more robust and comprehensive security strategy. Once understood, these methods provide the average webmaster an easy, effective way of defending against unwanted behavior and enhancing the overall security of their sites.
[ Gravatar Icon ]
rc phelps – #18
hi jeff,
is the following httpd log file entry indicative of mischievous behavior:
[Thu Apr 09 08:22:24 2009] [error] [client 204.234.223.2] Request exceeded the limit of 10 internal redirects due to probable configuration error. Use 'LimitInternalRecursion' to increase the limit if necessary. Use 'LogLevel debug' to get a backtrace.
Curous that the state of nebraska would be banging on my lowly webserver.
thanks for any insight.
rc phelps
[ Gravatar Icon ]
Jeff Starr – #19
@rc phelps: That error message is telling you that the URL request resulted in too many redirects, as specified by your server. This means that something requested a web page that was redirected by your server. That redirect was then redirected to another resource, and then that redirect was redirected, and so on until the maximum number was reached. This is probably due to some unruly HTAccesss directives or something script-related. I.e., most likely not the result of mischievous behavior.
[ Gravatar Icon ]
kg – #20
I also had trouble with the first REQUEST_URI line
RewriteCond %{REQUEST_URI} ^.*(,|;|:|<|>|">|"<|/|\\\.\.\\).* [NC,OR]
I had to remove the forward slash from the list, like so
RewriteCond %{REQUEST_URI} ^.*(,|;|:|<|>|">|"<|\\\.\.\\).* [NC,OR]
[ Gravatar Icon ]
Jeff Starr – #21
@kg: Thanks for the information. Escaping the character may also have worked ( \/ ):
RewriteCond %{REQUEST_URI} ^.*(,|;|:|<|>|">|"<|\/|\\\.\.\\).* [NC,OR]
Although I haven’t tested it..
[ Gravatar Icon ]
Vladimir – #22
Here’s another rule set that blocks many HTTP-scanners, maybe someone will find it useful:
RewriteEngine On
RewriteCond %{QUERY_STRING} [^?]*\? [OR]
RewriteCond %{QUERY_STRING} (\.\./|\.\.\\) [OR]
RewriteCond %{QUERY_STRING} (///) [OR]
RewriteCond %{THE_REQUEST} "^(GET|POST) /?https?:" [OR]
RewriteCond %{THE_REQUEST} "^(GET|POST|HEAD) //"
RewriteRule (.*) $1 [F]
The first RewriteCond checks if the query string has more than one question mark (this pattern is used in some attacks; moreover, extra question marks should be encoded tp %3F), the second one tries to prevent directory traversal attacks (for both Windows and Linux hosts), the third one disallows three or more slashes in the query string (common pattern in many attacks), the fourth and the fifth ones stops proxy checkers.
[ Gravatar Icon ]
Jeff Starr – #23
Another excellent post, Vladimir — thanks for sharing with us. This is a great set of HTAccess security directives, some of which are already included in my 4G Blacklist in the “Query String Exploits” section. I like the check for double question marks, and the proxy-checking directives are just plain sexy. A couple of questions for you:
1. What are your thoughts on simply blocking any instances of two periods (i.e., \.\. )?
2. What exactly is going on in the RewriteRule? Seems like a possible typo?
Thanks again for the comment :)
[ Gravatar Icon ]
Vladimir – #24
What are your thoughts on simply blocking any instances of two periods
Well, maybe… This depends upon what comes in GET and what in POST. For example, when you search for something in WordPress, the string is passed in GET request. A visitor could make a typo and by accident put two periods and it would not be user-friendly to show a 403 page.
And, if you try a directory traversal attack, you still need to use either forward or backward slash — cf. ..etc/password and ../etc/password. So it looks like a slash is a must in this type of attacks.
What exactly is going on in the RewriteRule? Seems like a possible typo?
RewriteRule (.*) $1 [F]
That is, anything that matches RewriteCond’s gets banned ([F])
Well, maybe
RewriteRule .* - [F]
is better, but both worked for me.
Would you mind if I scan your site with Nessus and Nikto? This can give you more attack patterns. I will launch the scanner from 195.10.218.132, please do not ban me :-)
[ Gravatar Icon ]
Jeff Starr – #25
I am on the fence about blocking any instance of two simultaneous periods in the query string. Then the presence of a forward-vs-backslash along with the request is also an interesting dilemma. Currently, I only block the case when \.\.\/ is present in the query string, but I am thinking that blocking the backslash case is also a good idea. I almost blocked backslashes (either one or two) via mod_alias as well. I just can’t imagine why they would be needed unless encoded. Anyway, food for thought.
I assumed that your RewriteRule was blocking any matching requests, but I had never seen that particular flavour before, so I thought I would ask. Very interesting.
Go ahead and scan my site using the specified IP. I am currently running a series of tests myself, so it will be interesting to see the results given the current conditions. I won’t ban you ;)
[ Gravatar Icon ]
Rima – #26
Hi
I have Wordpress blog and I’d like to hide the login URL totally.. or to be honest I do want visitors to know that I’m using wordpress because it’s not the only thing running the website.
I have tried a wordpress plug-in that generates this for example
RewriteRule ^logout wp-login.php?action=logout&_wpnonce=b4318ad0cd&stealth_out_key=cbmfojbhqsdaxjem1q0jyzyivq [L]
well that’s nice, but when you go to /login it redirects and shows in the address bar wp-login?action=……. etc.
I don’t want that to happen, I want the whole thing to be totally masked.
Any advices ?
[ Gravatar Icon ]
Jeff Starr – #27
Hi Rima, the URL of any webpage will always be available to the visitor. There is no way (of which I am aware) to completely hide it. Fascinating that you actually would want to deprive users of that information.
Reference: http://perishablepress.com/press/2009/02/03/eight-ways-to-blacklist-with-apaches-mod_rewrite/
