At last! After many months of collecting data, crafting directives, and testing results, I am thrilled to announce the release of the 4G Blacklist! The 4G Blacklist is a next-generation protective firewall that secures your website against a wide range of malicious activity. Like its 3G predecessor, the 4G Blacklist is designed for use on Apache servers and is easily implemented via HTAccess or the httpd.conf configuration file. In order to function properly, the 4G Blacklist requires two specific Apache modules, mod_rewrite and mod_alias. As with the third generation of the blacklist, the 4G Blacklist consists of multiple parts:

* HTAccess Essentials
* Request-Method Filtering
* IP Address Blacklist
* Query-String Blacklist
* URL Blacklist

Each of these methods is designed to protect different aspects of your site. They may be used independently, mixed and matched, or combined to create the complete 4G Blacklist. This modularity provides flexibility for different implementations while facilitating the testing and updating process. The core of the 4G Blacklist consists of the last two methods, the Query-String and URL Blacklists. These two sections provide an enormous amount of protection against many potentially devastating attacks. Everything else is just icing on the cake. Speaking of which, there are also two more completely optional sections of the 4G Blacklist, namely:

* The Ultimate User-Agent Blacklist
* The Ultimate Referrer Blacklist

These two sections have been removed from the 4G Blacklist and relegated to “optional” status because they are no longer necessary. Put simply, the 4G Blacklist provides better protection with fewer lines of code. Even so, each of these blacklists have been updated with hundreds of new directives and will be made available here at Perishable Press in the near future. But for now, let’s return to the business at hand..
Presenting the Perishable Press 4G Blacklist

As is custom here at Perishable Press, I present the complete code first, and then walk through the usage instructions and code explanations. So, without furhter ado, here is the much-anticipated 4G Blacklist [for personal use only - may not be posted elsewhere without proper link attribution]:

### PERISHABLE PRESS 4G BLACKLIST ###

# ESSENTIALS
RewriteEngine on
ServerSignature Off
Options All -Indexes
Options +FollowSymLinks

# FILTER REQUEST METHODS

RewriteCond %{REQUEST_METHOD} ^(TRACE|DELETE|TRACK) [NC]
RewriteRule ^(.*)$ - [F,L]


# BLACKLIST CANDIDATES

Order Allow,Deny
Allow from all
Deny from 75.126.85.215 "# blacklist candidate 2008-01-02 = admin-ajax.php attack "
Deny from 128.111.48.138 "# blacklist candidate 2008-02-10 = cryptic character strings "
Deny from 87.248.163.54 "# blacklist candidate 2008-03-09 = block administrative attacks "
Deny from 84.122.143.99 "# blacklist candidate 2008-04-27 = block clam store loser "
Deny from 210.210.119.145 "# blacklist candidate 2008-05-31 = block _vpi.xml attacks "
Deny from 66.74.199.125 "# blacklist candidate 2008-10-19 = block mindless spider running "
Deny from 203.55.231.100 "# 1048 attacks in 60 minutes"
Deny from 24.19.202.10 "# 1629 attacks in 90 minutes"


# QUERY STRING EXPLOITS

RewriteCond %{QUERY_STRING} \.\.\/ [NC,OR]
RewriteCond %{QUERY_STRING} boot\.ini [NC,OR]
RewriteCond %{QUERY_STRING} tag\= [NC,OR]
RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
RewriteCond %{QUERY_STRING} http\: [NC,OR]
RewriteCond %{QUERY_STRING} https\: [NC,OR]
RewriteCond %{QUERY_STRING} mosConfig [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)|<|>|'|"|;|\?|\*).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(%22|%27|%3C|%3E|%5C|%7B|%7C).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(%0|%A|%B|%C|%D|%E|%F|127\.0).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(globals|encode|config|localhost|loopback).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]
RewriteRule ^(.*)$ - [F,L]


# CHARACTER STRINGS

# BASIC CHARACTERS
RedirectMatch 403 \,
RedirectMatch 403 \:
RedirectMatch 403 \;
RedirectMatch 403 \=
RedirectMatch 403 \@
RedirectMatch 403 \[
RedirectMatch 403 \]
RedirectMatch 403 \^
RedirectMatch 403 \`
RedirectMatch 403 \{
RedirectMatch 403 \}
RedirectMatch 403 \~
RedirectMatch 403 \"
RedirectMatch 403 \$
RedirectMatch 403 \<
RedirectMatch 403 \>
RedirectMatch 403 \|
RedirectMatch 403 \.\.
RedirectMatch 403 \/\/
RedirectMatch 403 \%0
RedirectMatch 403 \%A
RedirectMatch 403 \%B
RedirectMatch 403 \%C
RedirectMatch 403 \%D
RedirectMatch 403 \%E
RedirectMatch 403 \%F
RedirectMatch 403 \%22
RedirectMatch 403 \%27
RedirectMatch 403 \%28
RedirectMatch 403 \%29
RedirectMatch 403 \%3C
RedirectMatch 403 \%3E
RedirectMatch 403 \%3F
RedirectMatch 403 \%5B
RedirectMatch 403 \%5C
RedirectMatch 403 \%5D
RedirectMatch 403 \%7B
RedirectMatch 403 \%7C
RedirectMatch 403 \%7D
# COMMON PATTERNS
Redirectmatch 403 \_vpi
RedirectMatch 403 \.inc
Redirectmatch 403 xAou6
Redirectmatch 403 db\_name
Redirectmatch 403 select\(
Redirectmatch 403 convert\(
Redirectmatch 403 \/query\/
RedirectMatch 403 ImpEvData
Redirectmatch 403 \.XMLHTTP
Redirectmatch 403 proxydeny
RedirectMatch 403 function\.
Redirectmatch 403 remoteFile
Redirectmatch 403 servername
Redirectmatch 403 \&rptmode\=
Redirectmatch 403 sys\_cpanel
RedirectMatch 403 db\_connect
RedirectMatch 403 doeditconfig
RedirectMatch 403 check\_proxy
Redirectmatch 403 system\_user
Redirectmatch 403 \/\(null\)\/
Redirectmatch 403 clientrequest
Redirectmatch 403 option\_value
RedirectMatch 403 ref\.outcontrol
# SPECIFIC EXPLOITS
RedirectMatch 403 errors\.
RedirectMatch 403 config\.
RedirectMatch 403 include\.
RedirectMatch 403 display\.
RedirectMatch 403 register\.
Redirectmatch 403 password\.
RedirectMatch 403 maincore\.
RedirectMatch 403 authorize\.
Redirectmatch 403 macromates\.
RedirectMatch 403 head\_auth\.
RedirectMatch 403 submit\_links\.
RedirectMatch 403 change\_action\.
Redirectmatch 403 com\_facileforms\/
RedirectMatch 403 admin\_db\_utilities\.
RedirectMatch 403 admin\.webring\.docs\.
Redirectmatch 403 Table\/Latest\/index\.


That’s the juice right there. This 4G Blacklist is some powerful stuff, blocking and filtering a wide range of potential attacks and eliminating tons of malicious nonsense. Much care has been taken to beta test this firewall on multiple configurations running various types of software, however, due to my limited financial resources, it is impossible to test the 4G as comprehensively as I would have preferred. Even so, for the average site running typical software, everything should continue to work perfectly. With that in mind, please read through the remainder of the article before implementing the 4G Blacklist.
Installation and Usage

Before implementing the 4G Blacklist, ensure that you are equipped with the following system requirements:

* Linux server running Apache
* Enabled Apache module: mod_alias
* Enabled Apache module: mod_rewrite
* Ability to edit your site”s root htaccess file (or)
* Ability to modify Apache’s server configuration file

With these requirements met, copy and paste the entire 4G Blacklist into either the root HTAccess file or the server configuration file ( httpd.conf ). After uploading, visit your site and check proper loading of as many different types of pages as possible. For example, if you are running a blogging platform (such as WordPress), test different page views (single, archive, category, home, etc.), log into and surf the admin pages (plugins, themes, options, posts, etc.), and also check peripheral elements such as individual images, available downloads, and alternate protocols (FTP, HTTPS, etc.).

While the 4G Blacklist is designed to target only the bad guys, the regular expressions used in the list may interfere with legitimate URL or file access. If the directives in the blacklist are blocking a specific URL, the browsing device will display a 403 Forbidden error; similarily, if the blacklist happens to block a file or resource required for some script to function properly, the script (JavaScript, PHP, etc.) may simply stop working. If you experience either of these scenarios after installing the blacklist, don’t panic! Simply check the blocked URL or file, locate the matching blacklist string, and disable the directive by placing a pound sign ( # ) at the beginning of the associated line. Once the correct line is commented out, the blocked URL should load normally. Also, if you do happen to experience any conflicts involving the 4G Blacklist, please leave a comment or contact me directly.
Set for Stun

As my readers know, I am serious about site security. Nothing gets my juices flowing like the thought of chopping up mindless cracker whores into small, square chunks and feeding their still-twitching flesh to a pack of starving mongrels. That’s good times, but unfortunately there are probably laws against stuff like that. So in the meantime, we take steps to secure our sites using the most effective tools at our disposal. There is no one single magic bullet that will keep the unscrupulous bastards from exploiting and damaging your site, but there are many cumulative steps that may be taken to form a solid security strategy. Within this cumulative context, the 4G Blacklist recognizes and immunizes against a broad array of common attack elements, thereby maximizing resources while providing solid defense against malicious attacks.
Many Thanks

A huge “Thank You” to the dedicated people who helped beta test the 4G Blacklist. Your excellent feedback played an instrumental role in the development of this version. Thank you!
Further Reading

For more insight into the mysterious realms of blacklisting, the creation of the Perishable Press Blacklist, and DIY site security in general, check out some of my other articles:

* Eight Ways to Blacklist with Apache’s mod_rewrite
* Blacklist Candidate Series Summary
* How to Block Proxy Servers via htaccess
* 2G Blacklist: Closing the Door on Malicious Attacks
* Series Summary: Building the 3G Blacklist
* Building the Perishable Press 4G Blacklist

Next Up

Next up in the March 2009 Blacklist Series: The Ultimate User-Agent Blacklist. Don’t miss it!
Updates

Since the release of the 4G Blacklist, several users have discovered issues with the following 4G directives:

Joomla
In the query-string section, Joomla users should delete the following patterns:

request
config
[
]

In the character-string section, Joomla users should comment-out or delete the following lines:

RedirectMatch 403 \,
RedirectMatch 403 \;
RedirectMatch 403 config\.
RedirectMatch 403 register\.

WordPress
In the query-string section of the 4G Blacklist, the following changes have been made:

"%3D" character-string has been changed to "%5C"

Likewise, in the character-string section, the following change has been made:

"wp\_" character-string has been removed

And in the request-method filtering section, the following change has been made:

"HEAD" method has been removed

Also, the following changes may be necessary according to which plugins you have installed:

Ozh' Admin Drop Down Menu - remove "drop" from the query-string rules
WordPress' Akismet - remove "config" from the query-string rules

OpenID
OpenID users should read the information in this comment.

SMF
SMF users should read the information in this comment.

vBulletin
vBulletin users should read the information in these comments.
Copyright2009PerishablePress
Short URL:
78524 visits

.Post #677 categorized as Websites, last updated on Jul 10, 2009
Tagged with apache, blacklist, htaccess, mod_rewrite, security, tips
Share this article

* Bookmark at Delicious
* Tweet this!
* Stumble it
* Digg this post
* Add Perishable Press to Technorati
* Share on Facebook
* More..

Related articles

* Perishable Press 3G Blacklist
* Series Summary: Building the 3G Blacklist
* Building the Perishable Press 4G Blacklist
* Blacklist Candidate Series Summary
* Building the 3G Blacklist, Part 4: Improving the RedirectMatch Directives of the Original 2G Blacklist
* Building the 3G Blacklist, Part 2: Improving Site Security by Preventing Malicious Query-String Exploits
* Building the 3G Blacklist, Part 5: Improving Site Security by Selectively Blocking Individual IPs

153 Responses

Add a comment
[ Gravatar Icon ]

Karthik Viswanathan – #1

This is a comprehensive yet compact way to deal with malicious attacks. I’m truly impressed with the effort you’ve put into this. Unfortunately, this line poses a small problem to my website:

RedirectMatch 403 \/\/

How important would you consider blocking the // to be? A few WordPress plugins such as the WP-Clickmap require this string to function properly.
[ Gravatar Icon ]

Jeff Starr – #2

The list is cumulative in nature, meaning that each directive blocks a certain array of potential attacks and thus contributes a percentage of the blacklist’s overall effectiveness. Thus, the line may be commented out or removed entirely without significantly impacting the effectiveness of the remaining directives. Please let me know if you have any further questions about this.
[ Gravatar Icon ]

Awesome Blacklist – #3

This is an awesome blacklist.

Very effective :D
[ Gravatar Icon ]

Jonathan Ellse – #4

Works a treat. Very comprehensive customisation methods.

Thanks very much
[ Gravatar Icon ]

Arwin – #5

Great blacklist and clear story, fun to read!

I had one problem,
I have to remove ‘config’ from the ‘QUERY STRING EXPLOITS’ because otherwise my Mambo and Joomla site will not work properly.
Problem with the configuration of the configuration.php in the administrator backend of Mambo/Joomla (403 Redirect)
Still no other problems found :)

Thank you very much!
[ Gravatar Icon ]

John – #6

You really put a lot of good effort into building this 4G blacklist to share with everyone else. I, as with many other silent supporters, thank you for your efforts. Previously, some of my other sites that I am the head manager of got hacked from time to time, and I’ll be eagerly waiting to see if there are future hacking attempts after I’ve implemented this.

Again, thank you so much for making the world a bit better with your efforts.
[ Gravatar Icon ]

Greg – #7

Just tested, and for the moment have just to lines to be commented out for me.

One is for my downloads links which are unfortunately with some comma so:
#RedirectMatch 403 \,

And my registration page got the name “register” in the url so:
#RedirectMatch 403 register\.

Check few more things…
[ Gravatar Icon ]

Jeff Starr – #8

@Arwin: Thanks for the catch on the config query-string issue with Mambo/Joomla.

@John: It is my pleasure to help the community, and if a few spammers get hurt in the process, even better. ;)

@Greg: Thanks for the input! Are you using WordPress or some other platform? That information will help to improve the Blacklist.
[ Gravatar Icon ]

Greg – #9

Changed this one:
RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)||'|"|;|\?|\*).* [NC,OR]

to:
RewriteCond %{QUERY_STRING} ^.*(\[|\]|\(|\)||'|"|\?|\*).* [NC,OR]

Because of my pdf links on articles which are with semicolon

I’m under joomla 1.5.9
[ Gravatar Icon ]

Jeff Starr – #10

Duly noted, Greg — keep us posted on any other modifications used for Joomla.
[ Gravatar Icon ]

Greg – #11

Ok Jeff ;-)

Just posted your 4G on the french joomla forum, so we will have more feedback soon for joomla (hope it’s ok for you ?)
[ Gravatar Icon ]

Greg – #12

Commented this one for joomla:
#RedirectMatch 403 config\.

Because of the name of an icon of the administration:
/icon-48-config.png
[ Gravatar Icon ]

Sven – #13

Thanks a lot Jeff :-)

Are there any advantages when you use
“RedirectMatch 403″ and not “RewriteCond” ?
[ Gravatar Icon ]

Arwin – #14

Jeff, I also need to remove the [ and ] from the QUERY STRING EXPLOITS section, otherwise you can not set the plugin-options in Joomla(1.5.9).

Example url:
http://....option=com_plugins&view=plugin&client=site&task=edit&cid[]=1
[ Gravatar Icon ]

Arwin – #15

Oeps, that example url did not do what I want.. Without the http.. : index.php?option=com_plugins&view=plugin&client=site&task=edit&cid[]=1
[ Gravatar Icon ]

Jeff Starr – #16

@Greg: Thank you for your help with Joomla. I will be posting an update to the article with all of the updates and edits that people have made. I even caught one myself for WordPress and will add it to the list after a bit more testing.

@Sven: That’s debatable, but I use both in the 4G because I have found that each method seems to work best in different environments and with different purposes. mod_rewrite is much more flexible than RedirectMatch, but not as easy to work with in all situations. All of the RedirectMatch directives in the 4G could be written with mod_rewrite, but not vice-versa.
[ Gravatar Icon ]

Jeff Starr – #17

@Arwin: I will add them to the list of modifications and post an update later today. Thank you for your help with Joomla.
[ Gravatar Icon ]

Greg – #18

Same as Arwin said:

just done this:
RewriteCond %{QUERY_STRING} ^.*(\(|\)||'|"|\?|\*).* [NC,OR]
[ Gravatar Icon ]

Sven – #19

Thank for the reply Jeff.

I asked because I would like to direct my visitors to a “friendly page” if they are blocked by httaccess (I also log all blocked requests in a txt-file). This is only possible if I use a Rewrite, or…?
[ Gravatar Icon ]

Jeff Starr – #20

@Sven: Yes, you will need mod_rewrite for that..
[ Gravatar Icon ]

Sven – #21

Thanks Jeff :-)
[ Gravatar Icon ]

Greg – #22

Hi,

For Joomla 1.5.9

Found today that we need to remove the “request” from the query string exploits conditions.

Because some components use it with ajax script on the backend (com_xmap for example)

So this:
RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]

Become:
RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare|drop).* [NC]
[ Gravatar Icon ]

Andrew – #23

Great list, thanks for sharing it!

The only major issue I’ve had so far was with the Blacklist Candidates (IP Blocking) section. My apache (Apache 2.2.8) kept giving me “Order not allowed here”, “Allow not allowed here”, and “Deny not allowed here” no matter what I tried.

I ultimately found out that the 2.2 module’s (mod_authz_host, previously mod_access) directives are only allowed inside , , and contexts and .htaccess files: http://httpd.apache.org/docs/2.2/mod/mod_authz_host.html

After wrapping the entire block in a Directory context matching all possible directories (), apache gave me no more trouble.

Strange that I seem to be the only one having this problem.

Also, I initially had wrapped that section in an IfModule block, which I think is a good idea for any block that depends on a module.
[ Gravatar Icon ]

Greg – #24

@Andrew

Hi,

Got Apache 2.2.10 and don’t have your issue….

Where’s located your htaccess ? in your www or elsewhere on up directory ?
[ Gravatar Icon ]

Deb Phillips – #25

Jeff, I implemented your 4G Blacklist code about two days ago, and I’m amazed at the “calmness” in my visitor logs now! I no longer feel besieged by one unruly bot after another. What a relief! I just had to say “Thank you!” Your blacklist is wonderful, and it is so good of you to share it.

All the best,
Deb
[ Gravatar Icon ]

Tony – #26

Jeff,
I’ve been waiting forever for the 4G! I installed it on my first Wordpress blog for testing purposes. Here is what I noticed:

1. If you have the WP plugin “Ozh’ Admin Drop Down Menu” (latest version is 3.1.3.3.7 now), you’ll have to delete “drop” from the last line of the Query Strings Exploits:

RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]

Becomes:

RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare).* [NC]

2. If you use the TinyMCE Advanced plugin (latest version is 3.2 now), the buttons won’t be visible because the css won’t load, because it’s located in a directory wp_theme. Comment out the line that says: RedirectMatch 404 wp\_

And a question: Wordpress gives a 404 instead of Apache giving a 403, unless I create a custom 403 ErrorDocument. Any idea why?

Thanks Jeff, and I’ll be testing more in the next few days.
[ Gravatar Icon ]

Deb Phillips – #27

I can confirm that the default TinyMCE toolbar buttons also do not show up correctly. Only by removing the 4G code from the .htaccess file will the TinyMCE toolbar be displayed correctly.

I use the 404 Notifier plugin to alert me to any 404s occurring, and I’ve been getting a number 404s generated only when the 4G code is loaded, Unfortunately, I’m not savvy enough to pinpoint the exact cause of these errors, but almost all of the 404s occur at the point of WordPress’ cron check. I’m getting hundreds of 404s — daily — that are related to the cron check.

Another issue that I wonder if it’s related to the 4G blacklist is with one bot repeatedly coming in looking for the 403.shtml file, and a 404 is generated. I don’t *think* this was occurring before I began using the 4G code. Perhaps this is what Tony is referring to.

Even with these few issues, though, I really do appreciate all the work that went into the 4G blacklist. Hopefully, we’ll get the few kinks worked out.

Thanks very much.
[ Gravatar Icon ]

Tony – #28

Deb, you’re right. I could have sworn that the standard tinyMCE loaded fine when I tried, and I had already cleared my browser’s cache. Now I try again and the standard tinyMCE also doesn’t load okay.

The problem is this address that tinymce calls:
/wp-includes/js/tinymce/themes/advanced/skins/wp_theme/ui.css?ver=20081129

BTW, I used Firebug to pinpoint that. For those interested, Firebug is a Firefox Extension.
[ Gravatar Icon ]

Jeff Starr – #29

@Greg: Thank you — the “request” character string has been added to the list of query-string modifications for Joomla.

@Andrew: That is interesting indeed. I will look into it and update the 4G with the appropriate location and IfModule containers.

@Tony: Thanks for the help! I have removed the “wp_” directive from the list and updated the article with information about Ozh’ plugin.

@Deb: I have removed the directive that was preventing TinyMCE from loading. Thanks to Tony for his help with this.
[ Gravatar Icon ]

Tony – #30

Jeff,
It’s the least I can do!

wp-cron is making HEAD requests, so it’s being blocked. What do you think we should do, besides removing HEAD from the 4G’s Request Methods Filter?

From the raw access log:
"HEAD /wp-cron.php?check=561caf12167cb54c25589d71581df596 HTTP/1.0" 403 - "-" "WordPress/2.7.1"

I apologize for repeating the question I asked above, but maybe nobody noticed it:
Wordpress gives a 404 instead of Apache giving a 403, unless I create a custom 403 ErrorDocument. Any idea why?

Thanks!
[ Gravatar Icon ]

Deb Phillips – #31

Yes, I strongly second Tony’s inquiry regarding wp-cron. I’m getting deluged with 404 notifications related to it. These are email alerts I receive from the 404 Notifier plugin. They’re occurring about every one or two minutes.

Also, Tony, what issue were you running into with the Ozh’ Admin Drop Down Menu plugin? I only ask because I’m running the Ozh’ Better Feed plugin. I’m guessing there’s probably not any basically similar code between the two, but I wanted to ask. Thanks.
[ Gravatar Icon ]

Tony – #32

Deb,

Sorry, I should have been more clear about Ozh’ Admin Drop Down Menu. This plugin calls this address:

/wp-content/plugins/ozh-admin-drop-down-menu/inc/adminmenu.css.php?p=%2Fwp-content%2Fplugins%2Fozh-admin-drop-down-menu%2Finc&i=1&w=1&m=0&c=0&h=0&f=1&g=%23676768&n=0

Since there’s the “drop” word in the query string, it gets a 403 and the whole admin page gets messed up.

And some good news: googlebot accessed my site today and got a 200. That’s the one thing I was worried about the most!
[ Gravatar Icon ]

Deb Phillips – #33

Tony, thanks for the info on the Ozh’ plugin.

I’m glad Google is showing your site as 200.

As clarification, I’ve not encountered any 404 pages when actually visiting my website. The 404 notifications I’ve referred to are all happening behind the scenes. I can also see them showing up in the cPanel “Latest Visitors” module.

I’ve also received some 404s related to accessing files in the wp-content/uploads directory. But every time I’ve gone to the pages containing those files on the website, the images are all there, and there’s no problem on those pages. So I suspect the 404s on those image files are somehow related to the 404 triggered by the wp-cron issue that occurred on those pages.

These are my observations so far. I don’t know all the technically correct terms to use, so I’m trying to relay the info as clearly as I can!
[ Gravatar Icon ]

Tony – #34

Deb, you’re welcome! Your site is really nice, and the design is great, not cluttered at all. I opened several pages and every single item (css, images, js, etc.) on every page loads flawlessly. I really recommend using Firebug when trying something as serious as the 4G blacklist.
[ Gravatar Icon ]

Deb Phillips – #35

Thanks so much, Tony. I’m honored that you stopped by my website.

As far as Firebug, I do use it, but I usually use it related to experimenting with graphics or layout changes prior to actually modifying the CSS.

Can you describe the steps (maybe just the buttons or the menu sequence) you typically take — without having to spend a lot of time to answer this — to use Firebug to troubleshoot issues such as the ones we’ve encountered here? (I had mainly been looking at the Latest Visitor module in cPanel and noting the error codes that were generated on specific URLs.)

Please don’t take a lot of time to answer this. (In that case, I’ll just have to do a little research on it.) I don’t want to create work for other people!

Thanks!
[ Gravatar Icon ]

Deb Phillips – #36

By the way, Tony. Please tell me it was you who happened to do some “testing” while visiting my site. That’ll make me feel a little better. Thanks!
[ Gravatar Icon ]

Tony – #37

Deb,
Jeff shared so much of his work with us; how do a few minutes of my time compare to that? :-D

For troubleshooting this kind of stuff, I use the “Net” tab in Firebug. Some screenshots of the Net tab from the official site:
http://getfirebug.com/net.html

Whenever an item of the page you’re loading gets a 403/404 response or other errors, you’ll see it in red. You can right click and copy its location, and do other stuff.

If you want to see how the page items are loading, but are only interested in, let’s say, the css files only, click on the button that says CSS above the Net tab. Of course the Net tab should be active for you to be able to even see those buttons.

I’m not that good at explaining things, especially when it’s not concrete. So please don’t be shy to ask questions. I am new at this myself, so we’ll both learn something! ;-)
[ Gravatar Icon ]

Tony – #38

Yes it was me who tested. I even wrote hideb.inc in the URL. I hope it’s okay…
[ Gravatar Icon ]

Deb Phillips – #39

Thanks for the Firebug info, Tony. I’ll do some experimenting with it.

Yes, I saw the “hideb.inc.” I’m relieved that was you. I was beginning to think the Internet was more treacherous than I’d imagined!

P.S. Go UNC!
[ Gravatar Icon ]

Jeff Starr – #40

@Tony: I have updated the post with the fix for the WP-Cron plugin. Thank you (again) for helping troubleshoot the 4G. As for why WordPress returns its 404 instead of Apache returning a 403, I really don’t know. I am using custom WP 404 pages for all of my themes and they are only returned for requests that are both not found and not blocked by the blacklist. It could have something to do with how your server is configured, but I don’t dare fathom a guess as to what that might be..
[ Gravatar Icon ]

Tony – #41

Jeff,

I’m happy to help, at least a little!

As for wp-cron, this is not the plugin (which is really, really outdated). It’s the standard wp-cron.php in the Wordpress package.

Is it possible to keep HEAD in the request-method filtering rules, but make some kind of exception for wp-cron.php?

If I find out why Wordpress is returning the 404s, I’ll post here.
[ Gravatar Icon ]

Andrew – #42

@Greg and @Jeff:

Oh, sorry, I forgot to mention: I was putting this in my httpd.conf file, not a .htaccess file. Sorry for the confusion!

As I mentioned, those directives are allowed in htaccess files, so if you just drop it into an htaccess file, it should be fine. It’s only in the httpd.conf file that it needs the extra wrapper.
[ Gravatar Icon ]

Deb Phillips – #43

Just to get some clarification, Jeff and Tony:

It sounds like the WP-Cron fix you’ve just implemented, Jeff, does not address the 404s received in relation to wp-cron.php? That’s what I’m continuing to receive droves of 404s on. (I’ve not tried your revised code yet.)

Thank you.
[ Gravatar Icon ]

Tony – #44

Deb,
I don’t think Jeff revised any of the code. He just advised to remove HEAD.

If you removed it yourself, can you paste the line from your raw access log where it says wp-cron.php?

Also, when you remove HEAD, be sure to remove the “|” after it.
[ Gravatar Icon ]

Jeff Starr – #45

@Andrew: thanks for the clarification on that — it will come in handy for others who might be experiencing the same issue. Cheers.

@Deb: As the issue is related to the WordPress core and not a plugin, I am going to I am going to update the 4G Blacklist itself by removing the HEAD method from the request-method filtering rules. This should eliminate the 404 errors related to wp-cron.php. I will update the code after leaving this comment.
[ Gravatar Icon ]

Jeff Starr – #46

Alright, I think we’re getting there for both Joomla and WordPress. I hate to delete the HEAD method from the request-filtering rules, so I will probably try to write some directives that omit requests for wp-cron.php. This most likely will happen in the next version of the Blacklist, which is actually already underway ;)
[ Gravatar Icon ]

Tony – #47

Jeff,
That’s very good to know! I’m already waiting for the 5G! :)
[ Gravatar Icon ]

Deb Phillips – #48

Thanks so much, Jeff. I must say, you’re one hard-working, conscientious dude! We’re fortunate you’re on the “good side”!
[ Gravatar Icon ]

Greg – #49

Ok with that.

No more feedback for joomla at this time.

Thanks for your work Jeff
[ Gravatar Icon ]

eezz – #50

Hi, Great work… one thing I have just picked up with Joomla when using OpenID… the redirect url back to Joomla from the OpenID server is denied. The url is very long and contains http and ? and plenty of %XX. I’ve tried a few OpenID’s and they work with the following mod:

RewriteCond %{QUERY_STRING} \.\.\/ [NC,OR]
RewriteCond %{QUERY_STRING} boot\.ini [NC,OR]
RewriteCond %{QUERY_STRING} tag\= [NC,OR]
RewriteCond %{QUERY_STRING} ftp\: [NC,OR]
RewriteCond %{QUERY_STRING} https\: [NC,OR]
RewriteCond %{QUERY_STRING} mosConfig [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(\(|\)||'|"|;|\*).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(%0|%A|%B|%C|%D|%E|%F|127\.0).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(globals|encode|localhost|loopback).* [NC,OR]
RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare|drop).* [NC]
RewriteRule ^(.*)$ - [F,L]
[ Gravatar Icon ]

Garrett W. – #51

Ran into a few problems so I thought I’d let you know.

Under Query String Exploits:
- Blocking “config” interferes with the Akismet settings page in Wordpress
- Blocking “request” interferes with phpMyID ( http://siege.org/projects/phpMyID/ ) if I remember correctly
- Blocking “;” interferes with all but a few pages of SMF ( http://www.simplemachines.org/ )

I had too many problems with phpMyID anyway and decided to stop using it, but I had to remove the other two offending strings.

Also, under Character Strings:
- Blocking “,” interferes with SMF forum/subforum/topic pages IF search-engine friendly URLs are enabled

Also, under Query String Exploits:
- Blocking “select” interferes with the “Announce topic” option when creating a new topic in SMF. URL: “.../forums/?action=announce;sa=selectgroup;topic=2;goback”
[ Gravatar Icon ]

Greg – #52

Hi Garrett W.

Ok I give it a try on my own SMF forum too (wthout sef rewrite)
Tell you some feedbacks…
[ Gravatar Icon ]

Jeff Starr – #53

@eezz, @Garrett W.: Thank you both — I have annotated the “Updates” section to include a link to this information. Also Garrett, I took the liberty of consolidating your posts to facilitate usability. I hope you don’t mind ;)
[ Gravatar Icon ]

Garrett W. – #54

I don’t mind at all - I’m glad you did :)
[ Gravatar Icon ]

eezz – #55

Glad to help… bandwidth usage has almost halved on my site with using both the 4G and UA ‘firewalls’ you have made. This is a fantastic contribution the fight against the bot hoards.
[ Gravatar Icon ]

Jeff Starr – #56

@eezz: Excellent! Great to hear the 4G work is helping in the bot wars! ;)
[ Gravatar Icon ]

Garrett W. – #57

Jeff: have you heard of Project Honeypot? ( http://projecthoneypot.org/ )
I recently blocked 4 IPs from my site due to comment spam (and now I’m getting none at all!) and sure enough, all 4 IPs were listed as comment spammers in the Honeypot database.
Wish I had noticed it sooner, but they offer a service called http:BL (BL standing for Blacklist) that queries their database about every visitor to your site (in the form of a quick DNS request), and blocks the ones that are malicious. It even comes in the form of a Wordpress plugin! So I think I’m gonna try it out.
http://projecthoneypot.org/httpbl_implementations.php
[ Gravatar Icon ]

Jeff Starr – #58

Garrett W.: Yes, and it is a great program they’ve got going over there. There are also several other useful WordPress plugins such as Maximum Security, WordPress Firewall Plugin, and even WPIDS. There also are many other applications, scripts, and methods that help fight against malicious mischief on the Web.
[ Gravatar Icon ]

Donace | The Nexus – #59

@Garrett; bad behaviour plugin has honeypot blacklist implemented into it; I advise you check that out.

Would also recommend doing subtle changes to the comment field names as that is how harvesters harvest the urls to spam ;)
[ Gravatar Icon ]

Deb Phillips – #60

Hey, Jeff —

I don’t want to put you on the spot, but I’m a real novice at the bot wars and how to deal with them. So when the mention was made in previous comments about using plugins to combat bad bots, it made me wonder why you chose to take the time to come up with code to add to the .htaccess file versus using a plugin or two.

Are there advantages to taking your route versus plugins? Perhaps WordPress performance advantages? Or something else?

I’m just curious! :) Thanks.

Deb
[ Gravatar Icon ]

Garrett W. – #61

Using .htaccess stuff heads ‘em off at the front, instead of allowing them to see your site first. Plus, it’s probably faster than letting the plugins deal with them in PHP.
[ Gravatar Icon ]

Deb Phillips – #62

Then I’ll stay on this path, Garrett W.! Thanks for answering.
[ Gravatar Icon ]

Tony – #63

What Garrett said, plus the wonderful thing about this blacklist is that you can use it even if you don’t use WordPress.

I feel great when I know that some malicious bots don’t consume any of my bandwidth and don’t slow my websites. Stop them at the door! ;-)
[ Gravatar Icon ]

Jeff Starr – #64

@Deb Phillips: Sure, it’s a good question that I am sure other people have wondered about. The main reason I work with Apache/HTAccess for blacklisting involves performance, as you suggest. In my experience using a plugin really slows things down, especially anti-spam and blacklisting plugins that must interact with both PHP and a database. Throw WordPress functionality into the equation and performance may be affected drastically. I say “may be affected” because different servers and configurations will also play a role in determining overall performance, as will the WordPress setup in question.

Also, as Garrett W. points out, stopping malicious behavior before it reaches the inside of your site eliminates potential vulnerabilities and thus provides a greater degree of security.

Tony also makes a good point. Even though the 4G Blacklist is geared heavily toward WordPress, there are many Joomla/Mambo users who also enjoy its benefits. I would suspect many other sites do as well. ;)
[ Gravatar Icon ]

Yieu – #65

I just wanted to say that I implemented the 4G Blacklist on my website, and it is a regular website — not a blogging website such as Wordpress. It is very handy, as I know of no other blacklist such as this, and it appears to be very comprehensive. I do hope the fact that it is geared towards Wordpress does not leave out some security for regular websites, though. Perhaps a version geared towards regular websites might benefit other webmasters?

I also make use of the proxy blocking blacklist found here which helps keep spammers away, and the universal URL canonization directive. Thank you for making all of these available, it has definitely helped greatly.
[ Gravatar Icon ]

Jeff Starr – #66

@Yieu: My pleasure, and thanks for the feedback. Other people have also requested a version of the Blacklist that is geared for “regular” websites, and I am certainly considering putting something together. Keep in mind, however, that WordPress is the most popular blogging/website software in the world, and as such it is highly targeted for malicious behavior. Thus, even if your site is not running WordPress, there are scores of WP-related scans and bad requests hitting your server and wasting bandwidth nonetheless. In my experience, there are far fewer attacks directed specifically at general sites, such that a regular-site (i.e., non-WP) blacklist would leave your site wide open to the relentless barrage of platform-specific attacks.
[ Gravatar Icon ]

Yieu – #67

When I said a version geared towards regular websites, I did not mean to only include directives relevant to regular websites and to leave out the Wordpress directives. I noticed that you were removing directives because they were in conflcit with Wordpress, so one geared towards regular websites would re-include those directives for a tighter level of security — so it would leave in the Wordpress directives, add in the ones that were removed for Wordpress compatibility, and add directives geared specifically for regular websites on top of that.

I am not sure if this is asking for a too much (the current 4G Blacklist is very nice as it is), but it would also be a more universal security solution for websites and I am sure that would be useful. Such a list may require some explanation for some of the directives that are more liable to cause issues, though (or to simply continue leaving some out if they are too restrictive).
[ Gravatar Icon ]

Greg – #68

Hi,

today I found that the %3D block some of my google ads, so:

RewriteCond %{QUERY_STRING} ^.*(%22|%27|%3C|%3D|%3E|%7B|%7C).* [NC,OR]

become (for me):

RewriteCond %{QUERY_STRING} ^.*(%22|%27|%3C|%3E|%7B|%7C).* [NC,OR]
[ Gravatar Icon ]

Jeff Starr – #69

@Yieu: Yes, I see.. Keep in mind that only a handful of the original directives were removed on account of WordPress, Joomla, and various plugins. Re-inclusion of these directives should be straightforward, however implementing additional rules for “regular” websites will take some time.

Actually — and you probably figured this — during the development of the 4G, many powerful rules were tested and ultimately dropped strictly for the sake of WordPress. These incompatible directives could be re-included in a non-WordPress blacklist, such as you have described.

It is a good idea and I will try to release something along those lines with the next version (5G) of the blacklist. Thanks for pushing the idea ;)
[ Gravatar Icon ]

Jeff Starr – #70

@Greg: Thank you Sir! — That information may come in handy for people running the same configuration :)
[ Gravatar Icon ]

Yieu – #71

I have just installed vBulletin version 3.8.2 on my website which runs the 4G Blacklist, and I had to comment out the following directives:

# RedirectMatch 403 display\.
# RedirectMatch 403 register\.

The 403 display directive was blocking the individual forums from displaying, because their link includes “forumdisplay.php”, and the 403 register directive was preventing the registering of new accounts because the link to register includes “register.php”.
[ Gravatar Icon ]

Dave Stuttard – #72

Hi, this innovative package looks great. Just implemented the 4G Blacklist in .htaccess on 3 sites (all standard, ie not WP or Joomla!), two being static, one being dynamic with a CMS. All features work fine, except:

1) I had to comment out http\ in the Query String Exploits before my Flash elements displayed - thought I’d mention that because nobody else has noted it.
2) I had to comment out RedirectMatch 403 config\ in the SPECIFIC EXPLOITS before the wysywyg Editor box was displayed for User data entry on the dynamic site - the code with ‘config’ in it is (the id):

Now I will study the stats to make sure Google, Yahoo! and MSN get 200s. I’ll let you know….. Many thanks
[ Gravatar Icon ]

Dave Stuttard – #73

Sorry, the code entered in my last post is:


[ Gravatar Icon ]

Yieu – #74

In order to allow private messages on vBulletin, I had to remove “insert” from the following directive:

RewriteCond %{QUERY_STRING} ^.*(request|select|insert|union|declare|drop).* [NC]

becomes:

RewriteCond %{QUERY_STRING} ^.*(request|select|union|declare|drop).* [NC]
[ Gravatar Icon ]

Dave Stuttard – #75

Hi, I found another thing needing to be commented out (this time from 403 errors):
RedirectMatch 403 \,
This was because there are some %2s, ie commas, in dynamically-generated property page urls.
Not much happening in my stats (it’s Sunday); I’ll check Monday evening to see if the search engines get 200s for any pages.
If it’s OK I’ll buy you a pint (and follow your progress of course)!
The importance of checking every user operation/display on the websites is clearly emphasised.
Thanks, Dave
PS: Don’t know about the ban proxy server script - I imagine thousands of legit ISP proxys needing to be identified and allowed to get any visitors (I posted an email on this but don’t know if it was sent).
[ Gravatar Icon ]

Greg – #76

Hi again ^^

Just commented out this one:
#RedirectMatch 403 \;

Cause block the vidéos player links of the jomtube component for Joomla 1.5.10
[ Gravatar Icon ]

Dave Stuttard – #77

Hi, just wanted to confirm that G,Y and M appear to be crawling successfully on my 3 sites. Wow, that means this blacklist is an excellent addition to a website’s security setup (with such things as captchas and URL verifications in forms and email address obfuscating) - when thorough testing has been done and any problem entries have been commented out. It saves a lot of time trying to keep track of and banning bad bots individually. Hope my posts were useful. Looking forward to its further development, eg versions for different types of website, with possible problem entries highlighted to help diagnosis when things go wrong?
I know - I owe the author a pint (Jeff?). And thanks to others who have contributed useful suggestions.
Regards, Dave
[ Gravatar Icon ]

Jeff Starr – #78

@Yieu: Thanks for the help with vBulletin. Good to hear that only a few items were removed. I have updated the post notes for people running vBulletin. If you find anything else, please let us know. Thanks :)

@Greg: Thanks! Post notes for Joomla have been updated accordingly.

@Dave Stuttard: Thank you for posting your findings with your sites. Any additional updates are encouraged, of course. Software-specific modifications are added to the “notes” section at the end of the article, but I am sure that the general information you provide will help others diagnose and troubleshoot if/when similar situations arise. Thanks for your help with the 4G — I am glad you have found it be a useful addition to your security strategy. And yes, I will definitely take you up on that pint! :)
[ Gravatar Icon ]

Greg – #79

More precisions about my previous %3D submission here:http://perishablepress.com/press/2009/03/16/the-perishable-press-4g-blacklist/#comment-71184

This one is only useful to be applied only when you need to preview some ads on your site, when you are in your googlads administration.
[ Gravatar Icon ]

Greg – #80

On my forum SMF 2.0RC1

RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare|drop).* [NC]

becomes:
RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare).* [NC]

Because all my dropdown menus were blocked
[ Gravatar Icon ]

Greg – #81

Hi,

Joomla 1.5.10

change:
RewriteCond %{QUERY_STRING} ^.*(%0|%A|%B|%C|%D|%E|%F|127\.0).* [NC,OR]

to:
RewriteCond %{QUERY_STRING} ^.*(%0|%A|%B|%C|%D|%F|127\.0).* [NC,OR]

because the %E blocked the poll component
[ Gravatar Icon ]

Tom – #82

What a great resource :) Works a treat, but I encountered a problem with Remository on Joomla 1.5.x. Here’s the offending URL string that triggers a 403:

/component/remository/func-download/248/chk,8e2144d3ae6dbfd7591edaa4c16526df/no_html,1/
[ Gravatar Icon ]

Greg – #83

Hi, Tom

did you check the updates for joomla on the end of this article ?
[ Gravatar Icon ]

Tom – #84

Did I check the updates? Not yet :(

I did narrow it down to the filtering of the comma in the URL string. If I comment that out, Remository downloads work. However, I detect from your writing that you really don’t like eliminating an entire class of filter, because it then leaves other potential exploits available. That’s why I was supplying the “bad” URL string…which has *two* instances of a comma in it.

Will check the Joomla updates later this morning. And Greg: thanks again. You provide some wonderful stuff :)
[ Gravatar Icon ]

Greg – #85

ok, thanks Tom

…the stuff master is Jeff Starr ;-)
[ Gravatar Icon ]

Tom – #86

I see Dave Stuttard in comment #75 already picked up this particular issue, albeit it with a different Joomla! component. The comma in a URL does affect Remository in Joomla! 1.5.x, and one way to fix it is to comment out the “RedirectMatch 403 \,” directive.
[ Gravatar Icon ]

Dave Stuttard – #87

Tom
Just a thought - I don’t use Joomla! or Wordpress or SMF or vBulletin, etc. My site with the comma in some address strings is a conventional php/MySQL driven dynanic site - the point is that the 4G Blacklist may have been aimed at Joomla! and Wordpress scenarios originally but it can also apply to other scenarios like mine.

General Point
‘SpamBlockerUtility’, ScoutJet and maybe others are still getting in so maybe these should be dealt with individually with ‘RewriteCond’s (otherwise one usually has to deny ranges of IPs, not single IPs, possibly blocking some legitimate visitors):

RewriteCond %{HTTP_USER_AGENT} spamblockerutility [NC,OR]
RewriteCond %{HTTP_USER_AGENT} scoutjet [NC]
RewriteRule ^.* - [F,L]

(note the absense of ^ before the UA element - without it we can ban any UA string that includes that element (somebody correct me please if my code is wrong) and don’t forget the NC to make it case-insensitive.
[ Gravatar Icon ]

Chaos Inc. – #88

Hiya Jeff!

Already mailed ya yesterday regarding this but I want to add a little more (a lot more lol) feedback with the ones I already mentioned in the mail. Also, I hope that this will help you and in turn, help us, to generate the ultimate GX (generation X) Blacklist.

After implementing only your 4g blacklist on my blog:

My Wordpress Auto Thumb Generation plugin (TimThumb) is not working, and it maybe because the auto generated images/thumbs carries some & or % characters/strings.

Users browsing from all and any kind of mobile platform is being shown the
403.

Users browsing from Opera mobile, opera mini, and other symbian/iphone
applications are being blocked. There are two kinds of connection for Opera
mini : Socket and Http based connections. Neither works. Some mobile phone
browsers also user shadow connects by default and those are also being
blocked.

Both symbian, python and flashlite based browsers are being shown 403.

I’m a mobile theme and ui developer and people similar to me who actually
produces content are being scraped severely, suffering stolen content
dilemmas and other general dos based attacks. Is there a way I can block
someone/everyone who regularly visits a site like scraping.com and visiting
mine as well. What i mean to say is that can i block someone who is probably
stealing my content and posting in his site? Since there is a usage pattern
and trend for that visitor like he visits his site first and then visits
mine to copy and he does this for every post i make. Cant they be blocked
based on their behaviorat pattern?

Almost all telecom operators usually provide 3g, gprs or 2.5g based mobile
internet to all their clients. Typically a mobile user from a country like
Bangladesh (my country) will have the ip of Norway on his phone because the
telecom or gsm operator’s mobile internet gateway for all the countries
they operate in are going through a single wireless internet platform (to be
cost effective and so on). This platform actually works as a hub for their
global 3g network in different countries. Is there a way to somehow let
users enter my site by skipping all the htaccess rules set by me when they
are from a mobile phone/pda/iphone etc.

Lastly, almost anyone who have implemented cname records for his WAP or
Online Store for a subdomain on his site cannot go back to the main
page/normal-pages when they have visited the store ( otherwise referred by
the store). This can be because of the weird caniocal url for any content on
the store/wapsite typically produce (not sure though). Is there a way to let
users browse my site normally when referred from a specific
domain/subdomain/platform etc?

Some mobile phone focused search engine crawlers are being blocked. Like
those from google mobile or mobile meta tag search. Is there a way to fix
this and let them in?
[ Gravatar Icon ]

Greg – #89

Done this today for SMF 2.0 RC1

RewriteCond %{QUERY_STRING} ^.*(\(|\)||'|"|\?|\*).* [NC,OR]

Because with the \[|\] we can’t erase some messages in the private box of the forum (MP)
[ Gravatar Icon ]

Lee – #90

Ah yes! Working great! Thanks, man. We all appreciate it.
[ Gravatar Icon ]

John Hoff - WpBlogHost – #91

Jeff, this is an excellent list of directives. Thank you for sharing.

Question. One of the services we provide is hardening people’s WordPress blog. I’d like to include these in our upgrades. Would you be open to letting me paste these directives in to our customer’s .htaccess files? I’d be sure to leave the comment showing your site and anything else you might request.

Thanks. Stumbling this article and Tweeting it!
[ Gravatar Icon ]

Nihar – #92

Awesome list.

I will add it in .htaccess file.

Have on one doubt, Where should i add the above list? after wordpress lines in htaccess file or before?
[ Gravatar Icon ]

Garrett W. – #93

Before.
[ Gravatar Icon ]

Jeff Starr – #94

@John Hoff - WpBlogHost: Thanks for the great feedback — much appreciated.

As for including the 4G blacklist into your customer’s HTAccess files, contact me directly to discuss further. Thanks.
[ Gravatar Icon ]

Douglas Machado – #95

Have you ever tested Apache mod_security?
Which is the best (most reliable) and faster security solution 4G Blacklist of mod_security?

ModSecurity website:
http://www.modsecurity.org/projects/modsecurity/apache/index.html
[ Gravatar Icon ]

Jeff Starr – #96

@Douglas Machado: Without a doubt mod_security. 4G is designed to thwart specific types of attacks while mod_security is a much more comprehensive security strategy. I am running a modified version of 4G to complement mod_security, not replace it. So, if you have to pick, go with mod_security. Then, if you need additional protection, try the 4G Blacklist. ;)
[ Gravatar Icon ]

MileHighTechGuy – #97

This is great stuff.

Under WordPress I had to comment out the following ‘config’ line for the ‘Deans FCK Editor’ to work:

# SPECIFIC EXPLOITS
#this line conflicts with Deans FCK Editor WP plugin
#RedirectMatch 403 config\.

I’m posting a security related article later today that will reference your work.

Thanks for the great effort.

~Jeff (MileHighTechGuy)
[ Gravatar Icon ]

Jeff Starr – #98

@MileHighTechGuy: Awesome, Jeff - thanks for the mention in your article. I am taking a copy with me to read at work — it looks like a very informative reference that should help lots of people protect their WordPress installations. Thanks also for the heads up on the 4G modification for the FCK Editor WP plugin. Cheers.
[ Gravatar Icon ]

MileHighTechGuy – #99

I’m glad you think my post might have some value. Thanks for the feedback.

Not only is it posted here on my blog:
http://milehightechguy.com/2009/06/how-to-guide-for-securing-wordpress-and-protecting-websites/

But it is also posted here at Examiner.com:
http://www.examiner.com/…/WordPress-installation-and-recove
[ Gravatar Icon ]

Vladimir – #100

Jeff, if Apache stores its logs in CLF format and you have Linux, here’s a nice command to get all 404’s:

cat *.log | grep " 404 " | awk '{ print $7; }' | sort | uniq

I guess it can give you some more patterns to check against ;-)
[ Gravatar Icon ]

Brandon – #101

This line:

RewriteRule ^(.*)$ - [F,L]

stops WordPress 2.8’s widget manipulation screen from working. It kills the javascript.
[ Gravatar Icon ]

Jeff Starr – #102

@Vladimir: Very nice - thank you, Sir :)

@Brandon: That particular RewriteRule is associated with two different sets of directives: one for filter-request methods and the other for query-string exploits. Do you happen to know which set of directives is responsible for the issue you describe? Thanks.
[ Gravatar Icon ]

Brandon – #103

Sorry, that was query-string exploits.
[ Gravatar Icon ]

MIke Selvage – #104

Mike Selvage: Great work! But I have a question. I get the idea the 4G Blacklist supersedes the User_Agent blacklist … but does it also supersede the Referrer blacklist?
[ Gravatar Icon ]

MIke Selvage – #105

Mike Selvage: Disregard the last post, please. In the past 2 minutes I learned how to read …
[ Gravatar Icon ]

Garrett W. – #106

Just found another problem: The following from Wordpress is caught and returns a 403.
/wp-admin/plugins.php?action=delete-selected&checked[0]=adsense-manager%2Fadsense-manager.php&plugin_status=all&paged=1&_wpnonce=159d52cca3

…but I can’t figure out which block is triggering it. I tried taking out everything that blocked [ and ], as well as %5B and %5D, and it still wouldn’t go through. Any idea?
[ Gravatar Icon ]

Jeff Starr – #107

Hey Garret, it looks like most of that URL is comprised of a query string, so you may want to check that section closely. In particular, it looks like the brackets may be causing the issue.

Let me know how it goes and I will update the article.
[ Gravatar Icon ]

Tom – #108

I have to echo Brandon in comment #101. I’ve upgraded a couple of sites to WordPress 2.8, and cannot modify or move widgets. When I replace the 4G list with a standard WordPress .htaccess file, the ajaxy-goodness of WP 2.8 works again. Restore 4G in the .htaccess, and it does not. I’m sorry I can’t be more specific than that. I’m not really seeing what wp-admin is trying to call when I’m attempting to twiddle with widgets. My workaround for the moment is to temporarily replace the 4G-based .htaccess with a vanilla .htaccess, but that is not particularly convenient. Thoughts?
[ Gravatar Icon ]

Garrett W. – #109

Ah, so now that I’m experiencing the same thing, I realize that someone already commented about it. lol.
So the blacklist is responsible? That’s good - saved me from having to trudge through WordPress’s code.
However, I did look through it a little, and I found the offending piece of info.

These lines:
RewriteCond %{QUERY_STRING} (request|select|insert|union|declare|drop) [NC]
RewriteRule .* - [F,L]

are being triggered by:
wp-admin/load-scripts.php?[…]load=[…],jquery-ui-droppable,[…]
[ Gravatar Icon ]

Jeff Starr – #110

Thanks Garrett. For those that don’t speak voodoo, you may easily resolve this issue by removing the “|drop” from the Query-String Exploits section. That single edit should fix you up without compromising the overall effectiveness of the blacklist.
[ Gravatar Icon ]

Tom – #111

Jeff, your change in #110 solved the problem for me in WordPress 2.8. And thanks Garrett for providing enough detail, where my information was sorely lacking!
[ Gravatar Icon ]

Garrett W. – #112

fyi, my name has two Ts in it ;)
[ Gravatar Icon ]

Jeff Starr – #113

That’s what I get for not copy/pasting - Fixed. :P
[ Gravatar Icon ]

Garrett W. – #114

Thanks. I wouldn’t have mentioned it, but you were consistently misspelling it… and that’s a pet peeve of mine. ;)
[ Gravatar Icon ]

Jeff Starr – #115

God, I feel awful now ;)
[ Gravatar Icon ]

Garrett W. – #116

Aw, you poor thing .. it’s ok, I forgive you :)
[ Gravatar Icon ]

Jeff Starr – #117

Thanks! :)
[ Gravatar Icon ]

Scotia – #118

Joomla 1.5.12
Community Builder v1.21
SOBI2 v2.9.2.1
sh404SEF dev. 1.0.20 Build 237
Smartoptimizer v1.7b
CssJscompress v1.1

CBlogin module (to log into user accounts to work) I had to comment out:

# RewriteCond %{QUERY_STRING} ^.*(select|insert|union|declare|drop).* [NC] - I tried each of select|insert|union|declare|drop separately, still would not work. - What is the impact of losing this one?

SOBI2 Search functionality was messed until I commented out:

# RewriteRule ^(.*)$ - [F,L] - Can I change anything here for this to work? What is the impact of losing this one?

CBLogin mod Lost password functionality lost (Ajax) until one of these two was commented out:

# RedirectMatch 403 register\.
# Redirectmatch 403 password\.

I hope this is usefull for anyone else running Joomla, in combination with CB, sh404 etc.
[ Gravatar Icon ]

Greg – #119

For select|insert|union|declare|drop , so this is probably not the real cause.

For # RewriteRule ^(.*)$ - [F,L], this is the rewrite rule, so if you comment it out, all the directives abvo becomes inneficient.

Add something like at the first place of the query strings block: RewriteCond %{HTTP_REFERER} !^(.*)(put_the_name_of_thepage_suspected_here).* [NC] could help…

But you have to find the real parameter that is blocking SOBI.
[ Gravatar Icon ]

Greg – #120

Hi,

so many like that in my log file:

"HEAD /favicon.ico HTTP/1.0" 403 - "-" "-"

How should we deal with thoses ?

Should we had :

RewriteCond %{REQUEST_URI} !^/favicon.ico

Just after the # FILTER REQUEST METHODS ?
[ Gravatar Icon ]

Jeff Starr – #121

Hi Greg, you could try that, or use this method for additional help. Cheers.
[ Gravatar Icon ]

Greg – #122

Thanks Jeff

Ok with that, but I ask this, because those are requests with the HEAD method, so I was wondering how to deal with the favicon requested with HEAD.
Should we consider those as legitimate ?
Or it must be only allowed from a GET ?
[ Gravatar Icon ]

Jeff Starr – #123

Ah yes, sorry I missed that part of the request. The thing about HEAD requests is that the server ideally does not return a message-body in response. Other than that, GET and HEAD are essentially the same. HEAD is typically used for testing (either proper or malicious). Whether or not to allow HEAD requests for various files (such as favicon) is up to you.
[ Gravatar Icon ]

Greg – #124

Found it !

Lol it was me….all my tests with firebug an YSlow were made all this HEAD requests.
[ Gravatar Icon ]

Jeff Starr – #125

Lol, ah yes - I love it when that sort of thing happens =)

Glad to hear you got it sorted out :)
[ Gravatar Icon ]

ken – #126

If you block some ip addresses would you block out alot of people that do not have a static ip. I also saw some question about mobile phones not being able to access site. I did not see answer or if it is a problem.
[ Gravatar Icon ]

duck – #127

Firstly, thanks for such a great resource, this post and the building of the 4G blacklist are great posts.

I discovered that after implementing the 4G blacklist on a Wordpress 2.8.4 website that I was unable to delete plugins from the Plugin menu (was mentioned in comment 106). The link of the delete button takes the form:

...plugins.php?action=delete-selected&checked[0]=hello.php&plugin_status=all&paged=1&_wpno...

To fix this you need to comment out the lines blocking [ and ] as well as removing select from the last condition in the Query string exploits section.

(It might be useful to comment out the lines blocking %5B and %5D, but it wasn’t necessary for me)
[ Gravatar Icon ]

duck – #128

I need a bit of help as an implementation of the 4G blacklist doesn’t seem to be working. It is working perfectly on my local machine, giving me 403 errors if I put “password.” at the end of the url for example, however on the production server it is producing 404 errors instead. Looking in cPanel redirects I see that it is interpreting all the RedirectMatch 403 rules as pointing to a directory 403, being type permanent and the redirect url being something like \. or \,.

Do you think that this is an issue with cPanel, my webhost’s setup or anything else you have come across before?
[ Gravatar Icon ]

Jeff Starr – #129

Hi duck, I have seen this behavior before on one of my shared servers. Have not been able to pinpoint the exact cause, but have been able to workaround by either using a different error (something other than 403) or relocating the directives into a directory higher up the tree.
[ Gravatar Icon ]

jon (aka duck) – #130

I discovered that, at least for my situation, the server was still responding with a 403 status header but just not displaying an Access Denied page. All I had to do was declare an ErrorDocument in my htaccess file e.g.

ErrorDocument 403 "

Access Forbidden!

Error 403

"

and it worked. This is obviously something to do with the way the server had been configured as it was all working perfectly on my local test install of apache.
[ Gravatar Icon ]

Jeff Starr – #131

Interesting, jon — I will have to keep that in mind the next time I find myself trying to resolve that issue. Another thing to consider is that WordPress sometimes will override errors if a custom error page is available on the active theme. I am still working on how to workaround that one. Thanks for sharing your solution. Cheers.
[ Gravatar Icon ]

Cooltad – #132

I’m insane about speed and optimization.
Do you have a flattened version of your 4G list and it’s optional components?
[ Gravatar Icon ]

Jeff Starr – #133

Hi Cooltad, not at this time. Do you think a “flattened” version would help with performance?
[ Gravatar Icon ]

Cooltad – #134

Most certainly. We all should know that, even if there are whitespaces and new lines, each of those 2 require one byte, or 6 bits. Now, with your massive 4g list compiled together, you’d save a very large amount of bytes, which would make processing server-side for the .htaccess faster.
You wouldn’t think it would, because it doesn’t ‘parse’ whitespaces etc, however, it still needs to sift through such spaces to get to the real stuff.
[ Gravatar Icon ]

Greg – #135

I think all the Redirectmatch 403, could be in one line.
…for example.
[ Gravatar Icon ]

Jeff Starr – #136

Yes, I can see how this would help for tightly controlled or high-bandwidth scenarios, but even then, the gains would be modest at best.

Even so, I have done this for other blacklists, including the first “Ultimate HTAccess Blacklist” a few years ago. I will be sure to include a “flattened” version of the next, 5G Blacklist.
[ Gravatar Icon ]

Peekay – #137

Xoops users who allow user registration are going to need to comment out:

# RedirectMatch 403 register\.

from the ‘Specific Exploits’ section.
[ Gravatar Icon ]

Omar Ramos – #138

I was wondering how we might be able to test that the blacklist is working correctly?

For example, it seems like it would be as easy as using “select” within the query string of the URL, but when I try doing that I’m not getting a 403 response, the page just ends up loading the same way it did before (I’m using the above code within an .htaccess file).

Any thoughts on how this can be tested simply, just to make sure the rules are in effect?

Thank you!
[ Gravatar Icon ]

Omar Ramos – #139

I tried the visiting the following URL:
index.php?option=com_newsflash&id=8+and+1=1+union+select+1,username,password,4+from+mos_users&catid=0

On our remaining Joomla 1.0 site and it seems to be working.

I got the link from the milw0rm site here:
http://milw0rm.com/exploits/7718

( Hopefully you don’t mind I post the links on here, they seem to be a whitehat group:
http://en.wikipedia.org/wiki/Milw0rm )


Reference: http://perishablepress.com/press/2009/03/16/the-perishable-press-4g-blacklist/