Thursday, October 7, 2010

Apache Performance Tuning MaxClients

SkyHi @ Thursday, October 07, 2010


"Premature optimization is the root of all evil." -- Donald Knuth other words, don't implement in extra complexity if you don't need it. A site handling a few thousand requests per day will do fine on a default configuration and just about any hardware. This article is geared towards a site that needs to handle multiple concurrent requests [ten to several hundred per second].

General [in order of importance]


The single biggest issue affecting webserver performance is RAM. Have as much RAM as your hardware, OS, and funds allow [within reason].

The more RAM your system has, the more processes [and threads] Apache can allocate and use; which directly translates into the amount of concurrent requests/clients Apache can serve.

Generally speaking, disk I/O is usually a close 2nd, followed by CPU speed and network link. Note that a single PII 400 Mhz with 128-256 Megs of RAM can saturate a T3 (45 Mbps) line.

Select MPM

Chose the right MPM for the right job:

prefork [default MPM for Apache 2.0 and 1.3]:
  • Apache 1.3-based.
  • Multiple processes, 1 thread per process, processes handle requests.
  • Used for security and stability.
  • Has higher memory consumption and lower performance over the newer Apache 2.0-based threaded MPMs.
  • Apache 2.0-based.
  • Multiple processes, many threads per process, threads handle requests.
  • Used for lower memory consumption and higher performance.
  • Does not provide the same level of isolation request-to-request, as a process-based MPM does.
  • The only MPM choice under Windows.
  • 1 parent process, exactly 1 child process with many threads, threads handle requests.
  • Best solution under Windows, as on this platform, threads are always "cheaper" to use over processes.

Configure MPM

Core Features and Multi-Processing Modules
Default Configuration
<IfModule prefork.c>
StartServers            8
MinSpareServers         5
MaxSpareServers        20
MaxClients            150
MaxRequestsPerChild  1000

<IfModule worker.c>
StartServers            2
MaxClients            150
MinSpareThreads        25
MaxSpareThreads        75 
ThreadsPerChild        25
MaxRequestsPerChild     0

<IfModule mpm_winnt.c>
ThreadsPerChild       250
MaxRequestsPerChild     0

MaxClients, for prefork MPM

MaxClients sets a limit on the number of simultaneous connections/requests that will be served.

I consider this directive to be the critical factor to a well functioning server. Set this number too low and resources will go to waste. Set this number too high and an influx of connections will bring the server to a stand still. Set this number just right and your server will fully utilize the available resources.

An approximation of this number should be derived by dividing the amount of system memory (physical RAM) available by the maximum size of an apache/httpd process; with a generous amount spared for all other processes.

MaxClients ≈ (RAM - size_all_other_processes)/(size_apache_process)

Use 'ps -ylC httpd --sort:rss' to find process size. Divide number by 1024 to get megabytes. Also try 'top'.

Use 'free -m' for a general overview. The key figure to look at is the buffers/cache used value.

Use 'vmstat 2 5' to display the number of runnable, blocked, and waiting processes; and swap in and swap out.


  • System: VPS (Virtual Private Server), CentOS 4.4, with 128MB RAM
  • Apache: v2.0, mpm_prefork, mod_php, mod_rewrite, mod_ssl, and other modules
  • Other Services: MySQL, Bind, SendMail
  • Reported System Memory: 120MB
  • Reported httpd process size: 7-13MB
  • Assumed memory available to Apache: 90MB

Optimal settings:
  • StartServers 5
  • MinSpareServers 5
  • MaxSpareServers 10
  • ServerLimit 15
  • MaxClients 15
  • MaxRequestsPerChild 2000

With the above configuration, we start with 5-10 processes and set a top limit of 15. Anything above this number will cause serious swapping and thrashing under a load; due to the low amount of RAM available to the [virtual] Server. With a dedicated Server, the default values [ServerLimit 256] will work with 1-2GB of RAM.

When calculating MaxClients, take into consideration that the reported size of a process and the effective size are two different values. In this setup, it might be safe to use 20 or more workers... Play with different values and check your system stats.

Note that when more connections are attempted than there are workers, the connections are placed into a queue. The default queue size value is 511 and can be adjusted with the ListenBackLog directive.

ThreadsPerChild, for winnt MPM
On the Windows side, the only useful directive is ThreadsPerChild, which is usually set to a value of 250 [defaults to 64 without a value]. If you expect more, or less, concurrent connections/requests, set this directive appropriately. Check process size with Task Manager, under different values and server load.
Directive MaxRequestsPerChild is used to recycle processes. When this directive is set to 0, an unlimited amount of requests are allowed per process.

While some might argue that this increases server performance by not burdening Apache with having to destroy and create new processes, there is the other side to the argument...

Setting this value to the amount of requests that a website generates per day, divided by the number of processes, will have the benefit of keeping memory leaks and process bloat to a minimum [both of which are a common problem]. The goal here is to recycle each process once per day, as apache threads gradually increase their memory allocation as they run.

Note that under the winnt MPM model, recycling the only request serving process that Apache contains, can present a problem for some sites with constant and heavy traffic.
Requests vs. Client Connections
On any given connection, to load a page, a client may request many URLs: page, site css files, javascript files, image files, etc.

Multiple requests from one client in rapid succession can have the same effect on a Server as "concurrent" connections [threaded MPMs and directive KeepAlive taken into consideration]. If a particular website requires 10 requests per page, 10 concurrent clients will require MPM settings that are geared more towards 20-70 clients. This issue manifests itself most under a process-based MPM [prefork].

Separate Static and Dynamic Content

Use separate servers for static and dynamic content. Apache processes serving dynamic content will carry overhead and swell to the size of the content being served, never decreasing in size. Each process will incur the size of any loaded PHP or Perl libraries. A 6MB-30MB process size [or 10% of server's memory] is not unusual, and becomes a waist of resources for serving static content.

For a more efficient use of system memory, either use mod_proxy to pass specific requests onto another Apache Server, or use a lightweight server to handle static requests:

  • lighttpd [has experimental win32 builds]
  • tux [patched into RedHat, runs inside the Linux kernel and is at the top of the charts in performance]
The Server handling the static content goes up front.

Note that configuration settings will be quite different between a dynamic content Server and a static content Server.


Reduce bandwidth by 75% and improve response time by using mod_deflate.

LoadModule deflate_module modules/

<Location />

AddOutputFilterByType DEFLATE text/html text/plain text/css text/xml application/x-javascript


Loaded Modules

Reduce memory footprint by loading only the required modules.

Some also advise to statically compile in the needed modules, over building DSOs (Dynamic Shared Objects). Very bad advice. You will need to manually rebuild Apache every time a new version or security advisory for a module is put out, creating more work, more build related headaches, and more downtime.


Include mod_expires for the ability to set expiration dates for specific content; utilizing the 'If-Modified-Since' header cache control sent by the user's browser/proxy. Will save bandwidth and drastically speed up your site for [repeat] visitors.

Note that this can also be implemented with mod_headers.


Enable HTTP persistent connections to improve latency times and reduce server load significantly [25% of original load is not uncommon].

prefork MPM:

KeepAlive On

KeepAliveTimeout 2

MaxKeepAliveRequests 80

worker and winnt MPMs:

KeepAlive On

KeepAliveTimeout 15

MaxKeepAliveRequests 80

With the prefork MPM, it is recommended to set 'KeepAlive' to 'Off'. Otherwise, a client will tie up an entire process for that span of time. Though in my experience, it is more useful to simply set the 'KeepAliveTimeout' value to something very low [2 seconds seems to be the ideal value]. This is not a problem with the worker MPM [thread-based], or under Windows [which only has the thread-based winnt MPM].

With the worker and winnt MPMs, the default 15 second timeout is setup to keep the connection open for the next page request; to better handle a client going from link to link. Check logs to see how long a client remains on each page before moving on to another link. Set value appropriately [do not set higher than 60 seconds].


Make sure 'Options +FollowSymLinks -SymLinksIfOwnerMatch' is set for all directories. Otherwise, Apache will issue an extra system call per filename component to substantiate that the filename is NOT a symlink; and more system calls to match an owner.

<Directory />

Options FollowSymLinks



Set a default 'AllowOverride None' for your filesystem. Otherwise, for a given URL to path translation, Apache will attempt to detect an .htaccess file under every directory level of the given path.

<Directory />

AllowOverride None



If mod_status is included, make sure that directive 'ExtendedStatus' is set to 'Off'. Otherwise, Apache will issue several extra time-related system calls on every request made.

ExtendedStatus Off


Lower the amount of time the server will wait before failing a request.

Timeout 45


Cache all PHP pages, using Squid, and/or a PHP Accelerator and Encoder application, such as APC. Also take a look at mod_cache under Apache 2.2.

Convert/pre-render all PHP pages that do not change request-to-request, to static HTML pages. Use 'wget' or 'HTTrack' to crawl your site and perform this task automatically.

Pre-compress content and pre-generate headers for static pages; send-as-is using mod_asis. Can use 'wget' or 'HTTrack' for this task. Make sure to set zlib Compression Level to a high value (6-9). This will take a considerable amount of load off the server.

Use output buffering under PHP to generate output and serve requests without pauses.

Avoid content negotiation for faster response times.

Make sure log files are being rotated. Apache will not handle large (2gb+) files very well.

Gain a significant performance improvement by using SSL session cache.

Outsource your images to Amazon's Simple Storage Service (S3).

Measuring Web Server Performance

Load Testing

Apache HTTP server benchmarking tool


The Grinder, a Java Load Testing Framework


I have searched extensively for Apache, lighttpd, tux, and other webserver benchmarks. Sadly, just about every single benchmark I could locate appeared to have been performed completely without thought, or with great bias.

Do not trust any posted benchmarks, especially ones done with the 'ab' tool.

The only way to get a valid report is to perform the benchmark yourself.

For valid results, note to test under a system with limited resources, and maximum resources. But most importantly, configure each httpd server application for the specific situation.


System Tuning Info for Linux Servers

php Converting PDF-s to pictures Imagemagick GhostScript

SkyHi @ Thursday, October 07, 2010
Ever wondered how to render PDF-s into individual pictures page by page? It is really simple.

Warning to all of those who are allergic to command line: The following method does imply use (albeit very simple) of CLI.

Here we will be using a tool called ImageMagick. This is a widely available tool with unparalleled versatility in a variety of areas, but is by no means the only one: there are plenty of other tools to do the same thing. The following is only one method of many.

1) So first of all you will need to install ImageMagick. In Ubuntu you would do:

sudo apt-get install imagemagick

2) Then open up a terminal, and browse to the location of the PDF you want to convert. Say, if your PDF was in /home/username/PDFs, you would do the following:

cd /home/username/PDFs

3) Let’s say that you have a PDF file called test.pdf and you want it to convert to PNG files. You also want these PNG files’ filename to begin page. Then you would run:

convert test.pdf page.png

This will give you as many PNG files named page-01.png, page-02.png, … as there are pages in your original PDF.

If you wanted JPEG files instead just run:

convert test.pdf page.jpg

You might also want to control the resolution of the resulting image files. To do that you must specify the -density option. It is hard to predict what number you should give, so try to experiment to get the desired result. If you wanted a density of, say, 800, you would run:

convert -density 800 test.pdf page.png

Want to select certain pages only? No problem, to convert only the first page – run:

convert "test.pdf[0]" page.png

You may also specify a range. So to convert the first three pages run:

convert "test.pdf[0-2]" page.png

Several ranges are also possible:

convert "test.pdf[0-2,4-6]" page.png

(Note that the output files will be consecutively named in the latter example, so you will get files page-0.png, -1, -2, -3, -4 and -5.png. NOT as you would expect page-0.png, -1, -2, -4, -5 and -6)

Edit: If you are inclined to use another conversion utility, try pdftoppm from the package poppler-utils. Works very well, too…


gs -sDEVICE=jpeg -sOutputFile=test.jpg test.pdf 

Wednesday, October 6, 2010

port forwarding vs. port triggering?

SkyHi @ Wednesday, October 06, 2010
FTP has nothing to do with port triggering and everything to do with a statically forwarded port. Its the algorithm in the router that handles the random upper level ports that are used by the server software. Initial negotiation is done by a user outside on the WAN who can access the server because port 21 is forwarded (an open door if you will). Then the software authenticates the user (name/password, however you have set it up) and then starts to manage the request, which means opening up data ports for the actual transfer, the router handles all of this properly so that the outgoing is sent to the originator etc....)

Port triggering is different in that its designed to allow dynamic port forwarding An outgoing request by a user ON THE LAN, tells the router (by means of the port being used on the outgoing request) to open or forward a port or range of ports for a limited period of time, so that incoming traffic on different ports (those opened) can get through the router NAT and to the sender.

In order to use the same service on a different LAN computer, you have to manually replace the LAN computer's IP address in the forwarding port with another LAN computer's IP address.

Trigger port forwarding solves this problem by allowing computers on the LAN to dynamically take turns using the service. The router records the IP address of a LAN computer that sends traffic to the WAN to request a service with a specific port number and protocol (a "trigger" port). When the router's WAN port receives a response with a specific port number and protocol ("incoming" port or port range as designated by the end user), the router forwards the traffic to the LAN IP address of the computer that sent the request. After that computer's connection for that service closes, another computer on the LAN can use the service in the same manner. This way you do not need to configure a new IP address each time you want a different LAN computer to use the application.

For example:

Jane requests a file from the Real Audio server (port 7070).

Port 7070 is a "trigger" port and causes the router to record Jane's computer IP address.
The router associates Jane's computer IP address with the "incoming" port range of 6970-7170 because that what has been set up in the port triggering menu!!

The Real Audio server responds to Janes request and uses the expected port or ports ranging between 6970-7170.
The ZyWALL forwards the traffic to Jane's computer IP address.
Only Jane can connect to the Real Audio server until the connection is closed or times out. The router times out depending on the brand in use, both UDP and TCP. Should be configurable or at least known.

Port forwarding (static) is the process of opening a door in the router,
by the user, so that an outside person/entity can access a PC or server
behind the router.
The door is always open and requires no action by the users on the LAN.

triggering is based on an OUTGOING request by a user on the LAN. It
allows dynamic port forwarding to temporarily occur, so that the user
can access services (REAL AUDIO) or gaming etc..... It assists the
users behind the PC. A method of sharing a port or port forwarding

FTP algorithm is port negotiation between the FTP software
and the router. Initial communication is done by an outside (WAN side)
PC/user, using the open door/port 21. A standard port forwarding case.
FTP program then selects random ports and communicates that with the
WAN side user. The router firmware detects which ports are being used
and ensures that traffic from the WAN user reaches the FTP server, on
the high numbered ports.
This traffic is normally uploading by the
WAN User and thus the ports need to be open so that the return packets
(Ack) which communicate progress of the upload can reach the FTP server.
The other traffic would be in the case of downloading data to the
server by the WAN side user.

It is not triggering of a set range
of identified ports, its a negotiation of randomly assigned ports
between the router and FTP software.


Shell tip: Clear the command history and screen when you log out

SkyHi @ Wednesday, October 06, 2010
"reset" or "clear" will both clear your terminal. "reset" does a bit

more, such as set it back the way it was when you started, in case

some control characters have changed something. For example if you

run 'cat somefile.jpg", somefile.jpg is not plain text so will probably

mess up your terminal in some way.

If you are using "bash", "history -c" will clear the your command history,

and "history -r" will read it again from the history file. So if you do

"history -c; history -r;" your history will be back to what it was when

you started your terminal session.

Combining all three will pretty much put your teminal back to the way it

was when you opened it. This will not clear any variables you may have


"history -c; history -r; reset;"

If this is something you do frequently, you may want to append a line to

your .bashrc file,

alias histscreenclear="history -c; history -r; reset;"

to condense them into one command.


history -c will clear the entire bash history.

To only clear the current session history use:

history -r.

Kumar Chetan Sharma
November 20, 2008

even after running clear or history -c, which I find is the best
option to add to ./bash_logout, I can still scroll my putty screen and
see what all I have done. Is there any way to completely erase this

July 15, 2009

Kumar asked about clearing the information from his putty window.

To do so -

1. Right click on the top bar of the putty window

2. A Context menu will appear

3. Select “Clear Scrollback”

This will erase all details of your session from putty’s scrollback buffer.


Tuesday, October 5, 2010

Hardening Sendmail against DDoS

SkyHi @ Tuesday, October 05, 2010
For some time now, I was experiencing a strange behavior of my server: from time to time, without an ascertainable pattern, the server would stop reacting to network-requests. The teamspeak-server, which runs on it, would kick anyone connected to it, and nothing particular special could be found in the logs. When this happened last Thursday, and I was kicked out of the teamspeak-server myself, I tried to ssh onto my server - which took about 30 seconds. This was irritating, and I run "uptime" to check the server's load - it was way beyond 70. The next thing was a call to top, and here I saw the culprit: sendmail. A call to ps verified the sendmail was running with way too many processes, all in "RCPT TO:" state or something similar. I stopped sendmail and killed remaining processes manually, so that I could work again in real time. Looking through maillog, I began to understand what going on: spammers were DDoSing my mail-server. Though I already had some settings in my that would make the server unattractive for spammers, they were obviously not sufficient, and especially not against DDoS-attacks. So I changed my configuration a bit.

Connection Controlling

FEATURE(`delay_checks', `friend')dnl
FEATURE(`ratecontrol', `nodelay',`terminate')dnl
FEATURE(`conncontrol', `nodelay',`terminate')dnl
After the already existing line "FEATURE(`access_db')dnl" I added the lines to enable rate and connection-controlling.

The option "nodelay" is important, because I am using the delay-checks-feature, and these checks are not to be delayed.

The "terminate"-option tells sendmail to kill all connections exceeding the later defined limits with a temporary error-message. The properly configured and standard-compliant smtp-client will try again later, spammers usually don't.

The rate-control feature enables control over how often a single host is allowed to connect per a defined window. It was introduce by sendmail version 8.13.0, and uses the access.db for defining the limits for single hosts, groups of hosts or all hosts.
The window-size is defined using the this option:
with a default-value for window of "60s" (60 seconds).
My access.db-entries for the rate-control-feature look like this:
ClientRate:localhost                    0
ClientRate:localhost.localdomain        0
ClientRate:                    0
ClientRate:                             5
The first three lines tell sendmail to ingore rate-limits for the localhost, and the last line imposes a limit of 5 connections per window for all hosts.
In cases of a DDoS, it might not be sufficient to limit the connections of a single host per minute, because a DDoS comes from multiple hosts at the same time. This is why Sendmail come with another option:
define(`confCONNECTION_RATE_THROTTLE', `5')dnl
This defines the overall number of concurrent connection the server accepts per second, before queuing incoming connection-request regardless of the host. The connections will not be rejected but stalled until the next second. This means that for the above example that when 20 connection-requests arrive, the first five (1-5) are processed in second one, the second five (6-10) in second two, the third five (11-15) in second three, and the final five (16-20) in second four.

The conn-control feature enables control over the number of concurrent connections a single host is allowed to run simultaneously. Like rate-control, this feature was introduced with Sendmail 8.13, and the access.db is used to define settings for single-hosts, host-ranges and "all hosts", too.
My access.db-entries for the conn-control-feature look like this:
ClientConn:localhost                    0
ClientConn:localhost.localdomain        0
ClientConn:                    0
ClientConn:                             3
The entries are read similar to rate-control. The last line defines a default of 3 concurrent connections, the first three disable the feature for localhost.

Greeting Pause

A common technique of spammers, trojans and viruses is the so-called slamming. The SMTP-Standard requires the client to wait with the HELO/EHLO-Command until the server has sent its greeting line. Slamming is to ignore this, and to start sending immediately.
FEATURE(`greet_pause', `2000')dnl
With the above feature, Sendmail can be configured to delay the sending of this greeting. The value is in milliseconds, so in the example above, the greeting-pause would be two seconds. A client issuing the HELO/EHLO during this pause will cause Sendmail to answer with
554 not accepting messages
and the greeting will not be send. Sendmail will log such attempts with a message like
rejecting messages from  due to pre-greeting traffic.
and terminate the connection.
You can use the access.db again to define host-specific greeting-pause times, or to exclude certain hosts from the pause. The following example would exclude localhost from the delay. You can use this to whitelist smtp-servers who do slamming but are otherwise "friendly".
GreetPause:localhost                    0
GreetPause:localhost.localdomain        0
GreetPause:                    0
Please note:
RFC 2821 specifies 5 minutes as the maximum timeout for the initial connection greeting. Therefore, if you specify a time longer than 300000 milliseconds (i.e. 5 minutes), sendmail will not wait longer than 5 minutes, to maintain RFC compliance.


After setting up the controlling mechanisms for incoming connections, there is a another level of control that can be applied. Many spammers try to send a single mail with hundreds of recipients. This is also known as "recipient flooding". Sendmail can be configured to limit the number of recipients a message may have, as well throttling down all those clients who try to add more recipient than a certain threshold by pausing a hardcoded full second between each accepted recipient. The options are as follows:
define(`confBAD_RCPT_THROTTLE', `2')dnl
define(`confMAX_RCPTS_PER_MESSAGE', `25')dnl
BAD_RCPT_THROTTLE sets the threshold which invokes the one-second-delay. For the example above this means that with the third RCPT TO: sendmail will pause one full second, before sending the response.

MAX_RCPTS_PER_MESSAGE limits the absolute maximum number of recipients for each message to the value given (25 for the above example). Every RCPT TO: exceeding this number will be rejected with an appropriate message. The standard-compliant server will collect the rejected RCPT TOs and requeue the message for all yet outstanding recipients. (Yes, spammers won't.)


Sendmail, in order to get as many as possible mails through, has very generous timeout-defaults. These values are often measured in days, where today seconds or minutes would suffice. Long timeouts mean long bound resources for probably unsolicited connections. I have defined much shorter values for several timeouts:
define(`confTO_INITIAL', `30s')dnl
define(`confTO_CONNECT', `30s')dnl
define(`confTO_ACONNECT', `1m')dnl
define(`confTO_ICONNECT', `30s')dnl
define(`confTO_HELO', `30s')dnl
define(`confTO_MAIL', `30s')dnl
define(`confTO_RCPT', `30s')dnl
define(`confTO_DATAINIT', `1m')dnl
define(`confTO_DATABLOCK', `1m')dnl
define(`confTO_DATAFINAL', `1m')dnl
define(`confTO_RSET', `30s')dnl
define(`confTO_QUIT', `30s')dnl
define(`confTO_MISC', `30s')dnl
define(`confTO_COMMAND', `30s')dnl
define(`confTO_CONTROL', `30s')dnl
define(`confTO_LHLO', `30s')dnl
define(`confTO_AUTH', `30s')dnl
define(`confTO_STARTTLS', `30s')dnl
I won't go into much detail about each timeout, because that would be beyond the scope of this posting, but these values are much more reasonable than the defaults.

Other means of protecting your server agains spammers:


Besides everything sendmail can be configured to do and not to do, sendmail has another advantage: It can be compiled to use TCP Wrapper.
While scanning the logs for the causes of the astronomous load, I noticed millions of attempts from hosts of dial-in providers, which usually strongly indicates spam-bot afflicted private hosts.
I have added theses networks to my /etc/hosts.deny file, with the effect that the number of connections to the server was reduced almost immediately. While one might ask for the wisdom of blocking whole networks, think about this: by what necessity does a private dial-in host have to have its own smtp-server attempting to connect to your smtp-server? Usually, a private person can use the mail-server of his/her provider,  and that one won't be blocked, because I am blocking the dial-in-subnets specifically.
Here is the current (March 29, 2009) list of blocked networks:
sendmail: .adsl-1.sezampro.yu


After changing the configuration using the above described possibilities, the load of the sever decreased enormously, and there are far less sendmail-processes now running at the same time, thus binding far less resources. DDoS-spam-attacks are still not impossible, but they will have a harder time to get the machine down now.


sendmail limit connections

Whenever a sendmail receives SMTP request and connection from a remote host, it spawn’s a new copy of itself to deal with incoming mail messages. This approach makes it useful for sendmail to process multiple incoming mails simultaneously. However, the nature of linux application always comes with a trade off thing. This issue of spawning another sendmail daemon child would consume another portion of system and thus could hogged down your email server.
Considering the openness of email address nature, an email bomber could launch a malicious flood attack to an email server with bunch of emails for delivery that could easily be done automatically using automated software. This scenario could cause sendmail denial of service.
The below sendmail directives option could restrict and limit the number of sendmail children to spawn, limit the number of connections to receive per second, restrict sendmail process incoming header size, and lastly, limit the maximum acceptable size of one message. By setting this sendmail directive, the effectiveness of such denial of sevice attack to sendmail could be limited and/or delayed.

Sendmail Directives

This is number of connections sendmail server can receive per second. By default rpm installation, Sendmail does not limit the number of connections per second it can receive. If the value configured here is set and reached, any further connections are then delayed.

This is the number of child processes that can be spawned by the server, or refers to sendmail children. By default rpm installation, Sendmail does not define a limit to the number of spawned child daemon. If this limit is reached, further connections are also delayed.

This is the minimum number of free blocks is available for sendmail server when accepting mails. The default is 100 blocks. Once the value specified with this sendmail directive is reached, sendmail is instructed to stop accepting mails.

This is the maximum acceptable size (in bytes) of an email’s message header. Any email message headers that reached this limit would not be accepted and denied from being queued for delivery.


Limiting the the maximum acceptable size (in bytes) of an email message could also limit the possible attachment bomb attack against sendmail server. Implementing a maximum message size could set a limit to email message maximum size. An example entry would likely be\

This is the number of recipients per message. Limiting the maximum recipient one email message could have could restrict any email bomber trying to send an email to multiple recipients.
A sample entry of the above directive would be
define(`confMAX_DAEMON_CHILDREN’, 10)
define(`confMAX_HEADERS_LENGTH’, `32768′)
define(`confMAX_MESSAGE_SIZE’, 10000000)
define(`confMAXRCPTSPERMESSAGE’, `30′)
To implement the above sendmail directive, simply backup and modify your /etc/mail/ and recompile it again. For the changes to take effect, simply restart your sendmail service like so
# vi /etc/mail/
# m4 /etc/mail/ > /etc/mail/
# service sendmail restart
All is done.

Help My Mail Server Is Being DoSsed

The recent flood of spam, viruses and trojaned Windows machines has brought a dramatic increase in attacks on e-mail servers, including (but not limited to) denial-of-service (DoS) attacks, distributed denial-of-service (DDoS) attacks, Joe Jobs, dictionary attacks, slamming, and other assorted nuisances. This document describes some sendmail facilities–many of them new in version 8.13–that can help foil these attacks.
1. Limiting the Rate of Incoming Connections

1.1. The ratecontrol Feature

Version 8.13.0 of Sendmail introduced the ratecontrol feature, which uses the access database to limit the number of incoming connections a single server, group of servers or all unspecified (”wildcard”) servers can make to your server per minute. When an incoming server exceeds those limits, sendmail returns a temporary failure, causing the (standards compliant) incoming server to queue the mail and retry delivery later.
The “per minute” connection rate window is configurable. If, as you configure the ratecontrol feature, you decide that you want to change the connection rate window, add the following to your sendmail macro configuration (mc) file:
where secs is an integer that specifies the number of seconds you want in the window.
The ratecontrol feature declaration must follow the access db declaration (somewhere) in your sendmail mc file:
FEATURE(`access_db')dnl FEATURE(`ratecontrol', ,`terminate')dnl
If your existing configuration uses the delay_checks feature to whitelist recipients (for example), you must tell the ratecontrol feature not to delay those checks, by specifying “nodelay” as the second argument to the ratecontrol feature:
FEATURE(`access_db')dnl FEATURE(`delay_checks',`friend')dnl FEATURE(`ratecontrol', `nodelay',`terminate')dnl
The delay_checks feature is outside the scope of this document. See the cf/README file (in the sendmail distribution) or section 7.5.6 of the Bat Book.
The third argument (”terminate”, above) causes sendmail to immediately terminate an excessive incoming connection with a 421 status code. If you omit the third argument (declaring the feature something like FEATURE(`ratecontrol’), for example), sendmail will issue a 452 status, which leaves it up to the incoming mail server to issue an SMTP QUIT. If you omit “terminate”, an unfriendly incoming server can continue to issue (failing) SMTP commands–probably not what you want to have happen.
After modifying your mc file, generate and install a new file and restart sendmail.
The method of generating a varies from platform to platform. This link describes how to do it on FreeBSD. To restart sendmail on FreeBSD, enter the commands:
# cd /etc/mail # make restart
After modifying and restarting sendmail, edit /etc/mail/access (or wherever your distribution stores the source for the access database) and add entries something like the following:
ClientRate: 2 ClientRate: 0 ClientRate: 10 ClientRate:10.3.4 2
The above limits incoming server to two connections per minute, localhost is unlimited, each individual host in the subnet 10.3.4 can have at most 2 connections per minute, and any other host can have up to 10 connections per minute
Note about mail clients that relay mail through the server. Remote e-mail clients that (legitimately, using SMTP AUTH, for example) relay mail through the server are subject to the ClientRate settings in access db. In the above example, if the “wildcard” is changed from 10 to 1, and a client not matched by other ClientRate: entries attempts to send more than one e-mail in a minute’s time, all but the first e-mail will fail. … So be aware of your clients that use your server as their outgoing SMTP server as you configure ratecontrol in the access database.
After editing the access source file, regenerate access db with something like:
# cd /etc/mail # makemap hash access < access
On FreeBSD, you can generate access.db with the simpler
# cd /etc/mail # make

1.2. The Connection Rate Throttle

There are times when the ratecontrol feature might not give adequate protection. In a distributed denial-of-service (DDoS) attack, for example, an “army” of attacking servers individually limited to as few as 1 connection per minute might collectively overwhelm sendmail. The ConnectionRateThrottle causes sendmail to accept no more than a specified number of new connections per second.
If (for example) you set the ConnectionRateThrottle at 3, and 9 incoming connections arrive simultaneously, sendmail will handle the first three during the first second, the second three during the second second, and the third three during the third second. Connections that exceed the specified ConnectionRateThrottle are forced to wait their turn.
To set the ConnectionRateThrottle, insert the following in your sendmail mc file:
define(`confCONNECTION_RATE_THROTTLE', `num')dnl
where num is a positive integer.
If you specify a value or 0 or a negative integer–or omit the line completely–then sendmail will not enforce a connection rate throttle.

2. Limiting Simultaneous Connections with the conncontrol Feature
Limiting the rate of incoming connections alone might not protect your server from a DoS attack. In order to ensure that the “mail will go through,” sendmail’s default timeout values for each phase of a mail transaction are very generous (in keeping with values recommended by Internet standards documents). As a result, it is conceivable (in fact not uncommon) for a single envelope delivery to take well over an hour. Unless prevented from doing so, malevolent attackers can slowly accumulate connections to your server, causing it to exhaust its resources.
Fortunately, sendmail 8.13 introduced the conncontrol feature to limit the number of simultaneous connections a single server can maintain with your server.
If you’ve read and understood the ratecontrol feature (above), conncontrol will seem familiar. Begin by editing your sendmail mc file. The conncontrol feature declaration must com after the access db declaration in the mc file:
FEATURE(`access_db')dnl FEATURE(`conncontrol', ,`terminate')dnl
If your existing configuration uses the delay_checks feature to whitelist recipients (for example), you must tell the conncontrol feature not to delay those checks, by specifying “nodelay” as the second argument to the conncontrol feature:
FEATURE(`access_db')dnl FEATURE(`delay_checks',`friend')dnl FEATURE(`conncontrol', `nodelay',`terminate')dnl
The delay_checks feature is outside the scope of this document. See the cf/README file (in the sendmail distribution) or section 7.5.6 of the Bat Book.
As with ratecontrol, conncontrol’s third (”terminate”) argument causes sendmail to immediately terminate an excessive incoming connection with a 421 status code. If you omit the third argument (declaring the feature something like FEATURE(`conncontrol’), for example), sendmail will issue a 452 status, which leaves it up to the incoming mail server to issue an SMTP QUIT and break the connection(!). If you omit “terminate”, an unfriendly incoming server can continue to issue (failing) SMTP commands–holding the connection open!! (definitely not what you want to happen).
After modifying and restarting sendmail, add ClientConn: entries to the access db. For example:
ClientConn: 2 ClientConn: 0 ClientConn: 10 ClientConn:10.3.4 2
will limit incoming server to a maximum of two simultaneous connections, allow localhost to have unlimited connections, limit each host in the subnet 10.3.4 to two simultaneous connections, and any other individual host can have up to 10 simultaneous connections.
After adding ClientConn-tagged entries to access source file, regenerate access.db.

3. Thwarting Dictionary Attacks
In a typical dictionary attack, the attacking server sends a single message to a list of well-known or “promising” e-mail addresses at the target domain. After exchanging greetings and issuing the MAIL FROM: command to identify the (probably spurious) sender, the attacker issues a series of RCPT TO: commands, something like:
RCPT TO:<> RCPT TO:<> RCPT TO:<> RCPT TO:<> RCPT TO:<> RCPT TO:<> [more recipients ...] RCPT TO:<>
After identifying the recipients, the attacker continues the transmission by issuing the DATA command and sending the message headers and body.
Because of the guessing involved, dictionary attacks have sometimes been called Rumpelstiltskin attacks (named after the dwarf in the Grimms’ fairy tale who tells a woman he will not hold her to a promise if she can guess his name–Rumpelstiltskin).

3.1. Limiting the Number of Recipients per Message
At least one major ISP makes dictionary attacks more difficult by limiting the number of RCPT TO: commands allowed in a single incoming “envelope”. You can configure sendmail to impose the limitation by adding the following to your sendmail mc file:
define(`confMAX_RCPTS_PER_MESSAGE', `25')dnl
(substituting your own preferred maximum for the `25′ used above).
If (for example) the maximun is configured at 25 and the sender specifies a 26th recipient, sendmail will respond with
452 4.5.3 Too many recipients
If the sending server ignores the 452 response and continues issuing RCPT TO: commands, the receiving sendmail will send the same response to all remaining RCPT TO: commands. Then, when the sender issues the DATA command and completes the message envelope, the message will be delivered only to the first 10 (in this example) recipients.
The idea here is that RFC-compliant sending servers will detect the temporary failure(s) and queue the message for later delivery to the temporarily rejected recipients. A spammer or trojan will likely never retry the failed recipients.
Caution: Remote mail users that specify your sendmail server as their outgoing SMTP server are subject to the same constraints as incoming SMTP servers that connect to your server: If you limit the maximum recipients to 25 and a user sends a message to 26 users via your sendmail server, sendmail will respond with a (possibly hard-to-understand) temporary failure. You should therefore set the number high enough to accomodate legitimate client senders but low enough to discourage dictionary attackers.

3.2. Reacting to “Bad” Recipients
“Bad” recipients are those that your server doesn’t receive mail for–addresses not listed in virtusertable, aliases, /etc/passwd (or whatever mechanism(s) your server uses for identifying legitimate recipients).
I have heard of some sites that block no mail, but instead route mail for unknown recipients to /dev/null. This seems like a particularly bad idea to me. Not only are spammers and attackers lead to believe that all their mail is being delivered, but legitimate senders that mistype the recipient’s address receive no notification that the e-mail they sent was delivered to the “bit bucket.”
Use of the option described in this section requires that mail to unknown users be rejected during the SMTP conversation.
You can tell sendmail to respond more slowly to the incoming server after a specified number of unknown recipient addresses have been specified, by inserting the following line in your mc file:
If, for example, you specify num as 2, sendmail will respond normally to the first RCPT TO: commands. As soon as 2 unknown users have been identified, sendmail will record the following in the mail log:
Possible SMTP RCPT flood, throttling.
and sleep for one second (hardcoded) after all subsequent RCPT TO: commands.

4. Blocking Slammers with the greet_pause Feature
When an incoming mail server connects to yours, it is supposed to wait for your server’s welcome greeting before commencing the SMTP conversation with the EHLO or HELO command. It is common for attackers and poorly written trojans and viruses to send the EHLO/HELO command without waiting for your server’s greeting. The practice of not waiting for the greeting is called “slamming.”
There are rumors that at least one large ISP’s servers practice slamming. If you identify such an ISP, you can whitelist it in your access database (See below.)
The greet_pause feature (new in sendmail 8.13) causes sendmail to pause for a specified period of time at the beginning of the connection, before returning its greeting. If the incoming server issues the EHLO/HELO command during the pause, then sendmail will issue something like the following
554 not accepting messages
instead of the standard greeting
220 ESMTP Sendmail 8.13.3/8.13.3; Thu, 3 Mar 2005 18:16:23 GMT
and an entry is written to the mail log:
rejecting commands from [] due to pre-greeting traffic
The following line (inserted in sendmail’s mc file) will cause sendmail to pause a default 700 milliseconds before issuing its greeting:
FEATURE(`greet_pause', `700')dnl
If you omit the second argument (using just FEATURE(`greet_pause’) in the mc file), or if you want to override the default, then you need to add entries to the access database. In the following example 5000 0 GreetPause: 0 GreetPause:192.168.120 5000
the first and last lines will force a greet pause of 5 seconds (5000 milliseconds); the second and third lines tell sendmail not to wait at all.
All other connecting hosts (or domains or addresses or networks) will have to wait the default time specified as the second argument to the FEATURE macro. (If the second argument was omitted from the FEATURE macro, hosts that don’t match any access db GreetPause-tagged lines will not have to wait).
RFC 2821 specifies 5 minutes as the maximum timeout for the initial connection greeting. Therefore, if you specify a time longer than 300000 milliseconds (i.e. 5 minutes), sendmail will not wait longer than 5 minutes, to maintain RFC compliance.

5. Other Ways to Protect Your sendmail Server
Strictly speaking, this section is beyond the scope of this document. As you think about protecting your sendmail server, however, you will likely want to consider the following:
* DNS Blacklists. My DNSBL: Configuring Sendmail for DNS-Based Blacklisting is one source of information on the subject. (There are many other good sources.)
Hint: My current favorite DNS blacklist–if I were forced to use only one–is
* Greylisting. The greylisting milter I use on my main mail server is responsible for the rejection of more unwanted mail than any other other filtering mechanism. See or Google to find more about the greylisting milters available.
* Virus Scanning. I use a combination of ClamAV and Procmail to scan all incoming mail for viruses.
I scan (for viruses and spam) only the mail that gets past my DoS-blocking measures, DNS blacklists, access db entries and greylisting milter. This cuts down on server load required by the relatively CPU-intensive content scanning.
* Spam Filtering. There are many choices for filtering spam. My current favorite is SpamAssassin and (good old) Procmail rules. With SpamAssassin you can scan
1. During the SMTP conversation. The MIMEDefang and spamass-milter milters are popular for scanning at this stage of delivery.
2. During local delivery. Using procmail to scan during local delivery is probably the most common way of using SpamAssassin.
3. When mail is retrieved from one’s mailbox, using a POP proxy.

6. Afterword
I have personally used most of the techniques discussed above, so I know that they work–on my server, at least. If you decide to implement them on your server, be careful to avoid typos. You will probably want to keep a backup copy of your old configuration file in case a rollback is necessary. After configuring any of the features and options described above, be sure to send test messages and monitor the mail logs to make sure that sendmail behaves as expected.




* High Volume Mail

For high volume mail it is necessary to be able to control the load
on the system.  Therefore the 'queue' delivery mode should be used,
and all options related to number of processes and the load should
be set to reasonable values.  It is important not to accept mail
faster than it can be delivered otherwise the system will be
overwhelmed.  Hence RefuseLA should be lower than QueueLA, the number
of daemon children should probably be lower than the number of queue
runnners (MaxChildren vs. MaxQueueChildren).  DelayLA is a new option
in 8.12 which allows delaying connections instead of rejecting them.
This may result in a smoother load distribution depending on how
the mails are submitted to sendmail. 
There are two important settings in the file:

confQUEUE_LA            QueueLA         [varies] Load average at which
                                         queue-only function kicks in.
                                         Default values is (8 * numproc)
                                         where numproc is the number of
                                         processors online (if that can be
confREFUSE_LA           RefuseLA        [varies] Load average at which
                                         incoming SMTP connections are
                                         refused.  Default values is (12 *
                                         numproc) where numproc is the
                                         number of processors online (if
                                         that can be determined).

On a dedicated mail server, you want to set QueueLA *higher* than
RefuseLA - the other way around will cause your queue to explode when
the load is high (assuming you reach QueueLA of course), you accept lots
of new messages but can't get rid of them as the load average hovers
around RefuseLA - the many messages in the queue drive up disk I/O and
thereby the load average.

The default of QueueLA lower than RefuseLA only makes sense for a server
that does other things besides mail, where you want to reduce the impact
of the mail load on the other functions by queuing instead of delivering
when the load is high.

Your original file may state:

dnl define(`confQUEUE_LA', `12')dnl
dnl define(`confREFUSE_LA', `18')dnl

I normaly change it to for a web server, that uses sendmail for formmail scripts etc:

dnl define(`confQUEUE_LA', `3')dnl
dnl define(`confREFUSE_LA', `6')dnl

You will need to remake the config files before restarting the server so the changes 
take effect, to do this run the following command:

make -C /etc/mail

With Redhat make sure you have the "sendmail-cf" packaged installed to do this.






Recipe 10.17 Denying DoS Attacks


You want to limit the possibility that a denial-of-service (DoS) attack aimed at sendmail will cripple other services offered by the sendmail server.


Add confCONNECTION_RATE_THROTTLE and confMAX_DAEMON_CHILDREN defines to the sendmail configuration to set limits on how fast sendmail accepts mail connections and how many concurrent connections are accepted. Here are examples:
dnl Accept up to 10 connections per second
define(`confCONNECTION_RATE_THROTTLE', `10')
dnl Allow no more than 200 concurrent connections
define(`confMAX_DAEMON_CHILDREN', `200')
Following the guidance in Recipe 1.8, build the file, copy it to /etc/mail/, and restart sendmail.


A denial-of-service (DoS) attack can overwhelm a server and effectively block useful inbound and outbound mail. For example, a mail-bombing attack delivers so much mail so rapidly to a single target that the target system is unable to keep up with the workload. Worse yet, sendmail can becomes so busy handling the attack, it takes all of the system's resources and thus prevents other useful services from running. This recipe uses two configuration commands that lessen the impact that a DoS attack aimed at sendmail has on its other services.
The confMAX_DAEMON_CHILDREN define sets the maximum number of sendmail processes that this sendmail daemon can run simultaneously. Every system has an upper limit on the number of processes it can effectively handle. A variety of DoS attacks are designed to overwhelm a system by launching so many mail processes that the system is no longer able to do any productive work. The confMAX_DAEMON_CHILDREN define protects the operating system from this type of attack.
By default, sendmail sets no upper limit on the number of child processes that can be launched to process mail. This recipe limits sendmail to 200 children so that sendmail accepts mail only when fewer than 200 children are running. This is not a recommended value; it is just an example. Study the actual usage patterns of your system before you select a value, and then set the value at least 50 percent above the observed value to allow for usage spikes.
confMAX_DAEMON_CHILDREN is most useful for systems that do more than just provide sendmail service. A good example is a general purpose server that has a relatively light email workload and that provides many other important user services. The confMAX_DAEMON_CHILDREN define protects the other services from runaway sendmail daemons. It makes sure that the user can still open an X window or start an editor even if the sendmail server is under attack. confMAX_DAEMON_CHILDREN provides this protection for the other services at the cost of making it simple for the attacker to shut down mail service. Simply sending enough mail to exceed the maximum number of children shuts down the SMTP port. Use this define carefully.
The confCONNECTION_RATE_THROTTLE define sets the maximum number of SMTP connections that are permitted for any one-second period. Like confMAX_DAEMON_CHILDREN, the confCONNECTION_RATE_THROTTLE command protects the server from being overwhelmed by email. Neither of these defines protects sendmail from attack; they both protect the system when sendmail is under attack.
By default, no limit is set on the rate at which new SMTP connections are accepted. sendmail handles connections as fast as they arrive. This recipe limits sendmail to 10 new connections per second so that sendmail rejects any network connections after the first 10 in any 1-second period. As before, this is not a recommended value; it is just an example. Study the actual usage patterns of your system before you select a value, and then set the value at least 50 percent above the observed value to allow for usage spikes.
Often, confCONNECTION_RATE_THROTTLE and confMAX_DAEMON_CHILDREN are used together. confMAX_DAEMON_CHILDREN sets an upper limit on the number of simultaneous email connections and confCONNECTION_RATE_THROTTLE sets the rate at which the system climbs to that maximum. An attacker can easily overrun the connection rate you set in order to deny mail access. However, setting the rate protects the other services on the system so that the system remains operational to help you deal with the attack.
It is impossible to completely protect sendmail from denial-of-service attacks. Limiting the amount of damage done by the attack is the real purpose of the defines used in this recipe, which is really meant to protect the system as a whole.

See Also

The sendmail book covers the confCONNECTION_RATE_THROTTLE define in Section 24.9.21 and the confMAX_DAEMON_CHILDREN define in Section 24.9.60.


Sunday, October 3, 2010

Recursively List all directories and files

SkyHi @ Sunday, October 03, 2010
Suppose the directory structure on the file system is like this:


The output from the script must be like:





Hi could you guys tell me how to keep the output of "find . -type d" and "find . -type f" into another file.. thanks a lot guys


Bash/Shell Into a file


find ./ -type d  > somefile.txt


find ./ -type f  > somefile.txt

In windows, to list only directories:
dir /ad /b /s

to list all files (and no directories)

dir /a-d /b /s 

redirect the output to a file by adding (without quotes)

> filename.txt'