Command Center

We have recently upgraded our mail servers running Debian Etch to Lenny and noticing that Dovecot 1.2 has been backported to Lenny backports, we decided to upgrade to Dovecot to take advantage of its fixes, improved security and quota settings.
Setting up quotas and warnings for any Dovecot above 1.0 is very simple.
1) First step is to enable the quota plugin for all protocols enabled in Dovecot ( pop3, imap and lda if you use it). This is done by just adding the line:

mail_plugins = quota

and for the IMAP protocol:

mail_plugins = quota imap_quota

2) Setup the plugin:

# Quota plugin settings plugin { quota = maildir:User quota #Quota limit is 1GB quota_rule = *:storage=1G #We add 100Mb more for Trash quota_rule2 = Trash:storage=100M #We set up warnings at 75% and 90% quota_warning = storage=75%% /opt/mail.sh 75 %u quota_warning2 = storage=90%% /opt/mail.sh 90 %u }

We can add as many rules and warnings as we want, just by adding and incrementing a number at the end of the warning or rule ( like above: quota_rule, quota_rule2 and so on ).
The first variable, “quota”, is the quota root and is a name that is sent to the IMAP client and can be anything you want.
The second variable is the quota itself and you can set it up using several limits: storage,bytes,messages,backend and ignore (this ignores quota for a specific mailbox). All variables support besides numbers the b/k/M/G/T/% suffixes. For example: 100M, 1G, 30% and so on.
The third variable sets up the quota warning. Using the limit you set up above in the rule you set up the warning at what level you want, by using a fixed number or percent. In the above example we used percent. In this case you need the double % ( %% ) so it can be escaped by dovecot. After the warning limit is set, in the same line, separated by space, is the command to run when that limit is reached. In this case it is a custom script that takes two command line arguments: the first one is the percent and the second one is the user that has reached the limit and to which to send the warning.
We set up the second argument (the user) as dovecot’s %u variable which is set in the ldap configuration, in the filter string. You can make this anything you want, taking the user dynamically through whichever system you have configured as your user database and whichever variable you have set up as you user.
For example, in my dovecot-ldap.conf this is what we have:

pass_filter = (&(objectClass=person)(userPrincipalName=%u))

and this is where I take my %u from.
And finally this is the script I use to send the warning. It uses the default sendmail binary, a simple text file and the two arguments taken from the command line:

#!/bin/bash PERCENT=$1 USER=$2 echo “From: postmaster@domain.org To: $USER Subject: Your email quota is $PERCENT% full Content-Type: text/plain; charset=”UTF-8″ This is an automatic message to warn that your mailbox is now $PERCENT% full.” > /tmp/quota.email.$USER cat /tmp/quota.email.$USER | /usr/sbin/sendmail -f postmaster@domain.org $USER rm /tmp/quota.email.$USER

That’s it! Whenever the user reaches its defined quotas he / she will receive a warning email. This quota are global, any user will have the same quota. If you want per-user quotas and / or soft-quotas check out my later edit below!

LATER EDIT:

If you want to make your quotas non-enforcing, that means the user will still receive the warnings but if he or she reaches the quota, their e-mails will not be blocked and they will still be able to receive, you must modify your “quota” variable in the plugin’s settings (the quota root). Make your settings look like this:

# Quota plugin settings plugin { quota = dict:user::noenforcing:file:/opt/data/mailboxes/%u/Maildir/dovecot-quota #Quota limit is 1GB quota_rule = *:storage=1G #We add 100Mb more for Trash quota_rule2 = Trash:storage=100M #We set up warnings at 75% and 90% quota_warning = storage=75%% /opt/mail.sh 75 %u quota_warning2 = storage=90%% /opt/mail.sh 90 %u }

Notice the only thing changed is the first variable: “quota”.
Also, if you want to use LDAP attributes to change your users’ quotas on a per-user basis, you need to do the following:
1) Rename your dovecot-ldap.conf file to dovecot-passdb-ldap.conf :

mv /etc/dovecot/dovecot-ldap.conf /etc/dovecot/dovecot-passdb-ldap.conf

2) Create a symlink to dovecot-passdb-ldap.conf called dovecot-userdb-ldap.conf :

ln -s /etc/dovecot/dovecot-passdb-ldap.conf /etc/dovecot/dovecot-userdb-ldap.conf

3) Modify your dovecot.conf file to point to these files as DBs for users and passwords:

passdb ldap { args = /etc/dovecot/dovecot-passdb-ldap.conf } userdb ldap { args = /etc/dovecot/dovecot-userdb-ldap.conf }

4) And finally make sure your ” /etc/dovecot/dovecot-passdb-ldap.conf ” file looks like this:

hosts = dn = dnpass = auth_bind = yes ldap_version = 3 base = dc=example, dc=org pass_filter = (&(objectClass=person)(userPrincipalName=%u)) user_filter = (&(objectClass=person)(userPrincipalName=%u)) user_attrs = otherPager=quota_rule=*:bytes=%$, userPrincipalName=home=/opt/data/mailboxes/%u

Notice we use the ” userPrincipalName ” as the attribute for username in dovecot (the username the users will also use to authenticate to dovecot). If you want, you can change this to whatever you want, like sAMAccountName.
Also, if you look carefully, notice that we used the ” otherPager ” attribute from LDAP as the attribute for user quota. We just modify this attribute for whatever we want to override the default quota for a certain user. For example we want to modify for user jon.doe, we just put in that attribute: ” 2G ” and the user will have 2 gigabytes. You can use any existing attribute that is not used and will not be or you can add your own to all the users in the LDAP tree.


REFERENCES
http://www.hostatic.ro/2009/11/11/dovecot-1-2-with-quotas-and-quota-warnings/
http://wiki.dovecot.org/Quota/Dict

Greylisting (or graylisting) is a method of defending e-mail users against spam. A mail transfer agent (MTA) using greylisting will "temporarily reject" any email from a sender it does not recognize. If the mail is legitimate the originating server will, after a delay, try again and, if sufficient time has elapsed, the email will be accepted. If the mail is from a spam sender, sending to many thousands of email addresses, it will probably not be retried.
Contents

* 1 How it works
* 2 Why it works
* 3 Advantages
* 4 Disadvantages
* 5 See also
* 6 References
* 7 External links

[edit] How it works

Typically, a server employing greylisting will record the three pieces of data known as a "triplet" for each incoming mail message:

* The IP address of the connecting host
* The envelope sender address
* The envelope recipient address(es)


The temporary rejection can be issued at different stages of the SMTP dialogue, allowing an implementation to store more or less data about the incoming message. The trade-off is more work and bandwidth for more exact matching of retries with original messages[1]. Rejecting a message after its content has been received allows the server to store a choice of headers and/or a hash of the message body.

In practice, most greylisting systems do not require an exact match on the IP address and the sender address. Because large senders often have a pool of machines that can send (and resend) email, IP addresses that have the most-significant 24 bits (/24) the same are treated as equivalent, or in some cases SPF records are used to determine the sending pool. Similarly, some e-mail systems use unique per-message return-paths, for example variable envelope return path for mailing lists, Sender Rewriting Scheme for forwarded e-mail, Bounce Address Tag Validation for backscatter protection , etc. If an exact match on the sender address is required, every e-mail from such systems will be delayed. Some greylisting systems try to avoid this delay by eliminating the variable parts of the VERP by using only the sender domain and the beginning of the local-part of the sender address.
[edit] Why it works

Greylisting is effective because many mass email tools used by spammers will not bother to retry a failed delivery, so the spam is never delivered. A spam sender may retry with a different sender, and possibly a different message, because it has a queue of victims rather than the proper queue of messages that regular mail servers maintain.

In addition, if a spammer does retry a delivery after the waiting period has expired, any one of a number of automated spamtraps will have had a good chance of identifying the spam source and listing both the source and the particular message in their databases. Thus, these subsequent attempts are more likely to be detected as spam by other mechanisms than they were before the greylisting delay.
[edit] Advantages

The main advantage from the users' point of view is that greylisting requires no additional configuration from their end. If the server utilizing greylisting is configured appropriately, the end user will only notice a delay on the first message from a given sender, so long as the sending email server is identified as belonging to the same whitelisted group as earlier messages. If mail from the same sender is repeatedly greylisted it may be worth contacting the mail system administrator with detailed headers of delayed mail.

From a mail administrator's point of view the benefit is twofold. Greylisting takes minimal configuration to get up and running with occasional modifications of any local whitelists. The second benefit is that rejecting email with a temporary 451 error (actual error code is implementation dependent) is very cheap in system resources. Most spam filtering tools are very intensive users of CPU and memory. By stopping spam before it hits filtering processes, far fewer system resources are used. This allows more layers of spam filtering or higher throughput since greylisting can easily be configured as a first line of defense with SpamAssassin etc. handling messages that go through.

Some greylisting packages support a SQL backend which allows for a distributed multiple-server frontend to be deployed with the same greylisting data on all frontends.
[edit] Disadvantages

The biggest disadvantage of greylisting is that it destroys the near-instantaneous nature of email that users have come to expect. Mail from unrecognised senders is delayed by typically about 15 minutes, and up to four hours. A customer of a greylisting ISP can not always rely on getting every email in a pre-determined amount of time.

However, the original specification for email states that it is not a guaranteed delivery mechanism and not an instantaneous delivery mechanism. This means that greylisting is a perfectly legitimate process and does not break any protocols or rules.

If mail from a particular frequent sender is sent from any of several mail servers, mail may be delayed unless the greylisting server recognises the different servers as belonging to the same whitelisted group.

On a technical level, some SMTP clients and SMTP servers acting as clients may interpret the temporary rejection as a permanent failure. Old clients conforming only to the obsolete specification (RFC 821) and ignoring its recommendations may give up on delivery after the first failed attempt—RFC 821 states that clients "should" retry messages rather than using the word "must". RFC 2119 dictates that "should" means recommended and to ignore at your own risk, and it is a violation of the current SMTP standard for the client to fail to retry. The current SMTP specification (RFC 5321) clearly states that "the SMTP client retains responsibility for delivery of that message" (section 4.2.5) and "mail that cannot be transmitted immediately MUST be queued and periodically retried by the sender." (section 4.5.4.1).

This problem can affect SMTP clients in unexpected ways. Most MTAs will queue and retry messages, but a small number do not.[2] A similar concern exists for applications which act as SMTP clients and fail to incorporate any form of queueing for deferred SMTP mail. This can be mitigated on the sending side by configuring the application to use a local SMTP server as an outbound queue, instead of attempting direct delivery. For the server operator who uses greylisting, clients which are known to fail on temporary errors can be supported by whitelisting or exception lists.

Some MTAs, upon encountering the temporary failure message from a greylisting server on the first attempt, will send a warning message back to the original sender of the message.[2] The warning message is not a bounce message, but it is often formatted similarly to one and reads like one. This practice often causes the sender to believe that the message has not been delivered, when in fact the message will be delivered successfully at a later time.


Greylisting delays much of the mail from non-whitelisted mail servers—not just spam—until typical patterns of communication are recorded by the greylisting system. For best results, whitelisting should be used extensively. A static list of public servers worth being whitelisted can be found in the official repository.[clarification needed]

Also, legitimate mail might not get delivered if the retry does not arrive within the time window the greylisting software uses, or if the retry comes from a different IP address than the original attempt. When the source of an email is a server farm or goes out through an anti-spam mail relay service, it is likely that on the retry a server other than the original server will make the next attempt. Since the IP addresses will be different, the recipient's server will fail to recognize that the two attempts are related and refuse the latest connection as well. This can continue until the message ages out of the queue if the number of servers is large enough. Such server farming techniques can be construed as breaking RFCs detailed above since the original sending machine has absolved itself of the responsibility of mail delivery by tossing it back into the pool, which breaks the state of the mail delivery process. This problem can partially be bypassed by identifying and whitelisting such server farms in advance. However, it is not possible on a distributed network the size of the Internet to maintain a complete list of all such server farms.[3]

Greylisting can be a particular nuisance with websites that require an account to be created and the email address confirmed before they can be used. If the sending MTA of the site is poorly configured, greylisting may delay the initial email containing the signup confirmation link, thus introducing a waiting period even though the actual website may have attempted to send out the email confirmation code immediately. Almost all stock-configured Sendmail MTAs (sendmail being the most widely deployed MTA on the internet) will retry after a few minutes, leading to typical delays of under 10 minutes in most cases (still dependent on the greylisting configuration).

Greylisting is particularly effective in many cases at weeding out misconfigured MTAs, and is gaining in popularity as a very effective anti-spam tool. It is likely that those MTAs that do not correctly handle greylisting will become less numerous as greylisting spreads.


Also, if certain details of the sending vary and the receiving MTA is not programmed to notice this, a message may be greylisted eternally and never delivered.
[edit] See also

* Nolisting
* Bandwidth throttling
* Tarpit (networking)

[edit] References

1. ^ John Levine (2005). "Experiences with Greylisting". Second Conference on Email and Anti-Spam. http://www.ceas.cc/papers-2005/120.pdf. Retrieved 2009-04-18.
2. ^ a b Evan Harris (21 August 2003). "The Next Step in the Spam Control War: Greylisting". PureMagic Software. http://projects.puremagic.com/greylisting/whitepaper.html. Retrieved 2008-01-09.
3. ^ "Filtering Spam: Combined techniques give best results". Shamrock Software GmbH. December 2007. http://www.shamrock-software.eu/spam.htm#grey. Retrieved 2008-01-09.

[edit] External links

* Greylisting.org: Repository of greylist info
* A greylisting whitepaper by Evan Harris
* A greylisting implementation for netqmail
* Microsoft Exchange Greylisting Problems - Newsgroup Article

Retrieved from "http://en.wikipedia.org/wiki/Greylisting"
Categories: Spam filtering

Problem:
I've come across a few of my sites on the server at my work that are showing up with Black Diamonds with question marks in them. It seems like just recently have we started having this problem, sites that have been up for a while now without these problems are now having them. What could be the cause of this so that we may look into it?



#
# Specify a default charset for all content served; this enables
# interpretation of all content as UTF-8 by default.  To use the
# default browser choice (ISO-8859-1), or to allow the META tags
# in HTML content to override this choice, comment out this
# directive:
#
###comment out for geohalifax09 conflict
##AddDefaultCharset UTF-8
AddDefaultCharset UTF-8

Solution 1:

AddDefaultCharset iso-8859-1


Solution 2:
k thanks for the replies. I am using Firefox. I looked at the default Character coding is and it was Unicode UTF-8.

I switched to ISO-8859-1, Windows-1251, Windows-1252, and US-ASCII and it worked. The ISO-8859-15 and the default Unicode UTF-8 didnt work. So in the database, you change the character set how? Seems like it would be under Collation. I am using PHPMyAdmin

Edit: Ok nevermind then, I just had to change

to

Here is a checklist:
---------------------
1. TEXT FILES, SAVE AS: If you are uploading any pages to the server, you should make sure the page content is SAVED AS (option) Unicode (UTF-8, no BOM), rather than the default (probably Latin1). The UTF-8 BOM causes a lot of problems at this time (especially in PHP); so output _without_ the BOM.

2. DYNAMIC SCRIPT: script output:

PERL: Instead of the requisite line,
print "Content-type: text/html\r\n\r\n";
use the following line:
print qq|Content-type: text/html; charset="utf-8"\r\n\r\n|;

PHP: Output this header before content output:
header("Content-type: text/html; charset='utf-8'");

3. HTML: On any html pages output by a script, or by a WYSIWYG editor, or rolled by hand, add the meta tag "charset" identifier before opening the title tag. In fact, it should be the first meta tag after the head tag is opened. This method is suppose to force the browser to reparse everything if it wasn't parsing in that character set in the first place. (However, I can't validate that.) In other words, do this:





4. XML/XHTML: With XML pages, declare the "charset" in the opening line, like:

In xhtml documents, I still add in the meta tag from tip #3, above, for good measure.

5. FORM: To assist with properly encoded form input from your guests and members, use the "accept-charset" form attribute, like this:

6. DATABASE

: Finally, you may want to set your database default character set to utf-8 if you are storing utf-8 input.

Fran Corpier 



Solution 3:
don't copy and paste from Words

REFERENCES
http://www.webhostingtalk.com/showthread.php?t=621254
http://www.webhostingtalk.com/showthread.php?t=622439
http://www.phpwact.org/php/i18n/charsets

In my professional IT career I've rarely experienced greater rage and frustration than when realizing that an important file or directory has been inadvertently deleted. The feeling is exponentially more nauseating when the data is related to an important client or other mission-critical project, particularly when that data cannot be easily recreated. Unfortunately, such data is plentiful in corporate environments, whether its sales records, carefully managed customer contact data, or archived support forum messages.
Whether you host your website and database internally or you use a hosting provider, it is of paramount concern that you institute a rigorous backup policy in order to avoid such disastrous consequences. If you're using MySQL, several easily implementable options are at your disposal. In this article I'll introduce you to one such option by showing you how to backup your database using MySQL's native mysqldump client.

Introducing mysqldump

MySQL is bundled with a number of command-line utilities, among them mysql (a general purpose utility for interacting with MySQL in many ways, including database schemas and data, users, and configuration variables), mysqladmin (useful for carrying out various administrative tasks), and mysqldump. The mysqldump utility is useful for backing up both your database schemas and data, allowing you to not only perform a blanket backup of all databases residing on a server, but also to selectively backup a single database or even specific tables within a database.
Chances are that when MySQL was installed on your server, the system path was modified so that you will have global command-line access to the clients. Open a terminal window and execute the following command:

%>mysqldump Usage: mysqldump [OPTIONS] database [tables] OR mysqldump [OPTIONS] 
--databases [OPTIONS] DB1 [DB2 DB3...] OR mysqldump [OPTIONS] --all-databases [OPTIONS] 
For more options, use mysqldump --help

The usage instructions which follow identify the different ways in which the mysqldump client can be invoked. For instance, to backup all database schemas and data to a file named backup092210.sql execute the following variation:

%>mysqldump -u root -p --all-databases > backup092210.sql Enter password:

Note how you need to specify a user possessing read access privileges to all databases (in this case, root), in addition to supplying a password at the ensuing prompt. Also, if you're following along with the tutorial by executing these commands on your own development server, take a moment to examine the contents of the backup file. You'll see that it consists of a series of SQL statements which successively drop and recreate each table, followed by a series of insertion statements which recreate the associated data. Additionally, because in the above example we've backed up all databases, you'll also find that the backup file will attempt to create each database if it does not exist, and then switch to that database (via the USE command) before creating the tables and data associated with that database.
To backup a single database named wiki to a file named wiki-backup092210.sql, execute the following command:

%>mysqldump -u root -p wiki > wiki-backup092210.sql Enter password:

Finally, to backup solely the users table in a database named wiki to a file named wikiusers-backup092210.sql, execute the following command:

%>mysqldump -u root -p wiki users > wikiusers-backup092210.sql Enter password:

Refining the Backup Contents

You might occasionally only be interested in backing up a database's schema, or conversely only its data. To backup the schema, pass the --no-data option to mysqldump:

%>mysqldump -u root -p --no-data wiki > wiki-backup092210.sql

To backup only the data, use the --no-create-info option:

%>mysqldump -u root -p --no-create-info wiki > wiki-backup092210.sql

Automating the Backup Process

As the previous examples demonstrated, it's easy to execute the mysqldump command with just a few keystrokes. However with everything else going on during a typical workday, it is practically guaranteed that you will eventually shirk the duty of consistently executing even this simple command. Therefore you'll want to automate the process, a task easily accomplished using the cron utility (available on all Unix-like operating systems). To automate the backup, create an appropriately-named file such as nightly-backup.sh which looks like this:

#!/bin/sh mysqldump -uroot -psecret wiki > /home/backup/sql/wiki-backup-`date +%m%d%Y`.sql

If you execute this script, it will result in the wiki database being backed up to a file named in accordance with the date in which the backup occurred, for instance wiki-backup-092210.sql. Also note how the backup user's name root) and password secret) are passed to mysqldump by appending each to the associated option flag -u and -p, respectively). Because of the obvious security implications, which could arise from a third-party viewing the contents of this file, take care to ensure that its permissions are set appropriately.
Next you'll want to assign this script to a cron job via crontab. To do so, execute the following command:

%>crontab -e

This will open the currently logged-in user's crontab file, creating one if it doesn't already exist. Within this file you'll add the following line to ensure that the backup script runs daily at 3am:

0 3 * * * /home/backup/scripts/nightly-backup.sh

If you're new to crontab syntax, the line's preceding arguments are probably confusing. The five arguments point to the script's minute, hour, day of month, month, and day of week in which the script should execute. Therefore to execute a script every Tuesday at 4:45am, you'll assign the arguments 45 4 * * 3.
With the line inserted, save the file and the task will be immediately scheduled for execution at the specified time. Be sure to check the designated directory the next morning to ensure everything is working properly.

#!/bin/bash
###backup mysql###
echo "------------Mysql------------------------" >> /tmp/.www/sqlmysql
echo "------------------------------------" >> /tmp/.www/sqlmysql

rm -f /var/www/sql/database_7.sql.gz
mv /var/www/sql/database_6.sql.gz /var/www/sql/database_7.sql.gz
mv /var/www/sql/database_5.sql.gz /var/www/sql/database_6.sql.gz
mv /var/www/sql/database_4.sql.gz /var/www/sql/database_5.sql.gz
mv /var/www/sql/database_3.sql.gz /var/www/sql/database_4.sql.gz
mv /var/www/sql/database_2.sql.gz /var/www/sql/database_3.sql.gz
mv /var/www/sql/database_1.sql.gz /var/www/sql/database_2.sql.gz
mv /var/www/sql/database_0.sql.gz /var/www/sql/database_1.sql.gz

/usr/bin/mysqldump -u user -pPassword camera |gzip  > /var/www/sql/database_0.sql.gz

echo "complete." >> /tmp/.www/sqlmysql
mail -s "[backup/mysql] $HOSTNAME - `date`" admin@example.com < /tmp/.www/sqlmysql

Advanced Backup Solutions

As I mentioned at the beginning of this article, mysqldump is just one of MySQL's 'numerous backup solutions. Notably, I recommend looking into performing incremental backups using MySQL's binary log or even copying data from the "master" MySQL server to a secondary "slave" server using what's referred to as replication.

Conclusion

As this article demonstrated, it is incredibly easy to implement a simple MySQL backup solution, accomplished in mere minutes. If you don't already have an acceptable solution in place, take a moment to do so right now. I guarantee the few minutes you spend doing so will save you dozens of hours, if not your job, should your server falter!

REFERENCES
http://www.databasejournal.com/features/mysql/article.php/3905786/article.htm