Friday, February 5, 2010

Recommended Drive Configuration for a Web Server

SkyHi @ Friday, February 05, 2010
What's your typical web server drive configuration? Typically I'll have a drive for the OS and a drive for data. The data drive is typically a RAID 5, but I can't remember what I used to recommend for the OS drive. Is RAID 1 ideal for that?

==============================================================

I work for a major hosting company, and the most common thing I see in my enterprise segment (not necessarily that I recommend, but what I see), assuming that the server is standalone / using local storage, is a RAID 1 OS array, and a RAID5 data array.

Now, as hard drives get bigger, RAID 5 really becomes less ideal, as your likelihood of hitting a URE during a single-drive rebuild is pretty high.

But since you seem to be specifically asking about the OS drive, yes, RAID 1 is standard and usually sufficient, unless you're going to be running other apps off that drive.


REFERENCE
http://serverfault.com/questions/105706/recommended-drive-configuration-for-a-web-server

Should a sys admin backup important data in anyway he can

SkyHi @ Friday, February 05, 2010

Recently, one of the main file servers at our company failed. It was using a 4 disk RAID array, but apparently 3 of the disks died, and all the data on the server has been lost.

Speaking to the sys admin, he says that he has been warning the upper management about the backup situation for months. He had been trying to get approval to buy an enterprise-level backup solution, but he never got the budget approved for it - because management thought it was over the top.

The sys admin is a dedicated properly certified sys admin, whereas his managers are not IT-oriented.

His manager is asking why he didn't buy a cheap external drive and use this to backup the file server. The sys admin thinks that this is just a mickey-mouse solution that's suitable for use at home, but not a professional IT company - which is why he did not do it.

It seems to me that the sys admin wants proper, text-book IT strategy that costs a lot more money, whereas the management (without a deep IT understanding) wants cheaper solutions that they think are adequate.

I'm wondering what is the opinion of other sys admins? Was this sys admin correct in his actions? Or should he always make sure there is a backup of the important data, even if he believes that the cheaper way is not good enough?


Edit: based upon the answers, I'll add that the sys admin has an IT manager who would've known of the situation. He reports to the ultimate boss. I don't know if the manager ever reported the full situation to the boss. I think it is quite tough for the manager, as he is stuck in the middle, and he wants to be diplomatic with both sides.

=============================================================


I would agree that doing it right is the preferred method. But, to stand by and do nothing is unprofessional. Was management informed that there was no backup in place? It is the admins job to present the options, including costs and risks, to management. He presented his preferred option, and when it was denied, he did nothing. Not cool.

would honestly say it is a failure on both parts.

The logistics of the situation might mean that he would have to take away time that he should have been spending doing other, immediate, important tasks.

However, ultimately, yes, he should have done something. A bunch of hard drives from here and there would have been better than nothing as has been said repeatedly.

On the other hand, the entire purpose of management is to make sure that the people beneath you can do their jobs, and do. and thus from a leadership point of view, the managers failed miserably and can be held equally responsible, if not moreso.


=============================================================

If there are no backups, as far as I'm concerned it's the sysadmin's responsibility to:

1) Explicitly tell the higher ups that there is NO backups, in no uncertain terms, so that they are aware of it

2) Back up the data anyway, any way he can

Frankly I would expect to be fired if this happened, because even if management are making my life hard, that's not an excuse, especially if they're still under the impression that they have something rather than nothing.


=============================================================

Unfortunately, companies skimping on backups is all too common. Most never change until they get burned and lose everything.

BUT

If you are employed to be the sysadmin you have to work with the tools that you have including your brain. No matter what management or anyone else says on good days, when the poop hits the fan everyone gets selective memory.

A mickey mouse backup is better than no backup at all.


==============.===============================================

It's damned if you do, damned if you don't. Frankly, if there was no money spent by management on a backup solution, then it's their fault. On the other hand, the admin should have been active in trying to work out a stopgap solution, rather than just sitting on his ass waiting for something to break (I don't think any sort of external drive solution is acceptable. You're never going to get a decent backup with that.) You can't just say, "Well I don't have what I want, so I'm not responsible" but you can say, "I've repeatedly tried to get you to do something and you've given me nothing and this is not my problem."

I was actually in a situation once--I wasn't even an ADMIN at this job--where I was working on a database, and made a backup before I changed it (which is s.o.p), and I (as I usually do, whenever I can) saved it to my own local machine. Two days later they lost the raid array, and ooops, turned out there was no backup solution. They'd been backing up the database to the raid array.

So I come in late on this, and I say, "Oh, I backed it up myself day before yesterday."

You know what the outcome was? I was censured for my bad backup solution. For a machine that I was in no way responsible for. And it wasn't because the backup I had was too old, it's because I'd only backed up the database I was working on, not every database.

So the problem is this: if you do a mickey mouse solution, if you do anything, and it's not quite good enough, you're going to get just as much hell as if you do nothing at all. If backups are your responsibilty, explicitly, and there is no budget, you should try to cobble something together, but you better make damn sure it works, and you need to raise hell about it. Repeatedly. At every opportunity.

If it's not your responsibility, point out that there exists a problem, and absolutely, categorically, refuse to take responsibility for an unfunded mandate when they try to assign it to you. No one makes disaster recovery a priority until there is a disaster, and then they scapegoat everyone to try and make up for their own shortsightedness.


=============================================================

I'll add my voice to those saying that the admin should have implemented something here. He's badly at fault for not having done so. There's a part of me that would like to sympathise with his position, but in an ideal world backup and restore would take no time, always work, and never be needed. This isn't that world and even the best backup solution is going to have flaws that you'll need to accept and learn to work with.

Half-assed is better than no-assed, and even using an el-cheapo USB HD would have gotten him out of the woods, and would have given weight to his position when management are told that they can't get data more than day or two old back. But it would have still saved his neck in this case.


==============================================================

Should a sys admin backup important data in anyway he can.

I don't know that I would say a you should make a backup under any conditions. There are some things you might be tempted to do that would possibly be illegal. For example I would not backup health records over the network to my personal computer. I would not do something illegal just to have a backup.

OTOH to have at least some backup system in place I would accept a lot of compromises. Then whenever a compromise was made I would make a point to make sure my objections are clear and documented about why it was a bad compromise that will cause problems, be inadequate, or become less useful in the future.



=============================================================

To me it sounds like the sysadmin wanted all or nothing. It's nice to get all, but if you can't have it should you accept nothing?

In my experience, the thing to do is evaluate all the possible options, (not in too much depth), and draw up a few bullet points for each indicating the pros and cons, costs (both inital and ongoing). Include in this the "do nothing" option.

Then you allow the managers to decide what solution they choose. It would seem to me that there was probably more than one possible option for your sysadmin. Perhaps he only saw the one he really wanted though?


=============================================================

As a sysadmin I believe it is my responsibility to ensure the systems under my care are as secure and reliable as I can possibly make them. Backups fall under the reliability tags. Frustrating as it may be to have to argue with non-understanding senior staff (I think we've all been there at some time or other), we still should be doing our jobs as best we can.

When the backup system I inherited in my current position failed and management hesitated about spending the money on the system I wanted I didn't leave the system without backups. Instead, I brought my personal external drive in and used that for a week or so. Despite having an absolute abhorrence for using hard drives for backups the fact remains that it was vastly preferable to having none at all.


=============================================================

If the sysadmin was unable to convince management of the importance of a good backup solution the only way they will ever be convinced is via catastrophic data loss, but as a sysadmin it is your responsibility to educate management and users about the importance of things like backup, and to make sure they thoroughly understand the current state (in this case "no backups") and potential consequences ("We lose a disk and your precious data is gone forever").

My personal opinion is that the admin kinda screwed up here: Ad-Hoc backups are a bad idea (you'll miss stuff, important data will be lost, if you're not around backups don't happen), but at the same time they should have been able to find a reliable "enterprise" backup solution within the company's budget.
Software like Bacula and Amanda is available for free, and both of those can work with removable USB media and CDs securely and reliably. Including the cost of media and server hardware you could have a good system for less than $2000 US - even cheaper if you recycle hardware for the server.

Now if management is also opposed to the admin spending TIME on getting backups working there's just no helping this company: As I said above sometimes the only way to teach people is catastrophic data loss, and if that's the case it sucks for the poor admin who has to take the blame for institutional stupidity.


=============================================================

My personal opinion is that it's my job as a sysadmin to inform and impress upon management the need for and the importance of having an adequate, appropriate backup solution and requesting the neccessary budget for such, and to explain the risks associated with not doing so. It's not my responsibility to go "outside" of the mandate of the management and just do whatever I think is right regardless of how poor those management decisions are. It's not my responsibility to cobble together some half-baked, half-assed solution.

If I was an insurance agent and I told you it was important to have fire coverage in your home owner's policy, and if I adequately explained the risk of not having fire coverage, and you declined said fire coverage, and your house burned down, who's responsibility is it? Should I have given you fire coverage anyway?

My opinion is that the sysadmin exercised due dilligence in performing the duties of his job by bringing the matter to the attention of management, explaining the importance of having an appropriate backup solution, explaining the risks of not having it, and requesting the neccessary budget for such. If he was rebuffed in his efforts then the responsibility lies squarely on the shoulders of the management.

People make poor decisions all the time and bad things happen because of those poor decisions, that's a fact of life. I can't be responsible for every bad decision my boss makes, regardless of the risks associated with those decisions.


=============================================================

Did the same situation happen with the RAID array? As soon as one disk dies, you are in a situation where one more means data loss.. you better replace that drive immediately.

If I was in the sys admin's shoes, the instant the first drive went:

  1. Email manager with formal request to replace the drive, reminding that no backup system has been approved so this is a critical situation. Cite the prior requests for the backup system by attaching that email, or even better, the manager's response denying the request.
  2. If no response, re-send message, this time CC'ing your manager's manager.
  3. If still no response, well.. not much more you can do. Polish the resume and start looking for a better job.

If you get denied along the way, at least you have it in writing for when the shit hits the fan (Get it in writing/email, do not accept a verbal response. You need a paper trail here. If your manager refuses to write it, then go over his/her head, because that's just shady -- there is no legit reason not to write it down.)

The same process should have been followed for getting a backup system, though perhaps without escalation as quickly (or going over your manager's head at all). If none of the requests are in writing, well.. shit rolls downhill. At least it's a good life lesson.

If you don't lose your job over the situation, well, start making that request again, citing the disaster it caused last time your request was denied. If it's still denied, then you need to decide if that's an environment you want to work in, and it's worth the stress. If every morning you expect to walk into work finding a panic because data was lost, well, that's no way to live.



============================================================

The company is clearly looking for a scape goat in this, the sys admin is quite right not to backup critical data to a removable device.

1) They are not reliable 2) They are not secure

Ultimately it lies with the managers for not ensuring a proper DR (Disaster recovery) solution was put in place.

Look at it this way, how much has this data loss cost the company? Suddenly I'm sure the "over the top" solution doesn't look so expensive.

edit: yes I concede the fact any backup is more reliable than none, but my original point remains if this person has managers, the managers should of ensured the backup was in place, I'm not pardoning the sys admin of all blame here, but this this what the manager should be checking.

And what if the server failed and the data on the removable drives was irrecoverable for whatever reason? having had this occur myself in the past USB drives are far from reliable, but to some they can be used in a "pinch" the problem is as it appears in this case the management would of allowed removable drive backup to be used in the long run.



REFERENCE

http://serverfault.com/questions/109793/should-a-sys-admin-backup-important-data-in-anyway-he-can-even-if-he-disagrees-w




Bash For Loop Examples

SkyHi @ Friday, February 05, 2010
How do I use bash for loop to repeat certain task under Linux / UNIX operating system? How do I set infinite loops using for statement? How do I use three-parameter for loop control expression?

A 'for loop' is a bash programming language statement which allows code to be repeatedly executed. A for loop is classified as an iteration statement i.e. it is the repetition of a process within a bash script.
For example, you can run UNIX command or task 5 times or read and process list of files using a for loop. A for loop can be used at a shell prompt or within a shell script itself.

for loop syntax

Numeric ranges for syntax is as follows:
for VARIABLE in 1 2 3 4 5 .. N
do
 command1
 command2
 commandN
done
This type of for loop is characterized by counting. The range is specified by a beginning (#1) and ending number (#5). The for loop executes a sequence of commands for each member in a list of items. A representative example in BASH is as follows to display welcome message 5 times with for loop:
#!/bin/bash
for i in 1 2 3 4 5
do
   echo "Welcome $i times"
done
Sometimes you may need to set a step value (allowing one to count by two's or to count backwards for instance). Latest bash version 3.0+ has inbuilt support for setting up ranges:
#!/bin/bash
for i in {1..5}
do
   echo "Welcome $i times"
done
Bash v4.0+ has inbuilt support for setting up a step value using {START..END..INCREMENT} syntax:
#!/bin/bash
echo "Bash version ${BASH_VERSION}..."
for i in {0..10..2}
  do
     echo "Welcome $i times"
 done
Sample outputs:
Bash version 4.0.33(0)-release...
Welcome 0 times
Welcome 2 times
Welcome 4 times
Welcome 6 times
Welcome 8 times
Welcome 10 times

The seq command (outdated)

WARNING! The seq command print a sequence of numbers and it is here due to historical reasons. The following examples is only recommend for older bash version. All users (bash v3.x+) are recommended to use the above syntax.
The seq command can be used as follows. A representative example in seq is as follows:
#!/bin/bash
for i in $(seq 1 2 20)
do
   echo "Welcome $i times"
done
There is no good reason to use an external command such as seq to count and increment numbers in the for loop, hence it is recommend that you avoid using seq. The builtin command are fast.

Three-expression bash for loops syntax

This type of for loop share a common heritage with the C programming language. It is characterized by a three-parameter loop control expression; consisting of an initializer (EXP1), a loop-test or condition (EXP2), and a counting expression (EXP3).
for (( EXP1; EXP2; EXP3 ))
do
 command1
 command2
 command3
done
A representative three-expression example in bash as follows:
#!/bin/bash
for (( c=1; c<=5; c++ ))
do
 echo "Welcome $c times..."
done
Sample output:
Welcome 1 times
Welcome 2 times
Welcome 3 times
Welcome 4 times
Welcome 5 times

How do I use for as infinite loops?

Infinite for loop can be created with empty expressions, such as:
#!/bin/bash
for (( ; ; ))
do
   echo "infinite loops [ hit CTRL+C to stop]"
done

Conditional exit with break

You can do early exit with break statement inside the for loop. You can exit from within a FOR, WHILE or UNTIL loop using break. General break statement inside the for loop:
for I in 1 2 3 4 5
do
  statements1      #Executed for all values of ''I'', up to a disaster-condition if any.
  statements2
  if (disaster-condition)
  then
 break           #Abandon the loop.
  fi
  statements3          #While good and, no disaster-condition.
done
Following shell script will go though all files stored in /etc directory. The for loop will be abandon when /etc/resolv.conf file found.
#!/bin/bash
for file in /etc/*
do
 if [ "${file}" == "/etc/resolv.conf" ]
 then
  countNameservers=$(grep -c nameserver /etc/resolv.conf)
  echo "Total  ${countNameservers} nameservers defined in ${file}"
  break
 fi
done

Early continuation with continue statement

To resume the next iteration of the enclosing FOR, WHILE or UNTIL loop use continue statement.
for I in 1 2 3 4 5
do
  statements1      #Executed for all values of ''I'', up to a disaster-condition if any.
  statements2
  if (condition)
  then
 continue   #Go to next iteration of I in the loop and skip statements3
  fi
  statements3
done
This script make backup of all file names specified on command line. If .bak file exists, it will skip the cp command.
#!/bin/bash
FILES="$@"
for f in $FILES
do
        # if .bak backup file exists, read next file
 if [ -f ${f}.bak ]
 then
  echo "Skiping $f file..."
  continue  # read next file and skip cp command
 fi
        # we are hear means no backup file exists, just use cp command to copy file
 /bin/cp $f $f.bak
done

Recommended readings:

  • See all sample for loop shell script in our bash shell directory.
  • man bash
  • help for
  • help {
  • help break
  • help continue
Updated for accuracy!


REFERENCE
http://www.cyberciti.biz/faq/bash-for-loop/



Thursday, February 4, 2010

RPM packages for Red Hat, RHEL, CentOS and Fedora

SkyHi @ Thursday, February 04, 2010
B2. How to configure to use RPMforge ?
It's very easy. Just install the latest rpmforge-release package for your distribution and architecture.

This will automatically install the configuration and GPG keys that are for safely installing RPMforge packages.

Please select the correct command from the following list:
REFERENCE
http://dag.wieers.com/rpm/FAQ.php#B

Chapter 12 - Sendmail Server

SkyHi @ Thursday, February 04, 2010

Versions: - sendmail 8.13.6

- dovecot 1.0

- clamav 0.88.2

- clamav-milter 0.88.2

- spamassassin 3.1.3

Basic Configuration
Dovecot IMAP Server
Starting The Services
Preventing Abuse
Full SSL/TLS Encryption
Clam Antivirus
SpamAssassin

Sending emails to multiple recipients scattered around the world these days is such an easy everyday task, that its hard to image life without it. Sending an email through the Internet uses many different protocols and applications that all work seamlessly to ensure your message reaches its end destination.

This chapter will provide assistance in the configuration of your own server for sending and receiving emails, and will also provide details on some extra applications to ensure your system remains secure and relatively virus free. In chapter 13, we will configure a webmail application which provides a means to send and receive email while away from home. To make proper use of an email system requires the server to be configured with a fully registered Internet domain name; DNS is used extensively for email. In chapter 20 we will configure LDAP to provide our network with a share address book (with SSL) to be used by internal and/or roaming clients.

Before we begin, here are some of the basic components that help to make up the email world:

MUA: Mail User Agent The email application that a user sends/receives (thunderbird,pine,outlook)
MTA: Mail Transport Agent
The server agent responsible for sending the emails (sendmail,postfix,qmail)
MDA: Mail Delivery Agent
The server agent that accepts email from MTA, and places into users mailbox (procmail)
SMTP: Simple Mail Transport Protocol
MUAs and MTAs use this protocol for sending emails
POP3: Post Office Protocol (Ver 3)
MUAs use this protocol for receiving their emails from the final server
IMAP: Internet Message Access Protocol MUAs can use this protocol to send and receive emails on the servers

Procmail is normally installed and running on most systems already so it won't be covered here.

Note !! The preferred email protocol for this configuration is IMAPS as all emails are stored on the main server and then replicated through to your MUA client when it connects. Because the mail files on the server and client are synchronised, the webmail application will have all the emails contained on your local workstation and vice versa, including sent emails.

Basic Configuration

Sendmail

We are going to configure the MTA first, and the package that is most commonly used is sendmail. The configuration files for sendmail are extremely technical for new users, and should definately be backed up before making any changes, they are so complex in fact that one configuration file is actually used to configure the second file.

The important thing to remember is that we will make the changes only in the sendmail.mc file, and the changes will be moved into the sendmail.cf file for us. This is the preferred method for configuring sendmail.

[bash]# cp /etc/mail/sendmail.cf /etc/mail/sendmail.cf.original

Caution !! Do not edit sendmail's "cf" file, use the "mc" macro file to make the changes. This backup is only a precautionary measure in case everything goes bad.

[bash]# cp /etc/mail/sendmail.mc /etc/mail/sendmail.mc.original
[bash]# vi /etc/mail/sendmail.mc

Inside the sendmail macro file, there are many lines that start and end with "dnl". This acronym stands for "delete through newline", and is used to tell the macro process to ignore the current line when building the new sendmail.cf file. This should be treated just like a standard configuration comment.

When sendmail dispatches your email, it places the servers hostname behind your username, which becomes the "from address" in the email (ie. user@galaxy.example.com). Normally we only want to use the domainname and not the hostname, this setting allows all outgoing emails to be "user@example.com", the preferred method.

define(`confDOMAIN_NAME', `example.com')dnl
FEATURE(`relay_entire_domain')dnl

Email can be sent via a smarthost if the Internet connection is not on all the time. This needs configuring at your ISP or by whoever is providing your smarthost capability. This is generally not required on a dedicated link.

dnl define(`SMART_HOST',`smtp.your.provider')dnl

Below are some daemon options that specify (basically) who can connect and send emails. The top one (default) only allows emails to be sent from by the local server. The default options need to be adjusted before the server will accept emails from any other networked computer.

DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl Only local server can send email
DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl Users can now connect to send email

The aliases and virtusertable ("/etc/mail/virtusertable") are lists that allow incoming emails to be redirected to other local accounts or external email addresses. See "man aliases" or README for details on these files.

Many of the applications on the server are configured to send email to the root account when problems occur. It is recommended that an entry be placed into the /etc/aliases file to redirect all of root's email to the supervisors standard user account. This saves having to check the root account for emails as they can be automatically sent to another valid account.

define(`ALIAS_FILE', `/etc/aliases')dnl
FEATURE(`virtusertable',`hash -o /etc/mail/virtusertable.db')dnl

Note !! If alias file (/etc/aliases) is adjusted, it needs to be updated with the "newaliases" command before settings are implemented.

The access database ("/etc/mail/access") is a list of IP addresses and domainnames of allowable connections. Your IP subnet and other details need to be placed in here before you can send email, ensuring that you leave the localhost information as configured.

FEATURE(`access_db',`hash -T<TMPF> -o /etc/mail/access.db')dnl
# Example of /etc/mail/access
localhost.localdomain     RELAY
localhost                 RELAY
127.0.0.1                 RELAY
192.168.1                 RELAY
example.com               RELAY

Before accepting emails, sendmail can do a DNS lookup on the address of the person who is sending the email. It is recommended that this option be set up so that any email failing a DNS lookup is rejected; there is a fair chance the email was not sent from a valid domain name and possibly spoofed. Comment the following line to enable DNS lookup.

dnl FEATURE(`accept_unresolvable_domains')dnl

Caution !! You should consult the README file for further details on available settings ("/usr/share/sendmail-cf/README").

Mail Aliases - Who gets Who's Email

Sendmail includes a standard "/etc/aliases" file that can be used to redirect certain emails to a single user on the system, a group of users, or even to an email address external of the mail system (i.e. to another organisation). The basic alias file looks similar to below, it contains an alias on the left side followed by a colon and then the list of users that will receive the email, listed on the right hand side.

It is important to nominate an account to redirect all the system emails to, i.e. a "root" alias. This ensures that any system orientated alerts are directed to an individual who can monitor the system's status and warning messages.

[bash]# cp /etc/aliases /etc/aliases.original
[bash]# vi /etc/aliases
# Our Own Aliases
www:            root
admin:          root
sysadmin:       root
webmaster:      root
support:        helpdesk

# Person who should get root's mail
root:           john             <-- John will receive all system/security email alerts meant for root.

# People who have left our organisation - Mail redirection...
sarah:          sarah@otherdomain.org
tom:            tom@differentorganisation.org

Aliases can also be configured to for mutliple recipients, this is a convenient way to create generic group email accounts outside of a global address book. With this example, a file is created with a list of mail receipiants listed on one line at a time.

[bash]# vi /etc/mail/mailing-list
alice@wonderland.com
peter@nevernever.org
harry@potterworld.net

The "mailing-list" alias is listed in the alias table, and the file containing the list of users is also added with the use of an "include" statement; this tests sendmail where the list of users can be found for the mailing-list. Alternatively, an alias can have several recipients listed directly aside the alias, separated by commas.

[bash]# vi /etc/aliases
# Our Own Aliases
sysadmins:      john,mark,lisa
mailing-list:   :include:/etc/mail/mailing-list

After the alias table has been adjusted, the "newaliases" command needs to be executed before the table will be used by sendmail again.

[bash]# newaliases
/etc/aliases: 97 aliases, longest 31 bytes, 998 bytes total

Dovecot IMAP Server

Now that the sendmail server has been setup to allow the sending of emails, we need to configure a means for the user to retrieve any emails that are waiting for them on the server. One of the packages that does this is dovecot, which handles POP and IMAP mailboxes in clear text or with link encryption (POPS and IMAPS); IMAPS is the preferred mail protocol for MUAs.

Dovecot is relatively easy to configure, but backing up the config file is still important.

[bash]# cp /etc/dovecot.conf /etc/dovecot.conf.original
[bash]# vi /etc/dovecot.conf

Dovecot needs to be told which protocols it will accept for incoming client connections, the setting below is the main dovecot configuration file; note its detail for both IMAP and POP3 protocols.

protocols = imap pop3

login_dir = /var/run/dovecot/login
login_chroot = yes
login_user = dovecot

protocol imap {
login_executable = /usr/libexec/dovecot/imap-login
mail_executable = /usr/libexec/dovecot/imap
login_greeting_capability = yes
}

protocol pop3 {
login_executable = /usr/libexec/dovecot/pop3-login
mail_executable = /usr/libexec/dovecot/pop3
pop3_enable_last = no
}

auth_executable = /usr/libexec/dovecot/dovecot-auth
auth_process_size = 256
auth_cache_ttl = 3600

auth default {
  mechanisms = plain
  user = root
  ssl_require_client_cert = no
  passdb pam {
  }
  userdb passwd {
  }
}

Caution !! If you plan on setting up SquirrelMail for your webmail requirements, you will need to have the IMAP protocol enabled.

User Email Accounts

When a standard user account is created on the server, that user is automatically granted access to login to the system and is also able to send and receive emails. The user now really only needs to configure their MUA with the details of the servers address and their account details, and email should be fully accessible.

Creating a standard user account with a false shell stops that account from being able to log into the system (a Linux login), but still allows them to use the system for the purpose of sending and receiving emails. Creating a user account using the following example, is the easiest method for creating email only accounts for friends and family where they do not require login access to the server.

[bash]# useradd -c "Alice Jones" -s /sbin/nologin alice

Email only accounts forbid a user from logging in and accessing their home directories. These accounts may not be suitable if users expect access to their "public_html" directories if they are present.

You may also consider placing all of the email only accounts into an appropriately named group for easy management.

Starting the Services

Starting the newly configured services are relatively straight forward, however if any changes have been made to the sendmail macro file or access lists, then the sendmail.cf file will need to be recompiled before any of the changes will be accepted. These days many of the initscripts handle this task automatically, however its good to do it manually just to be certain.

[bash]# make -C /etc/mail

The services should now be set at the appropriate runlevels and then checked to ensure they are correct.

[bash]# chkconfig --level 2345 sendmail on
[bash]# chkconfig --level 2345 dovecot on
[bash]# chkconfig --list sendmail
[bash]# chkconfig --list dovecot

The services can now be started. The system logs should also be checked to ensure there are no errors.

[bash]# /etc/init.d/sendmail restart
[bash]# /etc/init.d/dovecot restart
[bash]# grep sendmail /var/log/maillog
[bash]# grep dovecot /var/log/maillog

You've got email !

Preventing Abuse

Open mail relays

The worst exploitation of an email system is if its able to relay emails for everyone, this allows spammers and other nasty organisations to use your unprotected system as a platform for their malicious activities. Although we have defined who can use the system in the above configurations (access database), we can take extra precautions to minimise the effects of any possible damage.

The following example is a basic open relay test that you can perform on your sendmail server as an example of how easy it is for a spammer to send emails through an insecured MTA. Firstly you need to telnet into your server on port 25 (SMTP) and then cut and paste the following text into the telnet session; remembering to change the "RCPT To:" address to your own email.

[bash]# telnet localhost 25
(CUT AND PASTE BELOW TEXT)
(Change "RCPT To:" email address)

HELO example.com
MAIL From: TheBoss@example.com
RCPT To: sysadmin@example.com                     <-- Change this to your own email to see results.
DATA
Subject: Think we're insecure...
I have a feeling our mail server is being abused...
.
QUIT

Now if you check your email you will notice an email from "TheBoss" and you'll see how easy it is to spoof an email. If this worked, it means that you have probably enabled your "/etc/mail/access" database with the correct RELAY permissions for it to occur. However, we only want to allow the correct clients to be able to relay through the MTA.

The following telnet test is an another open relay test that will automatically test your system from a remote Internet based relay attempt. Approximately 19 relay tests are conducted with results displayed as they occur, this is a good test of your systems configuration; it your server is open to relay, the spammers will find you soon.

[bash]# telnet relay-test.mail-abuse.org

Caution !! If any of the open relay tests return serious warnings, you should seriously check your systems configuration - guaranteed your system will be exploited.

Email limits

The default file size for sendmail is unlimited, so if someone tries to send a tar archive over 2 GB throught the system, it will most likely crash the MTA while trying. A maximum file size should be set so sendmail knows when to reject a file that is too large.

[bash]# vi /etc/mail/sendmail.mc
define(`confMAX_MESSAGE_SIZE',`52428800')dnl

If you are using a PHP based webmail application like SquirrelMail, you can adjust the max file size for PHP to match the same amount; this allows PHP applications to match your mail server size limits.

## ONLY NEEDED TO SUPPORT PHP WEBMAIL ##
[bash]# vi /etc/php.ini
post_max_size = 50M
upload_max_filesize = 50M
memory_limit = 64M

The following settings limit the rate of connections, the amount of running 'children', and the maximum number of recipients for each email. By specifying these types of directives it limits the rate of any possible exploitation or Denial of Service attacks. These would be suitable for a small office or home network, but may need to be increased for larger demands.

define(`confMAX_DAEMON_CHILDREN',`5')dnl
define(`confCONNECTION_RATE_THROTTLE',`3')dnl
define(`confMAX_RCPTS_PER_MESSAGE',`50')dnl

This section provides a few basic settings as an introduction to possible abuse situations and should provide you with some security considerations for your system. You should always check your system logs for any signs of abuse, and do some form of test attack on your own system (from the outside of course).

Full SSL/TLS Encryption

One of the easiest ways for any system to be exploited is for a username and password to be intercepted whilst traveling through the Internet, the basic action of logging into your server from the external (Internet) side opens your user credentials to such a risk. Luckly both Sendmail and Dovecot can both be covered by TLS and SSL encryption systems to ensure your credentials and correspondence stay safe.

To configure sendmail with TLS / SSL encryption, edit the main configuration file and make the following changes. The first setting disables plain text authentication, the second defines the trusted authentication mechanisums (to allow relaying), the third defines the SSL certificate files and the fourth enables the TLS link encryption and opens port 465 for secure emails (SMTPS).

[bash]# vi /etc/mail/sendmail.mc
define(`confAUTH_OPTIONS', `A p')dnl
TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confCACERT_PATH',`/etc/pki/tls/certs')dnl
define(`confCACERT',`/etc/pki/tls/certs/ca-bundle.crt')dnl
define(`confSERVER_CERT',`/etc/pki/tls/certs/sendmail.pem')dnl
define(`confSERVER_KEY',`/etc/pki/tls/certs/sendmail.pem')dnl
DAEMON_OPTIONS(`Port=smtps, Name=TLSMTA, M=s')dnl

Note !! The bottom option causes sendmail to additionally listen for secure connections on port 465 through enforced SSL. Basic SMTP is still configured through port 25 for remote MTA connections and TLS.

In our initial configuration, we allowed sendmail to accept SMTP email from all hosts (the default is only from 127.0.0.1, itself). By changing this setting back to it's default, then only the local server can send unsecured emails, this is ideal if you are going to configure webmail to run from the local Apache web server.

Change This:
DAEMON_OPTIONS(`Port=smtp, Name=MTA')dnl
Back To:
DAEMON_OPTIONS(`Port=smtp, Addr=127.0.0.1, Name=MTA')dnl

Now that sendmail had been configured for encrypted connections and authentication, you will need to create your SSL certificates before you can activate your new configuration; this can be done using the automated scripts located in the "/etc/pki/tls/certs" directory.

[bash]# cd /etc/pki/tls/certs
[bash]# make sendmail.pem
Country Name (2 letter code) [GB]:AU
State or Province Name (full name) [Berkshire]:QLD
Locality Name (eg, city) [Newbury]:Brisbane
Organization Name (eg, company) [My Company Ltd]:Miles Brennan
Organizational Unit Name (eg, section) []:Home Linux Server
Common Name (eg, your name or your server's hostname) []:galaxy.example.com
Email Address []:sysadmin@example.com

You should now be able to send secure emails to your MTA (Sendmail) for delivery to remote addresses.

The next step is to configure Dovecot so you may be able to successfully retrieve emails from your mail server using SSL encryption; sending is only half way there, the receive path also needs to be configured. Open your Dovecot configuration and make the following adjustments. A point to note is that we are configuring the Dovecot server to use the exact same SSL certificates that the Sendmail server is using.

[bash]# vi /etc/dovecot.conf
ssl_disable = no
ssl_verify_client_cert = no
ssl_parameters_regenerate = 168
ssl_cipher_list = ALL:!LOW
ssl_cert_file = /etc/pki/tls/certs/sendmail.pem     <-- NOTE: Can use same certificate as Sendmail
ssl_key_file = /etc/pki/tls/certs/sendmail.pem      <-- NOTE: Can use same certificate as Sendmail
disable_plaintext_auth = yes

We can also block Dovecot from allowing any insecure client connections by forcing the server to only accept secure IMAPS and POP3S connections.

Change This:
protocols = imap pop3
To This:
protocols = imaps pop3s

Now that the services have both been configured, the services will need to be restarted.

[bash]# /etc/init.d/sendmail restart
[bash]# /etc/init.d/dovecot restart

If you have users on the external side of your firewall (i.e. on the Internet), you can allow both SMPTS (port 465) and IMAPS (port 993) connections to pass through your firewall with the following adjustments to your firewall script. Remember that port 25 (SMTP) is still required and will use TLS as required.

[bash]# vi /root/firewall.sh
# New INBOUND Connection: SMTP and SMTPS (over SSL)
iptables -A INPUT -i $EXT_DEV -m state --state NEW -m tcp -p tcp --syn --dport 25  -j ACCEPT  <-- for TLS encryption (and basic SMTP)
iptables -A INPUT -i $EXT_DEV -m state --state NEW -m tcp -p tcp --syn --dport 465 -j ACCEPT  <-- for SSL encryption

# New INBOUND Connection: IMAPS Email Clients (Secure Link - In and Out)
iptables -A INPUT -i $EXT_DEV -m state --state NEW -m tcp -p tcp --syn --dport 993 -j ACCEPT  <-- for SSL encryption
[bash]# /root/firewall.sh

By default the Dovecot server will listen on all interfaces when we declare a protocol to be configured. However we can add the following options to the protocol declaration to define a more detailed configuration.

The following Dovecot configuration causes both the secure IMAPS and POP3S protocols on be active on all network interfaces, but also allow Dovecot to operate the insecure IMAP protocol only from the localhost server, this again is an ideal configuration if you intend to run SquirrelMail on your server and it is an IMAP based application.

[bash]# vi /etc/dovecot.conf
protocols = imap imaps pop3s

protocol imap {
listen = 127.0.0.1
ssl_listen = *
ssl_disable = no
}

Note !! The above Dovecot settings allow for the IMAP based SquirrelMail application to work on the local server without requiring TLS/SSL encryption. A remote user will still interface the HTTPS web interface through SSL, ensuring secure access through a web browser.

Clam Antivirus

There is really no such thing as a virus in the Linux world and what you would call a virus is really nothing more than an exploit or maliciously crafted piece of code. Some not so better operating systems however are quite susceptible to virus attacks and the number one means of virus infection is currently via the email system. Although your Linux MTA itself is quite safe from any possible virus threats, we have a duty of care to at least protect our internal workstations that may face a somewhat different fate.

Clam AntiVirus is an opensource antivirus scanning toolkit for UNIX systems. Effectively it runs as its own independent service with its own suite of applications, these need to be interfaced by other applications in order to use the scanning facilities. Sendmail interfaces into clamav with whats called a milter (Mail Filter), and the results are returned to sendmail for further processing.

Clamav-Milter

Firstly we need to install the clamav suite of applications, this command will install the clamav server and the milter required to work with Sendmail.

[bash]# yum install clamav*

The clam daemon configuration file should be backed up before any adjustments are made.

[bash]# cp /etc/clamd.d/milter.conf /etc/clamd.d/milter.conf.original
[bash]# vi /etc/clamd.d/milter.conf

The following is an example of a typical clamd configuration file. The "/etc/clam.d/clamd.conf" file is well documented with configuration options and further detailed information can be obtained from reading the supporting man page. Type "man clamd.conf" at the command prompt.

The "Example" option must be commented out before the daemon will start.

#Example                              <-- This must be commented out before the daemon will function
LogFile /var/log/clamd.milter
LogFileMaxSize 5M
LogTime
DatabaseDirectory /var/lib/clamav
LocalSocket /var/run/clamd.milter/clamd.sock
FixStaleSocket
#TCPAddr 127.0.0.1
#TCPSocket 3310
User clamilt
ScanMail
ScanHTML
DetectBrokenExecutables
ArchiveBlockEncrypted

On one side we have a running MTA (Sendmail) and the other side we have a fully functional antivirus system, clamav-milter is the glue that binds it all together. As an email is passed onto the MTA, the milter redirects it to the antivirus server and awaits the results.

The following parameters detail how the milter will act and respond between the other two systems; should it just drop all infected email, should it notify the recipient or the sender, should it inform the administrator; there are numerous available options. You should consult the man page to configure the settings that will best suit your needs. Type "man clamav-milter" at the command prompt for more details.

[bash]# cp /etc/sysconfig/clamav-milter /etc/sysconfig/clamav-milter.original
[bash]# vi /etc/sysconfig/clamav-milter
CLAMAV_FLAGS="--local \
              --bounce \
              --advisory \
              --force-scan \
              --dont-wait \
              --dont-log-clean \
              --max-children=2 \
              --server=localhost \
              --postmaster=sysadmin@example.com \
              --config-file=/etc/clamd.d/milter.conf \
              --pidfile=/var/run/clamav-milter/milter.pid \
              --signature-file=/etc/mail/clamav-email-signature \
              local:/var/run/clamav-milter/clamav.sock "
CLAMAV_USER='clamilt'

In the above sample configuration we are placing a footer (signature-file) on all the emails that successfully pass thru the system, as an assurance to the end recipients that the email is virus free. The following is an example signature file, it isn't required but is an option should your email policy require it.

[bash]# vi /etc/mail/clamav-email-signature
_________________________________
This email has been ClamScanned !
          www.clamav.net

The service is now ready to run after we set the correct runlevels. The clamav-milter logs directly to "/var/log/maillog" when it processes an email, which can be checked after all the configurations are complete.

[bash]# chkconfig --level 2345 clamav-milter on
[bash]# chkconfig --list clamav-milter
[bash]# /etc/init.d/clamav-milter restart

Now that the milter is running, we need to tell sendmail to use it. Place the following line into the sendmail.mc file, then restart the Sendmail service.

[bash]# vi /etc/mail/sendmail.mc
INPUT_MAIL_FILTER(`clamav-milter', `S=local:/var/run/clamav-milter/clamav.sock, F=T,T=S:4m;R:4m;E:10m')
[bash]# make -C /etc/mail
[bash]# /etc/init.d/sendmail restart

Thats it !! You are all configured now for antivirus scanning.

Testing the Clamav Scanner

The easiest way to check if the scanner is functioning, is to check the mail logs. The details of each email that gets sent by the milter to clamd is placed in the log, the example below shows how the milter added the extra header details to the incoming email.

[bash]# grep Milter /var/log/maillog
sendmail: Milter add: header: X-Virus-Scanned: ClamAV version 0.88.2, clamav-milter version 0.88.2 on galaxy.example.com
sendmail: Milter add: header: X-Virus-Status: Clean

The email can be checked at the MUA to confirm the presence of the extra milter headers on all of the incoming emails. This is confirmation that the system is configured and functioning as expected.

X-Virus-Scanned: ClamAV version 0.88.2, clamav-milter version 0.88.2 on galaxy.example.com
X-Virus-Status: Clean

The European Institute for Computer Antivirus Research (EICAR) have created an antivirus test signature that can be used to test many antivirus programs. Your new Sendmail configuration can be tested by sending the EICAR test signature through as an email and check to see if Clamav identified and labelled the email as an infected message.

Issue the following command at the command line interface, ensuring that you change the email address to your own (or one you can access). When the email arrives at the email address you inserted, you can check the headers within the email and see if Clamav has inserted the following flags.

[bash]# echo 'X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' | mail sysadmin@example.com
To: sysadmin@example.com
X-Virus-Scanned: ClamAV version 0.88.2, clamav-milter version 0.88.2 on galaxy.example.com
X-Virus-Status: Infected with Eicar-Test-Signature
Subject: [Virus] Eicar-Test-Signature

Note !! More information about the EICAR test antivirus signature can be seen at this site: http://www.eicar.org/anti_virus_test_file.htm

Freshclam updates

The Clam Antivirus system is only as effective as its latest copy of the virus definitions, these can be updated using the freshclam application. Make the following adjustments to your freshclam settings, ensuring you ADD a "#" (comment) to the "FRESHCLAM_DELAY" directive, this enables the update service.

[bash]# vi /etc/sysconfig/freshclam
FRESHCLAM_MOD=180                <-- Update interval in minutes
#FRESHCLAM_DELAY=disabled-warn   <-- Add "#" to activate (enable) clamav updates

The main antivirus functions are now running and freshclam can be configured to keep the system up to date with the latest virus patterns. As normal the configuration file should be backed up.

[bash]# cp /etc/freshclam.conf /etc/freshclam.conf.original
[bash]# vi /etc/freshclam.conf

This is an example of the freshclam configuration file. This file too is well documented and the supporting man page should be consulted for any further queries. Type "man freshclam.conf" at the command prompt.

If your system is behind a firewall or you prefer to use a proxy server, then ensure that you complete and enable the appropriate HTTPProxy options for your system.

#Example
DatabaseOwner clamav
DatabaseDirectory /var/lib/clamav
Checks 24
MaxAttempts 5
UpdateLogFile /var/log/freshclam.log
DNSDatabaseInfo current.cvd.clamav.net
DatabaseMirror db.??.clamav.net               ###  <-- See Note.
DatabaseMirror database.clamav.net
#HTTPProxyServer galaxy.example.com
#HTTPProxyPort 3128
#HTTPProxyUsername username
#HTTPProxyPassword password

Note !! Replace "??" (above) with the two letter country code for your region, or remove line from configuration file if you are unsure.

Freshclam will automatically update itself at intervals define with the earlier "FRESHCLAM_MOD" declaration. However, if you require to update your antivirus manually to test your configuration, you can execute the command at the xterm prompt.

[bash]# freshclam
ClamAV update process started at Sun May 21 12:01:57 2006
main.cvd is up to date (version: 38, sigs: 51206, f-level: 7, builder: tkojm)
Downloading daily.cvd [*]
daily.cvd updated (version: 1472, sigs: 4793, f-level: 8, builder: arnaud)
Database updated (55999 signatures) from db.au.clamav.net (IP: 61.8.0.16)

Freshclam keeps a log of all update details.

[bash]# tail /var/log/freshclam.log

SpamAssassin

One of the biggest wastes of bandwidth throughout the Internet today is from unwanted/unsolicitated bulk email, in other words spam; we hate it. Unfortunately for us it isn't going to go away for a while, fortunately we can at least filter some of it out of our Inbox by using SpamAssassin which comes standard now with many Linux distributions. SpamAssassin will at the very least allow you to take some control back over your email account.

To install SpamAssassin and the required milter for Sendmail, type the following command at the prompt.

[bash]# yum install spamass-milter spamassassin

SpamAssassin has many plugins written and available for use, they can be enabled in the "/etc/mail/spamassassin/v310.pre" plugin file. The plugins file is the first configuration to be loaded by SpamAssassin.

[bash]# cp /etc/mail/spamassassin/v310.pre /etc/mail/spamassassin/v310.pre.original
[bash]# vi /etc/mail/spamassassin/v310.pre
loadplugin Mail::SpamAssassin::Plugin::DCC
loadplugin Mail::SpamAssassin::Plugin::Pyzor
loadplugin Mail::SpamAssassin::Plugin::Razor2
loadplugin Mail::SpamAssassin::Plugin::SpamCop
loadplugin Mail::SpamAssassin::Plugin::AWL
loadplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold
loadplugin Mail::SpamAssassin::Plugin::WhiteListSubject
loadplugin Mail::SpamAssassin::Plugin::MIMEHeader
loadplugin Mail::SpamAssassin::Plugin::ReplaceTags

The "/etc/mail/spamassassin/local.cf" file is the main global configuration file for the whole server and can be configured like so.

[bash]# cp /etc/mail/spamassassin/local.cf /etc/mail/spamassassin/local.cf.original
[bash]# vi /etc/mail/spamassassin/local.cf
required_score          5.0
rewrite_header subject  [SPAM]
report_safe             1
use_bayes               1
use_bayes_rules         1
bayes_auto_learn        1
skip_rbl_checks         0
use_razor2              1
use_dcc                 1
use_pyzor               1
trusted_networks        192.168.1/24 127/8
internal_networks       192.168.1/24 127/8

Note !! If you need assistance in determining which configuration options are the best for your system, you can use the online "SpamAssassin Configuration Generator" located at: "http://www.yrex.com/spam/spamconfig.php". your customised online configuration can then be downloaded into your /etc/mail/spamassassin/local.cf file.

To define daemon runtime options, edit the system configuration file for SpamAssassin.

[bash]# cp /etc/sysconfig/spamassassin /etc/sysconfig/spamassassin.original
[bash]# vi /etc/sysconfig/spamassassin
SPAMDOPTIONS="-d -c -l -m5 -H"

Users can also fine tune their own options for SpamAssassin by editing their own user configuration files located in their home drives.

[/home/miles]$ vi ~/.spamassassin/user_prefs             <-- executed as basic user

The SpamAssassin mail filter (milter) can then be configured with runtime options.

[bash]# cp /etc/sysconfig/spamass-milter /etc/sysconfig/spamass-milter.original
[bash]# vi /etc/sysconfig/spamass-milter
SOCKET=/var/run/spamass-milter/spamass-milter.sock
EXTRA_FLAGS="-r 15"

The daemons need to be configured to start at the appropriate runlevels using the following commands, this should also be checked to ensure the daemons will initialise at expected if needed at next reboot.

[bash]# chkconfig --level 2345 spamassassin on
[bash]# chkconfig --level 2345 spamass-milter on
[bash]# chkconfig --list spamassassin
[bash]# chkconfig --list spamass-milter

Once the services has been configured, the daemons can both be restarted.

[bash]# /etc/init.d/spamassassin restart
[bash]# /etc/init.d/spamass-milter restart

The SpamAssassin configurations have now been completed, however the mail server has not yet been configured to use the spam filtering daemon yet. Before we do the final configuration, it is important to firstly test that SpamAssassin is functioning as expected and the email that passes through the daemon is handled correctly. If we don't do these tests then some of your email may just disappear because of a poorly configured service.

The first test passes a test email to the spam daemon and returns the results to the screen. This test email is considered by SpamAssassin to be clean and should return a clean result.

[bash]# spamassassin -t < /usr/share/doc/spamassassin-3*/sample-nonspam.txt | grep X-Spam
X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on galaxy.example.com
X-Spam-Level:
X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=unavailable

The second test passes an email that is considered by SpamAssassin to contain spam signatures, as such the test should return a positive result to the test. You should note the "X-Spam-Flag: YES" flag that SpamAssassin inserts into the email's headers, this will allow users an easily way to identify and automatically sort spam affected emails into junk email folders as they arrive in their Inboxes.

[bash]# spamassassin -t < /usr/share/doc/spamassassin-3*/sample-spam.txt | grep X-Spam
X-Spam-Flag: YES
X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on galaxy.example.com
X-Spam-Level: **************************************************
X-Spam-Status: Yes, score=1000.0 required=5.0 tests=GTUBE,NO_RECEIVED,

Once you are happy that the outcomes of both your spam and non-spam tests are successful, Sendmail can be configured to pass emails to the SpamAssassin daemon by use of the mail filter (milter).

[bash]# vi /etc/mail/sendmail.mc
INPUT_MAIL_FILTER(`spamassassin', `S=unix:/var/run/spamass-milter/spamass-milter.sock, F=, T=C:15m;S:4m;R:4m;E:10m')dnl
define(`confMILTER_MACROS_CONNECT',`t, b, j, _, {daemon_name}, {if_name}, {if_addr}')dnl
define(`confMILTER_MACROS_HELO',`s, {tls_version}, {cipher}, {cipher_bits}, {cert_subject}, {cert_issuer}')dnl

Now that the SpamAssassin milter settings have been inserted into the Sendmail configuration, the Sendmail server can be restarted; this will complete your installation.

[bash]# make -C /etc/mail
[bash]# /etc/init.d/sendmail restart

For further information on SpamAssassin configuration, you can view the following man pages: "Mail::SpamAssassin::Conf" and "Mail::SpamAssassin".

REFERENCE

http://www.brennan.id.au/12-Sendmail_Server.html