Friday, June 18, 2010

Openx Handbook Openx Server Optimization And Performance Tuning

Category: — SkyHi @ Friday, June 18, 2010
Openx Handbook : Openx Server Optimization And Performance Tuning
Disclaimer : Please notices this document will help you for configuring an Openx add server. This article is licensed under Creative Commons Attribution-Noncommercial 2.5 India .Please don’t copy and paste the configurations to your server if it is not the hardware that mentioned in the hardware requirements.

1. Introduction

1.1 What is Openx?

2. Requirements

2.1 Hardware Requirements
2.2 Software Requirements

3. Hardware Optimization

3.1 Port Connection Checking
3.2 I/O wait optimization
3.3 Understanding the Processor

4. Operating System and Service Hardening

4.1 Useless Package Removal
4.2 Useless Service Removal
4.3 Sysctl Configuration

5. Apache, PHP Configuration And Performance Tuning

5.1 Apache configuration
5.2 PHP configuration
5.3 Stripping and Tuning

6. MySQL Configuration and Performance Tuning

7. Server Monitoring And Service Tweaking

8. Introduction to Advanced Openx Servers

Appendix-A About The Author

Appendix-B License

Appendix-C FAQ

1. Introduction

This document will help you to configure and build a completely optimized Openx add server. This is a completely tested configuration and the methods used in this article is fully for a Linux( some for unix) operating system. Please don’t copy and paste the configurations into you’r server if you don’t have the software and hardware requirements mentioned in this document. This will help you to build and optimize your openx server upto 99% stability and performance and the rest depends on who configuring it.

Please note don’t use these configuration for normal shared hosting server, because it is only for Openx add servers. As per these configurations our add servers having load between 0 and 1 , and serving 3.6 million requests per minute. This is they way how we providing 70% of add in Croatia ( Zagrab ). I would like to share the server optimization and performance tuning techniques to the Openx community.

Also not I am not telling about how to secure your server, but how to optimize it.

1.1 What is Openx?

I think you know what is Openx otherwise you no need to look into this article. Am I right? If not please check the website to know more about openx. In short Openx is a community project for adds management and it is a nice application.

2. Requirements

This the requirements of our servers. I think this is not a bad server. Please remember don’t copy and paste the settings if you are not using these type of settings.

2.1 Hardware Requirements.

See how our box looks like.

  • Processor - 2X Quad core Xeon E5320
  • RAM - 16 GB ,
  • SWAP - 16 GB /

  • HDD - 8 x 73 SAS 10.000 rpm in RAID 10

2.2 Software Requirements

We are using centos 5.2 64bit Operating system with Cpanel. If you think why need for cpanel?, because cpanel have good scripts for building apache , php and mysql in the server. Also it is the best control panel for hosting servers. Please see the complete software requirements below,

  • Operating System - Centos 5.2 64 bit
  • Control panel - Cpanel
  • Webserver - Apache 2.2.9
  • PHP - 5.2.6
  • Mysql - 5.0

In general any LAMP environment is good. Even you can use Lighttpd as webserver for single add servers. If you are not using the Cpanel please use Lighttpd webserver.

3.Hardware Optimization

It an important thing and the first step before building the server.

3.1 Port Connection Checking

First you can check the NIC speed. It is better to connect your server to a 1000 MBit Port. You can check the NIC as follows.

# ethtool eth0

Here eth0 is NIC . Please notice the following values from the above report
Speed: You can see the transfer speed ( eg : 1000Mb/s )
Duplex: Full ( It must be Full duplex connection )
Auto-negotiation: on

Now please check the current port speed as follows

# mii-tool  -v

Here you can see some values as follows,

eth0: negotiated 100baseTx-FD, link ok  .

This means your server is connected to a 100 Mbit Full Duplex Stream.

3.2 I/O wait optimization

For a high traffic server the I/O wait is a major problem. I used the 8 SAS disks in a RAID10 array for getting the performance. I done the HDD partitioning as follows,
/ - For installing the OS
/home - This is the cpanel user home directory. It contains the files
/usr - It contains the OS components
/var - It is mainly for the mysql database
/tmp - It is the temp folder, which is for creating temporary files
/backup - This is my backup disk

Now open your file system table ( /etc/fstab ) add the following flags to /, /var and /home partitions.

 noatime

“noatime” a file system has been mounted with this option, reading accesses to the file system will no longer result in an update to the atime information associated with the file. So it is a benefit. An example is given below

 LABEL=/var              /var                    ext3    defaults,usrquota,noatime        1 2

NOTE: After modifying the value you may either need to reboot the system or need remount the disk partition to do this changes. I remounted the /var partition as follows 

# mount -o remount  /var

If you want to check the partition is mounted properly do this check as follows

 # mount

Now you can see the /var is showing “notime” option.
Now you need to check DMA parameter of the HDD , in normal disks you can use the command “hdparm” . But hdparm not giving SCSI or SATA disks information. So there is a utility called ” sdparm ” . You cam use it for checking the Direct Memory Access ( DMA ). It must be enabled.

3.3 understanding the Processor

You will get all processor information from the file /proc/cpuinfo . If you need to know how many processors are available in your server , please do the following command.

# grep processor /proc/cpuinfo | wc -l

It will show the number of physical/virtual processors. If you see a count of 8 then server load less than 8 is normal.
This is all about hardware optimization

4.Operating System and Service Hardening

Next is our software optimization part. This is a very important part. I am here giving an example of a Centos Operating system with Cpanel as a control panel.

4.1 Useless Package Removal / Os hardening

First you may need to remove the USELESS packages from the server. For a server operating system in Centos/RHEL arong 350 to 400 packages required. But by default it coming with 700+ rpms. You can find the rpm’s as follows ,

# rpm -qa

From the above list remove the useless rpm’s as ,

# rpm -e

Please not if you removing a System package it will corrupt the OS. So only do it if you know about the packages. Note , you are warned.

4.2 Useless Service Removal

So now we have removed a set of packages. So now we need to stop a lot useless services, which will use our server resources . You can stop the server as follows ,

# service stop <>

eg : service cups off
Please you need to disable it during to avoid it from starting during the reboot/booting time as follows ,

 # chkconfig  <> off

eg : chkconfig cups off<> Also you can verify that service is off or not as follows ,

 #    chkconfig  --list   <>
  eg : # chkconfig  --list cups
   cups            0:off   1:off   2:off   3:off   4:off   5:off   6:off

So please remove the following services from the server as mentioned above.

 cups ,    xfs , atd , nfslock, canna, FreeWnn,  cups-config-daemon, iiim , mDNSResponder, nifd, rpcidmapd, bluetooth, anacron,    gpm, saslauthd, avahi-daemon, avahi-dnsconfd , hidd, pcscd,   sbadm ,  webmin  and ossec

Note : If your server is not a normal production server, you may be using some of the above services.
Also please remove any firewalls like apf or csf from your server. Because those may simply block highly accessible ip’s .

4.3 Sysctl Configuration

So the next step is optimizing the sysctl. You can tune you kernel via sysctl. To know your current sysctl parameter please do the following,

# sysctl -a

If you want to know all parameters starting with vm please use the command as follows,

 # sysctl vm

Also now you can change any particular sysctl parameter value as follows.

# sysctl  -w variablename=value

But notice your changes only be affect current settings. If you wish to add those values as permanent please include it in /etc/sysctl.confi
You can see a tuned sysctl configuration as follows, it is using in our server having the above configuration.

# Disables packet forwarding
net.ipv4.ip_forward=0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Enable Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.eth0.log_martians = 0

# Disables IP source routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.lo.accept_source_route = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0

# Enable IP spoofing protection, turn on source route verification
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.lo.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# Disables the magic-sysrq key
kernel.sysrq = 0

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for tcp_keepalive_time connection
net.ipv4.tcp_keepalive_time = 1800

# Turn off the tcp_window_scaling
net.ipv4.tcp_window_scaling = 0

# Turn off the tcp_sack
net.ipv4.tcp_sack = 0

# Turn off the tcp_timestamps
net.ipv4.tcp_timestamps = 0

# Enable TCP SYN Cookie Protection
net.ipv4.tcp_syncookies = 1

# Enable ignoring broadcasts request
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Enable bad error message Protection
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Log Spoofed Packets, Source Routed Packets, Redirect Packets
net.ipv4.conf.all.log_martians = 1

# Increases the size of the socket queue (effectively, q0).
net.ipv4.tcp_max_syn_backlog = 1024

# Increase the tcp-time-wait buckets pool size
net.ipv4.tcp_max_tw_buckets = 1440000

# Allowed local port range
net.ipv4.ip_local_port_range = 16384 65536

Note: If you don’t know nothing about sysctl please leave it.

5.Apache and PHP Configuration And Performance Tuning

In this section you can see how to get a highly configured apache and php for better performance. Please not I am here using some commands and scripts related with the cpanel. But the building flags are coming with the sources itself.
Please use the following script to start the build process.

# /scripts/easyapache

Now select the Option “Basic” from the menu and ” start customization based on the profile ” . Select the Apaceh version 2.2 and php 5 from the coming menus. Remember don’t use pache 1.3 or apache 2.0.
Now please proceed with the Apache configuration section as follows.

5.1 Apache configuration

Please go to the ” Exhaustive Options” selections menu and enabled the following Apache modules.

        1) Asis
       2) Auth Digest
       3) AuthDefault
       4) Cache
       5)Casefilter
       6) DBD
       7)Deflate
       8)Disk cache
       9)Distcache
       10)Env
       11)Expires
       12)Mod fcGID
       13)File cache
       14) FileProtect
       15)Frontpage
       16)Headers
       17)MPM Prefork
       18) Memcache
       19) Proxy
       20) UniqueID
       21) usertrack
       22) Vhost Alias

Now from the other Modules section please select the following,

   1) EAccelerator
      2) IoncubeLoader
      3) Mod Security
      4) Suhosin
      5) Zend Optimizer

NOTE: Please don’t use SUPH configuration in Openx Add server. It will eat up your server resources. The apache mod_php is the best configuration.
Now please chose the following option from PHP or enable these options.

5.2 PHP configuration

   1)  Bcmath
  2) calendar
  3) Concurrent DSO Patch
  4) Curl, Curlssl, Curl wrappers
  5) Dbase
  6) Exif , FTP, GD, Gettext , Iconv
  7) Imap, MM, magicQuotes,  mailHeaders , Mregex, Mbstring , mcrypt , mhash , Mime
   8) Mysql of the System , Mysqli
   9)Openssl , PDO , PDO mysql
   10) POSIX, Pspell,  SOAP, Safemod
   11) TTF, Tidy, Wddx, XSL , XMLRPC, Zend , Zlib

Now enable the option reset apache configuration to default and then proceed to next step , save and build. It will take some long time to complete the build process.
So after completing the build process, please make sure apache is up and running. Now proceed to step 5.3

5.3 Stripping and tuning

This our configuration section. First we need to strip our binaries for getting light foot print. You can use the strip utility as follows.

#strip  -s  <>

eg : strip -s /usr/local/bin/php
Please strip the following binaries.

  1) /usr/local/bin/php
  2) /usr/bin/php
  3) /usr/local/apache/bin/httpd
  4) All binaries under the folder /usr/local/apache/modules/

Now it is the time to tune the apache. We have enabled a lot caching moduels and an MPM preforker module. Now we need to configure the MPM parameters. After finishing the build process the httpd-mpm configurations can be found under the extra folder in apache installations prefix ( /usr/local/apache/conf/extra/ ) . But it is not included in the httpd.conf . So open the file /usr/local/apache/conf/httpd.conf and add the following line.

 Include "/usr/local/apache/conf/extra/httpd-mpm.conf"

Now open the file /usr/local/apache/conf/extra/httpd-mpm.conf and modify the preforker module sections as follows ,

    StartServers          30
   MinSpareServers       5
   MaxSpareServers      50
   ServerLimit         2000
   MaxClients          1800
   MaxRequestsPerChild   0

Now please reduce the “Timeout” value in /usr/local/apache/conf/httpd.conf . Buy default it have value 300. Please reduce it to 100. It will be enough for Openx request.
So we completed modification of apache configurations. Please run the following command if you are in a cpanel server for updating the cpanel apache templates.

 #   /usr/local/cpanel/bin/apache_conf_distiller --update

Now open the php.ini file, we need to tweak it. In general the php.ini will be /usr/local/lib/php.ini. Please open this file using your favorite editor.

 # vi /usr/local/lib/php.ini

Now set the following values.

 memory_limit = 128M
max_execution_time= 30
register_globals = Off
mysql.allow_persistent = On
mysql.allow_persistent = On
mysql.trace_mode = Off

This is all about apache and php tuning . Now restart apache as follows.

# /etc/init.d/httpd  restart

Now we need to configure MySQL server. Please proceed with next step.

6. MySQL Configuration and Performance Tuning

The global MySQL configuration is /etc/my.cnf. You may need to configure the mysql variables. But its memory variable and threading variables depends on your server RAM and number of processors. So the mysql configurations vary from server to server. Please see our my.cnf for the above hardware. Please don’t copy and paste the values. Take advices from your experts. Only an experienced guy can help you for tuning your mysql server. Any way I hope the following my.cnf will help you.

[mysqld]
safe-show-database
old-passwords = 1
max_connections =2048
max_user_connections = 1024
key_buffer_size = 2048M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
read_buffer_size = 1M
sort_buffer_size = 2M
table_cache = 4000
thread_cache_size = 384
wait_timeout = 20
connect_timeout = 10
tmp_table_size = 2048M
max_heap_table_size = 512M
max_allowed_packet = 64M
net_buffer_length = 16384
max_connect_errors = 10
thread_concurrency = 16
concurrent_insert = 2
table_lock_wait_timeout = 30
read_rnd_buffer_size = 786432
bulk_insert_buffer_size = 8M
query_cache_limit = 7M
query_cache_size = 64M
query_cache_type = 1
query_prealloc_size = 262144
query_alloc_block_size = 65536
transaction_alloc_block_size = 8192
transaction_prealloc_size = 4096
max_write_lock_count = 16
long_query_time = 5
skip-name-resolve
skip-locking

[mysqld_safe]
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 384M
sort_buffer = 384M
read_buffer = 256M
write_buffer = 256M

Now please proceed with the following section.

7.Server monitoring And Service Tweaking

This is an important section. you need to monitor your server at least 48 hours to know the mysql and apache access limits. After that you can change the apache MPM configurations and mysql /etc/my.cnf variables. A perfect tuning will help you to get 100% uptime ( in service not in hardware ). So this is all about an Openx server building procedure.

8.Introduction to Advanced Openx Servers

This is an advanced section. Here I can give you some solutions for setting up advanced mysql servers. You can setup cluster for hosting your add server. You can chose any of the following server configurations for setting up an openx cluster Group

1) Simple cluster
It need the following configurations.
Load balancer – 1
Web server – 2
Database Server – 1
So a total of three Servers and one load balancer.
2) Redundancy clusters
It need the following hardwares,
Load balancer – 2
Web servers – 3
Database Servers – 2
So a total of 5 servers and 2 load balancers
3) Large Scale HA Cluster
These type of clusters are using in our high traffic add servers. This is little expensive, but give you high stability and uptime. Please see the hardware requirements below.
Hardware Firewall – 2
Load balancer – 2
Web servers – 3
Database servers – 3 File Servers - 2
So total of 7 servers , 2 load balancers and 2 hardware firewalls.
But the configurations of these type of openx clusters are out of the scope of this article. Some one is calling me now. So I will write those cluster configurations in a new article, if i get time.

Appendix-A About The Author

My Name : Sherin A
My Web Site : http://www.sherin.co.in/
If you wan’t my designation , you can call me, System Engineer , Security Expert , Software Engineer or a FOSS member.
About me :
1) Founder of vcPanel ( Virtual Private Server Control panel )
2) Founder of FOSS award winner project ffmpegistaller )<> 3) Founder of FossBlog ( A Free and Open Source Software Blog
My relation with Adds : I am here with adds since 2003. Now providing Openx services for companies US , Canad, UK, Australia, Japan , India ,Germany, Croatia and Russia.
Contact Me : All my contacts are available here or send email to me sherinmon[at]gmail[dot]com

Appendix-B License

This article is licensed under Creative Commons Attribution-Noncommercial 2.5 India It means you can share and redistribute it , but give a link back to this original document, because sometimes I will add more tips to this document.

Appendix-C FAQ

1) Is any other tool o tips available ?
Yes, sure. You can use the Apache bench ( ab ) tool for testing your apache sure. Also an expert can optimize your server easily .
2) Where can I find a server with the above type configurations?
You can contact me. I am not a webhost, but I know where to host an Openx Server.
3) I need to configure an Openx cluster server. but how ?
Yes you can contact me. But not sure I have enough time to help you. Even though i will surely reply to all my emails.
4) My questions is not here so what I do ?
Again please send an email to me.

REFERENCES

http://www.sherin.co.in/openxhandbook/