Friday, June 18, 2010

Tuning CentOS 5.x x64 for iSCSI Performance

SkyHi @ Friday, June 18, 2010

In order to get the best performance from CentOS and the iSCSI deployment we recommend the following tweaks. These tweaks will give you the best performance for your iSCSI deployments. The network configuration can be used with NFS as is. Click to read more...

Add the following to your /etc/sysctl.conf:

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_syncookies = 1

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.ipv4.tcp_rmem = 4096 87380 16777216

net.ipv4.tcp_wmem = 4096 65536 16777216

net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_moderate_rcvbuf = 1

net.core.netdev_max_backlog = 2500

Run the following to modify the TX queue of the ethernet adapters:

ifconfig eth1 txqueuelen 1000

(Only do this for 1GB links)

Modify the MTU on the ethernet adapters:

Add the following line to the IF configuration files: (located at /etc/sysconfig-network-scripts/ifcfg-eth*)


Add the following to your ietd.conf (iSCSI configuation files):

InitialR2T No

ImmediateData Yes

xMaxRecvDataSegmentLength 262144

iSCSI ietd.conf Optimizations

If you have a hardware RAID controller for the data set you will be exporting via iSCSI you can use the configuration called 'blockio'. This setting will allow the RAID card to do all the caching and buffering that it needs. If you are using Software RAID or 'fake' RAID you should use the type 'fileio' this will allow the iSCSI service to use the processor and built in RAM to do caching and buffering.

The next important setting is the IOmode (this setting is ignored with Blockio and only effective in fileIO); if you set this to WB you will have data loss in the event of a power outage that does not allow the iSCSI service to gracefully terminate and flush the cache. USE WITH CAUTION.

Here is a sample ietd.conf LUN definition:

Lun 1 Type=blockio,Path=/dev/ESX_VG1/esx_iscsi_1,IOMode=wb,ScsiId=1,ScsiSN=1

You notice the use of Volume Groups here, this is to allow snapshots of
the LUN (but more on that in a later posting).