Friday, January 22, 2010

A Journey On DRBD For MySQL

Category: , , , — SkyHi @ Friday, January 22, 2010
Alhamdulillah, I'm successfully install and deploy DRBD for MySQL, yet another Linux Cluster.

Prologue

The objective is to deploy a high-availability MySQL database cluster to use with Desknow. After getting frustrated (I mean really frustrated!) with MySQL Cluster and Continuent uni/cluster solution, I decided to try DRBD for MySQL. Thanks to MySQL Newsletter May 2007, which is mentioning about DRBD for MySQL.

Scenario

drbd_4_mysql.png

Server Function IP Address Operating System MySQL version
server1 MySQL server (primary) x.x.x.x RHEL 5 5.0.22-2.1
server2 MySQL server (secondary) y.y.y.y RHEL 5 5.0.22-2.1
desknow Desknow Application z.z.z.z RHEL 4 U5 -

Legend:
server1.tld = FQDN for server1
server2.tld = FQDN for server2
a.a.a.a = virtual IP for the heartbeat service
b.b.b.b = Gateway address for server1 and server2
x.x.x.x = IP address for server1
y.y.y.y = IP address for server2
z.z.z.z = IP address for desknow
?.?.?.? = broadcast address for a.a.a.a

Pre-Requisites

DRBD Configuration

1. Download and install latest DRBD on server1,
[root@server1 ~]# wget http://oss.linbit.com/drbd/8.0/drbd-8.0.3.tar.gz
[root@server1 ~]# cd /tmp/
[root@server1 tmp]# tar czvf /root/drbd-8.0.3.tar.gz
[root@server1 tmp]# cd drbd-8.0.3
[root@server1 drbd-8.0.3]# make tools
[root@server1 drbd-8.0.3]# make install
2. Test it,
[root@server1 drbd-8.0.3]# modprobe drbd
[root@server1 drbd-8.0.3]# cat /proc/drbd
version: 8.0.3 (api:86/proto:86)
SVN Revision: 2881 build by root@server1.tld, 2007-06-04 10:11:24
3. Repeat Step 1 & Step 2 for server2.
4. Configure
Edit /etc/drbd.conf on both server1 and server2,
HILITED HTML
CODE:
On server1,
[root@server1 etc]# drbdadm create-md mysql
[root@server1 etc]# drbdadm adjust mysql
[root@server1 ~]# drbdsetup /dev/drbd0 primary -o
[root@server1 ~]# service drbd start
[root@server1 ~]# service drbd status
On server2,
[root@server1 ~]# service drbd start
[root@server1 ~]# service drbd status

MySQL Configuration

Nothing much to talk about MySQL, I'm just using default packages (and default configurations) from RHEL 5.

Heartbeat Configuration

1. Installation
Ok, RHEL does not include Heartbeat. It sucks, isn't it?
It's not a big deal, though. Just get it from Linux-HA website.
Pre-compiled RPM is available from Linux-HA Download Site. Nevertheless, I've got problem to use it. Hence, I chose to build my own RPM package.
I need these 2 packages in order to compile the heartbeat source,
[root@server1 x86_64]# yum install pam-devel swig
Download the heartbeat RPM source package and install it,
[root@server1 ~]# wget http://linux-ha.org/download/heartbeat-2.0.8-1.src.rpm
[root@server1 ~]# rpm -ivh heartbeat-2.0.8-1.src.rpm
Build RPM package,
[root@server1 ~]# cd /usr/src/redhat/SPECS/
[root@server1 SPECS]# rpmbuild -bb heartbeat.spec
Install particular packages,
[root@server1 SPECS]# cd /usr/src/redhat/RPMS/x86_64/
[root@server1 x86_64]# rpm -ivh heartbeat-pils-2.0.8-1.x86_64.rpm
[root@server1 x86_64]# rpm -ivh heartbeat-stonith-2.0.8-1.x86_64.rpm
At first, the installation of heartbeat failed.
HILITED HTML
CODE:
But it is ok for the second time.
HILITED HTML
CODE:
2. Configuration
Create ha.cf, haresources and authekeys files by copying from /usr/share/doc/heartbeat-2.0.8/
[root@server1 ~]# cd /etc/ha.d/
[[root@server1 ha.d]# cp /usr/share/doc/heartbeat-2.0.8/ha.cf .
[root@server1 ha.d]# cp /usr/share/doc/heartbeat-2.0.8/haresources .
[root@server1 ha.d]# cp /usr/share/doc/heartbeat-2.0.8/authkeys .
Edit haresources,
HILITED HTML
CODE:
Edit ha.cf,
HILITED HTML
CODE:
Edit authkeys,
HILITED HTML
CODE:
Edit /etc/ha.d/resource.d/drbddisk
HILITED HTML
CODE:
Copy MySQL startup script,
[root@server1 ~]# cp /etc/rc.d/init.d/mysqld /etc/ha.d/resource.d/
Copy all relevant files from server1 to server2
[root@server1 x86_64]# scp -v heartbeat-* root@server2.tld:~
[root@server1 ha.d]# scp -v authkeys ha.cf haresources root@server2.tld:/etc/ha.d/

Integration & Testing

1. Stop all services,
[root@server1 ~]# service drbd stop
[root@server1 ~]# service mysqld stop
[root@server1 ~]# service heartbeat stop
[root@server2 ~]# service drbd stop
[root@server2 ~]# service mysqld stop
[root@server2 ~]# service heartbeat stop
2. Configure particular services to enable/disable automatic startup,
Please take note that mysqld should be off during startup, while drbd and heartbeat should be on.
[root@server1 ~]# chkconfig drbd on
[root@server2 ~]# chkconfig drbd on
[root@server1 ~]# chkconfig mysqld off
[root@server2 ~]# chkconfig mysqld off
[root@server1 ~]# chkconfig heartbeat on
[root@server1 ~]# chkconfig heartbeat on
3. Start drbd on both machines
[root@server1 ~]# service drbd start
[root@server2 ~]# service drbd start
4. Start heartbeat on both machines
[root@server1 ~]# service heartbeat start
[root@server2 ~]# service heartbeat start
server1 should get a virtual IP and become primary,
HILITED HTML
CODE:
5. Desknow Installation
Install Desknow on desknow machine, the MySQL database should be pointed to virtual IP, which is a.a.a.a.

Fail-over Testing

1. Stop heartbeat service on server1
[root@server1 ~]# service heartbeat stop
At server1, MySQL service has stopped while DRBD becomes secondary,
HILITED HTML
CODE:
At server2, you can verify that MySQL service is running while DRBD becomes primary.
HILITED HTML
CODE:
Output from /var/log/ha-log (server1),
HILITED HTML
CODE:
Output from /var/log/ha-log (server2),
HILITED HTML
CODE:
2. Desknow state
Test whether Desknow application is still running properly or not. In my case, it still good as before. Hence, the fail-over test is succeed.
3. Restart heartbeat service on server1
mysqld service should be running on server1 (and should be stopped on server2).
HILITED HTML
CODE:
Output from /var/log/ha-log (server1),
HILITED HTML
CODE:
Output from /var/log/ha-log (server2),
HILITED HTML
CODE:

Conclusion

  • I can achieve a penny-less MySQL Database Cluster by using DRBD for MySQL solution. If the price is reasonable, might be subscribed to MySQL professional service.
  • x86_64 architecture proves to be well-integrated with DRBD for MySQL




References
http://blog.irwan.name/?p=118