High availability refers to the practice of keeping online resources available through node failure or system maintenance. This guide will demonstrate a method for using two Linodes to provide file and database services with NFS, PostgreSQL, and MySQL. Services will be avilable even when one node is powered off or put into standby mode. IP failover, Heartbeat 3.x, Pacemaker 1.x, DRBD (Distributed Replicated Block Device), PostgreSQL, and MySQL will be used for this example configuration.
This setup is intended to provide a file and database server that offers services to other Linodes in the same datacenter. Other nodes could serve as web frontends, mounting content directories over NFS and accessing databases hosted on the highly available server. A simple network diagram illustrating such a setup might look like this:
In the diagram above, a highly available load balancer comprised of two nodes distributes inbound HTTP connections to multiple frontend web servers. These web servers connect to a highly available database and file server comprised of two nodes. While this guide only covers configuration of the file and database server portion of the diagram, future guides will cover integrating the other components into a complete highly available and load balanced network configuration.
As high availability is a complex topic with many methods available for achieving various goals, it should be noted that the method discussed here may not be appropriate for some use cases. However, it should provide a good foundation for developing a customized HA solution.
Contents
Throughout this document, the following terms are used:
You should substitute your own values for these terms wherever they are found.
This guide assumes you have two active Linodes, and that both are freshly deployed Ubuntu 10.04 LTS (Lucid) instances. If you currently only have one Linode, you can add another in Linode Manager by clicking the Linodes tab, and then clicking the Add a Linode link.
Both Linodes must reside in the same datacenter for IP failover to work. Future HA guides will address combining the principles demonstrated in this tutorial with cross-datacenter clustering techniques.
When you deploy your Linodes, be sure not to allocate all the available disk space to the main disk images. As part of this tutorial, you'll be creating three additional images on each Linode, so be sure to leave at least 2 GB free when deploying Ubuntu 10.04 to each. You may wish to leave more free disk space, depending on your needs. The additional disk images will be used to store web application and database data.
Each Linode must have a private IP address assigned. For instructions, see Adding Private IP Addresses.
Note
You'll need to open a support ticket requesting an additional private IP address for the primary Linode (to serve as a "floating" address). Once your primary Linode has been allocated a second private IP, reboot both Linodes to allow the new IP addresses to be properly routed.
Choose one Linode to serve as the "primary" node. Log into it via SSH as root and edit its /etc/hosts file to resemble the following:
File: /etc/hosts (on primary Linode)
127.0.0.1 localhost.localdomain localhost 12.34.56.78 ha1-db.example.com ha1-db 98.76.54.32 ha2-db.example.com ha2-db
Remember to substitute your primary and secondary Linode's IP addresses for 12.34.56.78 and 98.76.54.32, respectively, along with appropriate hostnames for each. You will find the IP addresses for your Linodes on their "Remote Access" tabs in the Linode Manager.
For the sake of simplicity, it is recommended that you keep the short hostnames assigned as ha1-db and ha2-db. Next, issue the following commands on the primary (ha1-db) Linode to generate SSH keys for the root user on each VPS, synchronize their SSH host keys, set their hostnames, and allow passwordless logins from each to the other. SSH host key synchronization will prevent issues with key checking later on, which might otherwise occur should you need to perform an SSH login via a hostname pointing to a floating IP while the secondary node is serving your content. You will be prompted to assign passphrases to the SSH keys; this is optional, and you may skip this step by pressing the "Enter" key.
ssh-keygen -t rsa scp ~/.ssh/id_rsa.pub root@ha2-db:/root/ha1_key.pub ssh root@ha2-db "ssh-keygen -t rsa" ssh root@ha2-db "echo \`cat ~/ha1_key.pub\` >> ~/.ssh/authorized_keys2" ssh root@ha2-db "rm ~/ha1_key.pub" scp root@ha2-db:/root/.ssh/id_rsa.pub /root cat ~/id_rsa.pub >> ~/.ssh/authorized_keys2 rm ~/id_rsa.pub scp /etc/ssh/ssh_host* root@ha2-db:/etc/ssh/ rm ~/.ssh/known_hosts ssh root@ha2-db "/etc/init.d/ssh restart" scp /etc/hosts root@ha2-db:/etc/hosts echo "ha1-db" > /etc/hostname hostname -F /etc/hostname ssh root@ha2-db "echo \"ha2-db\" > /etc/hostname" ssh root@ha2-db "hostname -F /etc/hostname"
By default, when Linodes are booted DHCP is used to assign IP addresses. This works fine for cases where a Linode will only have one IP address, as DHCP will always assign that IP to the Linode. If a Linode has or may have multiple IPs assigned to it, an explicit static configuration is required, as is the case with this configuration.
On the primary Linode, edit the /etc/network/interfaces file to resemble the following, making sure the values entered match those shown on the Remote Access tab for the primary Linode. The private IP address 192.168.88.88 should be changed to reflect the first private IP assigned to the primary Linode.
File: /etc/network/interfaces (on primary Linode)
auto lo iface lo inet loopback auto eth0 iface eth0 inet static address 12.34.56.78 netmask 255.255.255.0 gateway 12.34.56.1 auto eth0:0 iface eth0:0 inet static address 192.168.88.88 netmask 255.255.128.0
Issue the following command to restart networking on the primary Linode:
/etc/init.d/networking restart
On the secondary Linode, edit the /etc/network/interfaces file to resemble the following, making sure the values entered match those shown on the Remote Access tab for the secondary Linode:
File: /etc/network/interfaces (on secondary Linode)
auto lo iface lo inet loopback auto eth0 iface eth0 inet static address 98.76.54.32 netmask 255.255.255.0 gateway 98.76.54.1 auto eth0:0 iface eth0:0 inet static address 192.168.99.99 netmask 255.255.128.0
Issue the following command to restart networking on the secondary Linode:
/etc/init.d/networking restart
You should be able to ping each Linode's private address from the other. If you can't, review your network configuration for errors.
Navigate to the Remote Access tab for the primary Linode and make a note of its second private IP address. Next, navigate to the Remote Access tab for the secondary Linode and and set up IP failover. For instructions, see Configuring IP Failover.
Although DRBD is included in the vanilla mainline Linux kernel as of version 2.6.33, it isn't compiled into Linode kernels as of this writing. Even if it were, the version of DRBD included in the kernel should match the DRBD userspace tools, prompting the need to run a stock Ubuntu kernel and allow the drbd8-utils package to compile a module for DRBD. Fortunately, this is easy to accomplish via PV-GRUB on your Linodes.
On the primary Linode, issue the following commands to install a Xen-aware kernel, kernel sources and headers, and the grub boot manager. The second set of commands will install the packages on the secondary Linode.
apt-get update apt-get upgrade -y apt-get install -y linux-virtual mkdir /boot/grub ssh root@ha2-db "apt-get update" ssh root@ha2-db "apt-get upgrade -y" ssh root@ha2-db "apt-get install -y linux-virtual" ssh root@ha2-db "mkdir /boot/grub"
On the primary Linode, create a file named /boot/grub/menu.lst with the following contents. Adjust the "title", "kernel", and "initrd" lines to reflect the actual filenames found in the /boot directory.
File:/boot/grub/menu.lst
timeout 10
title Ubuntu 10.04 LTS, kernel 2.6.32-23-generic-pae
root (hd0)
kernel /boot/vmlinuz-2.6.32-23-generic-pae root=/dev/xvda ro quiet
initrd /boot/initrd.img-2.6.32-23-generic-pae
Issue the following command to copy the menu.lst file to the secondary Linode:
scp /boot/grub/menu.lst root@ha2-db:/boot/grub/
Note that if you install an updated kernel, you'll need to add an entry for it to your menu.lst file. By default, the first kernel in the list will be booted.
In the Linode Manager, create three extra disk images for each Linode, named as follows. The disks must be created as "raw" images (not ext3).:
For testing purposes, each may be 1000 MB in size. Please note that each Linode's corresponding disk images must be exactly the same size, as they will be replicated between the two nodes using DRBD later.
In each Linode's configuration (in the Linode Manager), change the "Kernel" dropdown to select "pv-grub-x86_32" (or "pv-grub-x86_64" if you deployed the 64-bit version of Ubuntu Lucid). Assign your newly created disk images to the configuration profile as follows:
Disable all filesystem/boot helpers and save the profiles. Reboot both Linodes from their dashboards to make sure everything comes back up properly under pv-grub. Issue the following command on each to make sure the newly created disk devices are available:
ls -all /dev/xvdc && ls -all /dev/xvdd && ls -all /dev/xvde
You should see output similar to the following:
brw-rw---- 1 root disk 202, 32 Aug 4 15:53 /dev/xvdc brw-rw---- 1 root disk 202, 48 Aug 4 15:53 /dev/xvdd brw-rw---- 1 root disk 202, 64 Aug 4 15:53 /dev/xvde
Issue the following command on each Linode to make sure you're running the stock Ubuntu 10.04 kernel:
uname -a
You should see output similar to the following:
Linux ha1-db 2.6.32-27-generic-pae #49-Ubuntu SMP Thu Dec 2 00:07:52 UTC 2010 i686 GNU/Linux
On the primary Linode, issue the following commands to install required packages. The second set of commands will ensure that the same packages are installed on the secondary Linode as well.
apt-get update apt-get upgrade -y apt-get install -y heartbeat pacemaker nfs-kernel-server postgresql mysql-server service nfs-kernel-server stop service postgresql-8.4 stop service mysql stop update-rc.d -f nfs-kernel-server remove update-rc.d -f postgresql-8.4 remove update-rc.d -f mysql remove ssh root@ha2-db "apt-get update" ssh root@ha2-db "apt-get upgrade -y" ssh root@ha2-db "apt-get install -y heartbeat pacemaker nfs-kernel-server postgresql mysql-server" ssh root@ha2-db "service nfs-kernel-server stop" ssh root@ha2-db "service postgresql-8.4 stop" ssh root@ha2-db "service mysql stop" ssh root@ha2-db "update-rc.d -f nfs-kernel-server remove" ssh root@ha2-db "update-rc.d -f postgresql-8.4 remove" ssh root@ha2-db "update-rc.d -f mysql remove"
After issuing the commands listed above, the required packages will be installed and services will be temporarily stopped on both Linodes. Additionally, the system startup links for these services will be removed on both Linodes, as Pacemaker will be responsible for starting and stopping them as necessary. To make sure MySQL won't start automatically, one additional step is necessary. On the primary Linode, edit the /etc/init/mysql.conf file. Locate the following section, and comment it out as shown:
File excerpt:/etc/init/mysql.conf
#start on (net-device-up
# and local-filesystems
# and runlevel [2345])
#stop on runlevel [016]
Issue the following command to copy the updated config file to the secondary Linode.
scp /etc/init/mysql.conf root@ha2-db:/etc/init/
On the primary Linode, create a file named /etc/heartbeat/ha.cf with the following contents. Replace 98.76.54.32 with the statically assigned public IP address of the secondary Linode.
File: /etc/heartbeat/ha.cf (on primary Linode)
logfacility daemon keepalive 2 deadtime 15 warntime 5 initdead 120 udpport 694 ucast eth0 98.76.54.32 auto_failback on node ha1-db node ha2-db use_logd yes crm respawn
On the secondary Linode, create a file named /etc/heartbeat/ha.cf with the following contents. Replace 12.34.56.78 with the statically assigned public IP address of the primary Linode.
File: /etc/heartbeat/ha.cf (on secondary Linode)
logfacility daemon keepalive 2 deadtime 15 warntime 5 initdead 120 udpport 694 ucast eth0 12.34.56.78 auto_failback on node ha1-db node ha2-db use_logd yes crm respawn
On the primary Linode, create the file /etc/heartbeat/authkeys with the following contents. Make sure to change "CHANGEME" to a strong password consisting of letters and numbers.
File: /etc/heartbeat/authkeys (on primary Linode)
auth 1 1 sha1 CHANGEME
On the primary Linode, issue the following commands to set proper permissions on this file, copy it to the secondary Linode, and start the Heartbeat service on both nodes:
chmod 600 /etc/ha.d/authkeys service heartbeat start scp /etc/ha.d/authkeys root@ha2-db:/etc/ha.d/ ssh root@ha2-db "chmod 600 /etc/ha.d/authkeys" ssh root@ha2-db "service heartbeat start"
On the primary Linode, issue the following commands to install DRBD and utilities for controlling it. The second update-rc.d command will remove the system startup links for DRBD, as it will be controlled by Pacemaker.
apt-get install linux-headers-server apt-get install -y drbd8-utils build-essential psmisc update-rc.d -f drbd remove ssh root@ha2-db "apt-get install linux-headers-server" ssh root@ha2-db "apt-get install -y drbd8-utils build-essential psmisc" ssh root@ha2-db "update-rc.d -f drbd remove"
Next, reboot both Linodes from their Linode Manager dashboards. Once they've come back online, log back into the primary Linode via SSH and create the file /etc/drbd.d/r0.res. Be sure to replace 192.168.88.88 and 192.168.99.99 with the statically assigned private addresses of your primary and secondary Linodes, respectively. Change the shared-secret directive to a strong password consisting of letters and numbers.
File: /etc/drbd.d/r0.res (on primary Linode)
resource r0 {
protocol C;
syncer {
rate 4M;
}
startup {
wfc-timeout 15;
degr-wfc-timeout 60;
}
net {
cram-hmac-alg sha1;
shared-secret "CHANGEME";
}
on ha1-db {
device /dev/drbd0;
disk /dev/xvdc;
address 192.168.88.88:7788;
meta-disk internal;
}
on ha2-db {
device /dev/drbd0;
disk /dev/xvdc;
address 192.168.99.99:7788;
meta-disk internal;
}
}
On the primary Linode, create another file called /etc/drbd.d/r1.res with the following contents. Again, be sure to modify the address and shared-secret lines appropriately.
File: /etc/drbd.d/r1.res (on primary Linode)
resource r1 {
protocol C;
syncer {
rate 4M;
}
startup {
wfc-timeout 15;
degr-wfc-timeout 60;
}
net {
cram-hmac-alg sha1;
shared-secret "CHANGEME";
}
on ha1-db {
device /dev/drbd1;
disk /dev/xvdd;
address 192.168.88.88:7789;
meta-disk internal;
}
on ha2-db {
device /dev/drbd1;
disk /dev/xvdd;
address 192.168.99.99:7789;
meta-disk internal;
}
}
On the primary Linode, create another file called /etc/drbd.d/r2.res with the following contents. Again, be sure to modify the address and shared-secret lines appropriately.
File: /etc/drbd.d/r2.res (on primary Linode)
resource r2 {
protocol C;
syncer {
rate 4M;
}
startup {
wfc-timeout 15;
degr-wfc-timeout 60;
}
net {
cram-hmac-alg sha1;
shared-secret "CHANGEME";
}
on ha1-db {
device /dev/drbd2;
disk /dev/xvde;
address 192.168.88.88:7790;
meta-disk internal;
}
on ha2-db {
device /dev/drbd2;
disk /dev/xvde;
address 192.168.99.99:7790;
meta-disk internal;
}
}
On the primary Linode, issue the following commands to copy the DRBD configuration files to the secondary Linode, zero out the new disk images on both Linodes, and create DRBD devices.
scp /etc/drbd.d/* root@ha2-db:/etc/drbd.d/ dd if=/dev/zero of=/dev/xvdc bs=1024k dd if=/dev/zero of=/dev/xvdd bs=1024k dd if=/dev/zero of=/dev/xvde bs=1024k drbdadm create-md r0 drbdadm create-md r1 drbdadm create-md r2 ssh root@ha2-db "dd if=/dev/zero of=/dev/xvdc bs=1024k" ssh root@ha2-db "dd if=/dev/zero of=/dev/xvdd bs=1024k" ssh root@ha2-db "dd if=/dev/zero of=/dev/xvde bs=1024k" ssh root@ha2-db "drbdadm create-md r0" ssh root@ha2-db "drbdadm create-md r1" ssh root@ha2-db "drbdadm create-md r2"
While executing the dd commands, you'll see messages indicating "No space left on device." This is normal, as the disks are being scrubbed with zeros until there is no space left to fill.
On both Linodes, in separate terminal windows, issue the following command to start DRBD. The commands must be issued fairly quickly to avoid timing out.
service drbd start
On the primary Linode, issue the following commands to start synchronizing the DRBD disk resources.
drbdadm -- --overwrite-data-of-peer primary r0 drbdadm -- --overwrite-data-of-peer primary r1 drbdadm -- --overwrite-data-of-peer primary r2
DRBD will begin synchronizing disk block data between your Linodes. You can watch the progress by issuing the following command on the primary Linode:
watch -n3 cat /proc/drbd
You should see output similar to the following:
Every 3.0s: cat /proc/drbd Thu Jan 20 21:54:44 2011
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@ha1-db, 2011-01-20 21:19:30
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:340512 nr:0 dw:0 dr:340712 al:0 bm:20 lo:0 pe:428 ua:0 ap:0 ep:1 wo:b oos:685132
[=====>..............] sync'ed: 33.2% (685132/1023932)K
finish: 0:05:21 speed: 2,012 (2,200) K/sec
1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:267888 nr:0 dw:0 dr:274348 al:0 bm:16 lo:1 pe:552 ua:1565 ap:0 ep:1 wo:b oos:758248
[====>...............] sync'ed: 26.0% (758248/1023932)K
finish: 0:06:54 speed: 1,824 (1,792) K/sec
2: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:246556 nr:0 dw:0 dr:252716 al:0 bm:14 lo:1 pe:652 ua:1490 ap:0 ep:1 wo:b oos:779980
[===>................] sync'ed: 24.0% (779980/1023932)K
finish: 0:05:54 speed: 2,132 (1,692) K/sec
You don't have to wait for the disks to be fully synced, although you may if you wish. Next, issue the following commands on the primary Linode to become the primary DRBD node and create an ext3 filesystem on each DRBD disk:
drbdadm primary r0 mkfs.ext3 /dev/drbd0 drbdadm primary r1 mkfs.ext3 /dev/drbd1 drbdadm primary r2 mkfs.ext3 /dev/drbd2
On the primary Linode, issue the following commands to create an exports directory and mount the NFS exports DRBD disk to it. In this example, the directory /var/exports/srv will be exported.
drbdadm primary r0 mkdir /var/exports mount /dev/drbd0 /var/exports mkdir /var/exports/srv umount /var/exports drbdadm secondary r0 ssh root@ha2-db "mkdir /var/exports"
On the primary Linode, edit the file /etc/default/nfs-kernel-server to set the following line:
File excerpt:/etc/default/nfs-kernel-server
NEED_SVCGSSD=no
On the primary Linode, edit the file /etc/default/nfs-common to set the following lines:
File excerpt:/etc/default/nfs-common
NEED_IDMAPD=yes
NEED_GSSD=no
On the primary Linode, edit the file /etc/default/portmap to set the following line (the entry should be commented out):
File excerpt:/etc/default/portmap
#OPTIONS="-i 127.0.0.1"
On the primary Linode, edit the file /etc/exports to add entries for each client node that will be connecting to your NFS server resources. Each client must have a private IP address, and must be located in the same datacenter as your HA server cluster. The following lines represent the required entries for one client, and should be duplicated (changing the IP address) for additional clients.
File excerpt:/etc/exports
/var/exports 192.168.22.22/32(rw,sync,no_subtree_check)
/var/exports/srv 192.168.22.22/32(rw,sync,nohide,no_subtree_check)
On the primary Linode, issue the following commands to copy your NFS configuration to the secondary Linode. Note that you must re-sync your /etc/exports file to the secondary node and issue the exportfs command any time the file is modified.
scp /etc/default/nfs-kernel-server root@ha2-db:/etc/default/ scp /etc/default/nfs-common root@ha2-db:/etc/default/ scp /etc/default/portmap root@ha2-db:/etc/default/ scp /etc/exports root@ha2-db:/etc/ exportfs -ra ssh root@ha2-db "exportfs -ra"
You will see warnings regarding the exported directories not existing; this is normal at this stage.
On the primary Linode, edit the /etc/mysql/my.cnf file. For the bind-address directive, replace 127.0.0.1 with the "floating" private IP address.
File excerpt:/etc/mysql/my.cnf
bind-address = 192.168.11.11
On the primary Linode, issue the following commands to copy MySQL's configuration to the secondary node, move its data files to the r1 DRBD disk resource and delete the data files on the secondary Linode:
scp /etc/mysql/my.cnf root@ha2-db:/etc/mysql/ mkdir /root/mysql_bak cp -Ra /var/lib/mysql/* /root/mysql_bak/ rm -rf /var/lib/mysql/* drbdadm primary r1 mount /dev/drbd1 /var/lib/mysql cp -Ra /root/mysql_bak/* /var/lib/mysql/ chown mysql:mysql /var/lib/mysql umount /var/lib/mysql drbdadm secondary r1 ssh root@ha2-db "rm -rf /var/lib/mysql/*"
On the primary Linode, edit the /etc/postgresql/8.4/main/pg_hba.conf file. Add the following line to allow MD5-authenticated connections from the private network.
File excerpt:/etc/postgresql/8.4/main/pg_hba.conf
host all all 192.168.0.0/16 md5
On the primary Linode, issue the following commands to copy PostgreSQL's configuration to the secondary node, move its data files to the r2 DRBD disk resource and delete the data files on the secondary Linode:
scp /etc/postgresql/8.4/main/pg_hba.conf root@ha2-db:/etc/postgresql/8.4/main/ mkdir /root/postgresql_bak cp -Ra /var/lib/postgresql/* /root/postgresql_bak/ rm -rf /var/lib/postgresql/* drbdadm primary r2 mount /dev/drbd2 /var/lib/postgresql cp -Ra /root/postgresql_bak/* /var/lib/postgresql/ chown postgres:postgres /var/lib/postgresql umount /var/lib/postgresql drbdadm secondary r2
It should be noted that unless you have a different editor set via the "EDITOR" environment variable, the cluster resource manager will use vim as its editing environment. If you would prefer to use nano instead, you may set this permanently by issuing the following commands on both Linodes:
export EDITOR=/bin/nano echo "export EDITOR=/bin/nano" >> .bashrc
For the purposes of these instructions, it will be assumed that you are are using vim as your editor. On the primary Linode, issue the following command to start the cluster resource manager in "edit" mode:
crm configure edit
You will be presented with information resembling the following. If you don't see anything, enter ":q" to quit the editor and wait a minute before restarting it.
node $id="0492ddb7-127f-4f96-beaf-7ce0a28e70b1" ha1-db
node $id="4c898f53-5b12-4e8b-866c-d67542411961" ha2-db
property $id="cib-bootstrap-options" \
dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
cluster-infrastructure="Heartbeat"
To begin editing your configuration, press the "i" key. To leave edit mode, press "Ctrl+c". To quit without saving any changes, press ":" and enter "q!". To save changes and quit, press ":" and enter "wq".
Insert the following lines in between the second "node" line at the top of the configuration and the "property" line at the bottom. Important: Be sure to replace both instances of 192.168.11.11 with the "floating" private IP address.
primitive drbd_nfs ocf:linbit:drbd \
params drbd_resource="r0" \
op monitor interval="15s"
primitive fs_nfs ocf:heartbeat:Filesystem \
params device="/dev/drbd/by-res/r0" directory="/var/exports" fstype="ext3" \
op start interval="0" timeout="60" \
op stop interval="0" timeout="120"
primitive drbd_mysql ocf:linbit:drbd \
params drbd_resource="r1" \
op monitor interval="15s"
primitive fs_mysql ocf:heartbeat:Filesystem \
params device="/dev/drbd/by-res/r1" directory="/var/lib/mysql" fstype="ext3" \
op start interval="0" timeout="60" \
op stop interval="0" timeout="120"
primitive drbd_postgresql ocf:linbit:drbd \
params drbd_resource="r2" \
op monitor interval="15s"
primitive fs_postgresql ocf:heartbeat:Filesystem \
params device="/dev/drbd/by-res/r2" directory="/var/lib/postgresql" fstype="ext3" \
op start interval="0" timeout="60" \
op stop interval="0" timeout="120"
primitive ip1 ocf:heartbeat:IPaddr2 \
params ip="192.168.11.11" nic="eth0:1" \
op monitor interval="5s"
primitive ip1arp ocf:heartbeat:SendArp \
params ip="192.168.11.11" nic="eth0:1"
primitive nfs lsb:nfs-kernel-server \
op monitor interval="5s"
primitive mysql ocf:heartbeat:mysql \
params binary="/usr/bin/mysqld_safe" config="/etc/mysql/my.cnf" user="mysql" \
group="mysql" log="/var/log/mysql.log" pid="/var/run/mysqld/mysqld.pid" \
datadir="/var/lib/mysql" socket="/var/run/mysqld/mysqld.sock" \
op monitor interval="30s" timeout="30s" \
op start interval="0" timeout="120" \
op stop interval="0" timeout="120"
primitive postgresql lsb:postgresql-8.4 \
op monitor interval="5s"
group HAServices ip1 ip1arp fs_nfs nfs fs_mysql mysql fs_postgresql postgresql \
meta target-role="Started"
ms ms_drbd_nfs drbd_nfs \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
ms ms_drbd_mysql drbd_mysql \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
ms ms_drbd_postgresql drbd_postgresql \
meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
colocation ms-drbd-mysql-with-haservices inf: ms_drbd_mysql:Master HAServices
colocation ms-drbd-nfs-with-haservices inf: ms_drbd_nfs:Master HAServices
colocation ms-drbd-postgresql-with-haservices inf: ms_drbd_postgresql:Master HAServices
order ip-before-arp mandatory: ip1:start ip1arp:start
order ip-before-ms-drbd-nfs mandatory: ip1:start ms_drbd_nfs:promote
order ms-drbd-nfs-before-fs-nfs mandatory: ms_drbd_nfs:promote fs_nfs:start
order fs-nfs-before-nfs mandatory: fs_nfs:start nfs:start
order ip-before-ms-drbd-mysql mandatory: ip1:start ms_drbd_mysql:promote
order ms-drbd-mysql-before-fs-mysql mandatory: ms_drbd_mysql:promote fs_mysql:start
order fs-mysql-before-mysql mandatory: fs_mysql:start mysql:start
order ip-before-ms-drbd-postgresql mandatory: ip1:start ms_drbd_postgresql:promote
order ms-drbd-postgresql-before-fs-postgresql mandatory: ms_drbd_postgresql:promote fs_postgresql:start
order fs-postgresql-before-postgresql mandatory: fs_postgresql:start postgresql:start
Change the "property" section to resemble the following excerpt. You'll be adding an "expected-quorum-votes" entry due to the fact that your cluster only has two nodes, as well as adding the lines for "stonith-enabled" and "no-quorum-policy". Don't forget the trailing "\" after the "cluster-infrastructure" line.
property $id="cib-bootstrap-options" \
dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
cluster-infrastructure="Heartbeat" \
expected-quorum-votes="1" \
stonith-enabled="false" \
no-quorum-policy="ignore"
Add the following excerpt after the "property" section:
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
After making these changes, press "Ctrl+c" and enter ":wq" to save the configuration and exit the editor.
On the primary Linode, issue the commnd crm_mon to start the cluster monitor. You'll see output resembling the following:
============
Last updated: Thu Aug 5 21:10:30 2010
Stack: Heartbeat
Current DC: ha1-db (ec7c37eb-c7fc-4eb4-b7dc-2f9dc6c9483b) - partition with quorum
Version: 1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd
2 Nodes configured, 1 expected votes
4 Resources configured.
============
Online: [ ha1-db ha2-db ]
Master/Slave Set: ms_drbd_mysql
Masters: [ ha1-db ]
Slaves: [ ha2-db ]
Master/Slave Set: ms_drbd_postgresql
Masters: [ ha1-db ]
Slaves: [ ha2-db ]
Resource Group: HAServices
ip1 (ocf::heartbeat:IPaddr2): Started ha1-db
ip1arp (ocf::heartbeat:SendArp): Started ha1-db
fs_nfs (ocf::heartbeat:Filesystem): Started ha1-db
nfs (lsb:nfs-kernel-server): Started ha1-db
fs_mysql (ocf::heartbeat:Filesystem): Started ha1-db
mysql (ocf::heartbeat:mysql): Started ha1-db
fs_postgresql (ocf::heartbeat:Filesystem): Started ha1-db
postgresql (lsb:postgresql-8.4): Started ha1-db
Master/Slave Set: ms_drbd_nfs
Masters: [ ha1-db ]
Slaves: [ ha2-db ]
In this example, the clustered resources are started on ha1-db. To simulate a failover situation, issue the following command to put ha1-db into standby:
crm node standby ha1-db
Within a few seconds, the resources will be stopped on the initial node and started on the other one. To bring ha1-db back online, simply issue the following command:
crm node online ha1-db
At this point, you should be able to shut down the Linode hosting your resources and watch them automatically migrate to the other Linode (provided you have crm_mon running in a terminal on the still-active Linode). Note that because "resource-stickiness" is set at "100", resources should stay wherever they are migrated they until you manually move them to another node. This can be helpful in cases where you need to perform maintenance on a node, but don't want services resuming on it until you're ready. Congratulations, you've successfully implemented a high availability file and database server configuration for two nodes!
You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.
This guide is licensed under a Creative Commons Attribution-NoDerivs 3.0 United States License.
Last edited by Matthew Cone on Monday, July 16th, 2012 (r2975).