Pacemaker, Corosync and OCFS2 don’t seem to play nice together on CentOS 6.2

One aspect of using Linux that often boggles my mind is how the difficultly of some tasks will vary from distribution to distribution. For example, a package or absolute dependency needed to compile an application or build a library may be present on OpenSUSE or SLES but be completely missing from say RHEL or CentOS. And although you can download it and attempt to build it on your distribution of choice, that in-and-of itself may present it’s own unique challenges! This story is one such illustration of this.

See, for a while now, I have been using SLES 10 as the base for an email cluster. I’ve been wanting to migrate that application to a more recent distribution, so I began testing Pacemaker, Corosync and OCFS2 on a Debian Squeeze VM running on.. yes you guessed it, XCP.
My testing went great and as a matter of fact, Debian Squeeze would make a great base OS, but I had a couple of additional requirements:

•      I wanted a more recent kernel than the one shipped with Debian Squeeze.
•      I was concerned that Debian Squeeze support was still “experimental” on XenServer.
•      I wanted OCFS2 1.8

OCFS2 1.8 is important (to me at least) because it provides indexed directories support which should improve the performance of my email servers. So this meant that I needed to use either Red Hat, CentOS, Scientific or Oracle Enterprise Linux. I choose CentOS because I have tons of experience with it, updates are free and timely, I have the installer on my ISO repository readily available, and because it’s supported by XenServer.

I fired up a new VM and pointed it to my favorite mirror – http://centos.mirrors.tds.net. I select “minimal” as my install type, clicked a few buttons, went to grab some coffee and my install was done. I reboot. After the reboot, I yum install wget and then pull down the Oracle Public Yum repository file. I run yum update then yum install kernel-uek-xen ocfs2-tools kern-uek-xen-devel then reboot once more. When my system came back up, I started installing the necessary package to get my cluster stack up and running. When I got all my packages installed and everything ready, I started adding my configuration.

node pcmk-dev1 
property $id="cib-bootstrap-options" \
        no-quorum-policy="ignore" \
        stonith-enabled="false" \
        cluster-infrastructure="openais" \
        expected-quorum-votes="2" \
	placement-strategy=balanced 
rsc_defaults $id="rsc-options" \
        resource-stickiness="0"
primitive p_o2cb ocf:pacemaker:o2cb \
        op start interval="0" timeout="90" \
        op stop interval="0" timeout="100" \
        op monitor interval="120s"
primitive p_dlm ocf:pacemaker:controld \
        op start interval="0" timeout="90" \
        op stop interval="0" timeout="100" \
        op monitor interval="120s"
clone c_dlm-clone p_dlm \
        meta globally-unique="false" interleave="true"
clone c_o2cb-clone p_o2cb \
        meta globally-unique="false" interleave="true"

This was the same configuration that I was using on Debian Squeeze and when I attempted it on CentOS 6.2, to my surprise it didn’t work. In reviewing the logs, it seemed that my assumption that CentOS’s Pacemaker packages would be at least identical to and Oracle’s was flat out wrong. CentOS 6.2 does not ship Pacemaker with /usr/sbin/dlm_controld.pcmk, the distributed lock management component needed to even run a cluster filesystem. At first I thought that perhaps I was just missing a package or two, but when I ran yum provides nothing turned up. I even checked the centosplus repository I’m currently inquiring with both the Pacemaker and CentOS mailing lists, but I have a feeling that Red Hat has taken a different path with clustering with the most recent releases of RHEL. Seems like they are moving away from supporting OCFS2 and pushing end users towards GFS.
This kind of fragmentation with respect to Linux is incredibly frustrating at times. I kind of wish that the community would settle on a standard and go with it, but that isn’t what Linux is about right 😉
Anyways, if you like using RHEL/CentOS and would much rather continue to use OCFS2, in the coming days, I’ll be posting a detailed “how to” on it. Stay tuned if you are interested.