Creating High Availability Cluster with Centos 7

When you need to run a mission-critical service,  you might need to consider the service will hang or crash, but you also need to think about how to deal with maintenance or service restart times. Fortunately, those problem could be solved by HA cluster to provide you almost 100% up-time for your service.

Introduction

The HA cluster have many different deployment scenarios, here I will use the basic scenarios: 2 VMs to emulate two server cluster.

The HA system is working like this:

• 192.168.1.1──────→ Server 1 (Service active, IP: 192.168.1.2)
• (Virtual IP)　　└───　 Server 2 (Service idle, IP: 192.168.1.3)

While service is running, users will be communicates with server 1 and server 2 remain idle. But when the server 1 hang or crash, the communication traffic will be redirect to server 2:

• 192.168.1.1──────　 Server 1 (Service DOWN, IP: 192.168.1.2)
• (Virtual IP)　　└───→ Server 2 (Service active, IP: 192.168.1.3)

For the user, they might not even noticed that they’re not communicate with Server 1 anymore, so they probably might think “Ah, this site’s server don’t even down, how fascinated!". XD

Setting HA Cluster

I will use CentOS 7 VMs to demonstrate how to create a HA cluster on Virtualbox. Here’s the environment settings:

• 1 CPU
• 1024 MB Memory
• 64 MB display memory
• 10 GB Disk Space

Those OS are newly installed, so it will be simplified for this article. This system might work on other version of CentOS like 6 or Fedora 15 ~ 20 without or with minor modification.

Ok, let’s do it!

1. Download and install Corosync & Pacemaker & pcs

In CentOS 7, heartbeat is deprecated. So you need to use Corosync for messaging / membership service, Pacemaker for resource management, pcs(or crmsh) allow you to manage your cluster nodes at command line.

If you already selected High Availability option while installing the CentOS 7, then corosync will be installed too; If no, you can use yum in terminal to get them:

• yum -y install corosync pacemaker pcs

If you don’t like pcs, you can replace pcs with crmsh.

By the way, if you need data consistency, you will need to install DRBD or GlusterFS to synchronize them.

2. Set up static IP

When one of your server crashed and reboot, it’s IP address might be change if it is running at DHCP mode. So you have to set up static IP for them or you might see one of your node go “Missing".

Also, you need to set your gateway

3. Firewall

In order to know your each machine’s status, Pacemaker & Corosync will need several ports open for communication. If you’re just try to do a demonstration like me, you can simply disable firewall and SELinux.

But if your system could be reach from the outside world, DON’T DISABLE FIREWALL! Then you have to open ports for Pacemaker and Corosync manually at each node:

• Open UDP 5404 & 5405 for Corosync
• Open TCP 5560(2224) for crmsh(PCS)
• Allow IGMP traffic
• Allow Multicast traffic

Don’t forget to save settings.

4. Start pcs

Before your cluster could be configured, you need to start the pcs daemon and boot up at each node, so type those command into your terminal:

• systemctl enable pcsd.service
• systemctl start pcsd.service

Also, yum will create a account “hacluster" for management, so you should change it’s password.

5. Set node name

If you already set up hostname for each node, you can skip this section.

Hostname is important for you to verify which node have something wrong when you’re checking logs, so I recommend you to set unique hostname for each node.

6. Configure Corosync

Now it’s time to group up each machine as one cluster. This part will use the hacluster account to get pcs authenticate, make sure you have setup those steps on above.

To do that, you need to type the following commands in the terminal(I used pc1 & pc2 as hostname for each node):

• pcs cluster auth pc1 pc2
• (Then you need to enter hacluster and password for authenticate)
• pcs cluster setup –name <cluster name> pc1 pc2

After the pcs configured the cluster group, you can start it with this command:

• pcs cluster start –all

Then you should see the cluster boot up.

• Check status: pcs status cluster
• Check nodes status: pcs status nodes

7. Virtual IP

Now you need to create an virtual IP address to link to this cluster. This virtual IP will represent the cluster as a entity, the IP at each node will only represent that node, that’s the different.

To create an virtual IP, you need to type:

• pcs resource create virtual_ip ocf:heartbeat:IPaddr2 ip=192.168.1.4 cidr_netmask=24 op monitor interval=20s

When this is done, check it’s status with this command:

• pcs status resources

If you see the virtual_ip condition at started, you can ping 192.168.1.4 and it should be reachable.

In this point: Congratulations, your cluster is up and ready!

More References:

Business Continuity Plan 1 : HA [Link]

Business Continuity / Disaster Recovery Plan [Link]

Using Audit on CentOS 7 [Link]

Using GlusterFS to do Cluster Synchronization on CentOS 7 [Link]

Corosync [Link]

Pacemaker [Link]

crmsh [Link]

DRBD [Link]

GlusterFS [Link]

Wiki: High Availability [Link]