Redhat Suite High Availability Overview

HIGH AVAILABILITY :

Group of computer working together to provide the continuous availability of application service by eliminating single point of failure.
LOAD BALANCING : Distributed workloads across the multiple compute nodes.

High availability Requirement.
Single node .
dual NIC cards,
Dual RAID Controller,
Multipath  Shared Storage.
Both Primary and Secondary node should have
same hardware configuration,
Partition layout,
Software & Service should be same
except IP Address and hostname should be differ.

 OS Version Details – Redhat 6.4

Redhat Packages

LUCI	WebInterface – Packages	Management Node
ricci	Web interfacfe (SSL packages)	Need to install cluster node(node1 & node2).
cman	Cluster Communication	Need to install cluster node (node1 & node2)
rgmanager	Resource Group Manager	Need to install cluster node (node1 & node2)
modcluster	-----	Need to install cluster node (node1 & node2)

Above Table - Represent in red color packages should be installed by manually.
Remaining Green color packages it will installed automatically by cluster.

 Cluster Resources

1)IP Address
2)Shared file system(Shared file system -Lustre, NFS,Device Multipath)
3)Script/service(http,mysql)
 

Step By Step High Availability Configuration Steps.

1)Install luci -> Management node.
2)Install ricci -> Cluster nodes (node1 & nod2)
3)Set the Password for ricci
4)From the Management node Login via web interface LUCI and then add
the two nodes(node1 & node2)
NOTE: Before adding node confirm network communication between nodes (Bridging,Networking, /etc/hosts,security  are updated properly).
NOTE: Network manager,IPtables should be in stop mode.
5)while adding cluster nodes management node itself will install (cman,rgmanager,modclusterd)packages on the cluster nodes. We can monitor via tail –f /var/log/messages /var/log/secure.
NOTE: Confirm cman and cluster.conf are correct on both cluster nodes.
6)Create the failover domain (nodes with Priority)
7)Create the Resources (IP address,script,File system).
NOTE: While add the resource read the statement and give the enable/disable option in the check box clearly.
8)Create the services( add the resource with failover domain)
9)Finally add the fencing devices.

Sample Cluster.conf file.
# cat /etc/cluster/cluster.conf
<?xml version="1.0"?>
<cluster config_version="20" name="MY_CLUSTER2">
        <clusternodes>
                <clusternode name="node10" nodeid="1">
                        <fence>
                                <method name="Method">
                                        <device domain="node10" name="KVM_Fencing"/>
                                </method>
                        </fence>
                </clusternode>
                <clusternode name="node11" nodeid="2">
                        <fence>
                                <method name="Method1">
                                        <device domain="node11" name="KVM_Fenhttp://linternux.com/2010/11/creating-cluster-part-1/cing"/>
                                </method>
                        </fence>
                </clusternode>
        </clusternodes>
        <cman expected_votes="1" two_node="1"/>
        <rm>
                <failoverdomains>
                        <failoverdomain name="Node11priotiry" ordered="1" restricted="1">
                                <failoverdomainnode name="node10" priority="11"/>
                                <failoverdomainnode name="node11" priority="10"/>
                        </failoverdomain>
                </failoverdomains>
                <resources>
                        <ip address="10.154.1.185" sleeptime="10"/>
                        <script file="/etc/init.d/httpd" name="http"/>
                        <fs device="/dev/mapper/1IET_00010001p1" force_fsck="1" force_unmount="1" fsid="44376" fstype="ext4" mountpoint="/var/www/html" name="webroot" quick_status="1" self_fence="1"/>
                </resources>
                <service domain="Node11priotiry" exclusive="1" name="Http_service" recovery="relocate">
                        <ip ref="10.154.1.185"/>
                        <script ref="http"/>
                        <fs ref="webroot"/>
                </service>
        </rm>
        <fencedevices>
                <fencedevice agent="fence_xvm" name="KVM_Fencing"/>
        </fencedevices>
</cluster>

Cluster In Healthy Condition.

Cman – Distributed Cluster Manager – it will communicate between the nodes to check active or down.
 Cluster in Poor Condition

Reference : http://linternux.com/2010/11/creating-cluster-part-1/

/var/log/message - Relocating service from node10 to node11
-----------------------------------------------------------------------------
Jun 20 16:52:09 node11 rgmanager[3171]: Member 1 shutting down
Jun 20 16:52:10 node11 rgmanager[3171]: Starting stopped service service:Http_service
Jun 20 16:52:10 node11 rgmanager[4761]: [fs] mounting /dev/dm-2 on /var/www/html
Jun 20 16:52:10 node11 rgmanager[4783]: [fs] mount -t ext4  /dev/dm-2 /var/www/html
Jun 20 16:52:10 node11 kernel: EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts:
Jun 20 16:52:11 node11 rgmanager[4865]: [ip] Adding IPv4 address 10.154.1.185/24 to eth0
Jun 20 16:52:14 node11 rgmanager[4954]: [script] Executing /etc/init.d/httpd start
Jun 20 16:52:14 node11 rgmanager[3171]: Service service:Http_service started 
------------------------------------------------------------------------------------

Manually Relocating The Service From One Node To Another Node
#clusvcadm -r Http_service  -m <node_name>