1. If you use gfs, then you may use following command to see if the list output and df output are the same on all the nodes, e.g:
| #gfs_tool list #gfs_tool df |
2. And for RHCS services, you may using following command to see if all the nodes in normal status,e.g:
| # clustat # cman_tool status |
3. To check failover function works or not, there are two level failover you may need to check: service level, node level,
most of time you may focus on the service level(you business logic) failover:
3.1 for service level failover, you unplug the virtual IP network to provide cluster services (business logic) and to see if the service can migrate to other nodes
Note:
before you test the service level failover, you may try to test if you can manually migrate the server from one node to another node, e.g:
# clusvcadm -r service -m other_node
3.2 for node level failover, you can try fencing function works in your cluster or not, at last, you may unplug the heartbeat network which used to communicate among nodes to see if the node can be rebooted, e.g:
1) Test by fence agent without cman/cluster.conf configured
# fence_xxxx -a <ipaddress> -l <username> -p <password>
The default action will reboot the other node, use "-o status" will not reboot.
2) Test fence while cman is started
# fence_node <node_name>
3) Test fence by cutting down the heartbeat network
Note:
1. ideally the RHCS could separate this three network to different NIC. If you use only one NIC, then all the failover test described here cannot be done
2. the heartbeat network is the one used to communicate with other nodes in the cluster, it don't specified in configuration file,
RHCS use the corresponding IP address for the hostname to communicate with the other nodes in the cluster, if fencing network and heartbeat network are setup on the same NIC,
unplug the heartbeat network cable cannot cause the node reboot.
