<div dir="ltr">maybe you missing log when you had fenced the node? because i think the clvmd hungup because your node are in unclean state, use dlm_tool ls to see if you any pending fencing operation.<br></div><div class="gmail_extra">
<br><br><div class="gmail_quote">2014/1/1 Bob Haxo <span dir="ltr"><<a href="mailto:bhaxo@sgi.com" target="_blank">bhaxo@sgi.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<u></u>
<div>
Greetings ... Happy New Year!<br>
<br>
I am testing a configuration that is created from example in "Chapter 6. Configuring a GFS2 File System in a Cluster" of the "Red Hat Enterprise Linux 7.0 Beta Global File System 2" document. Only addition is stonith:fence_ipmilan. After encountering this issue when I configured with "crm", I re-configured using "pcs". I've included the configuration below.<br>
<br>
I'm thinking that, in a 2-node cluster, if I run "stonith_admin -F <peer-node>", then <peer-node> should reboot and cleanly rejoin the cluster. This is not happening. <br>
<br>
What ultimately happens is that after the initially fenced node reboots, the system from which the stonith_admin -F command was run is fenced and reboots. The fencing stops there, leaving the cluster in an appropriate state.<br>
<br>
The issue seems to reside with clvmd/lvm. With the reboot of the initially fenced node, the clvmd resource fails on the surviving node, with a maximum of errors. I hypothesize there is an issue with locks, but have insufficient knowledge of clvmd/lvm locks to prove or disprove this hypothesis.<br>
<br>
Have I missed something ...<br>
<br>
1) Is this expected behavior, and always the reboot of the fencing node happens? <br>
<br>
2) Or, maybe I didn't correctly duplicate the Chapter 6 example?<br>
<br>
3) Or, perhaps something is wrong or omitted from the Chapter 6 example?<br>
<br>
Suggestions will be much appreciated.<br>
<br>
Thanks,<br>
Bob Haxo<br>
<br>
RHEL6.5<br>
pacemaker-cli-1.1.10-14.el6_5.1.x86_64<br>
crmsh-1.2.5-55.1sgi709r3.rhel6.x86_64<br>
pacemaker-libs-1.1.10-14.el6_5.1.x86_64<br>
cman-3.0.12.1-59.el6_5.1.x86_64<br>
pacemaker-1.1.10-14.el6_5.1.x86_64<br>
corosynclib-1.4.1-17.el6.x86_64<br>
corosync-1.4.1-17.el6.x86_64<br>
pacemaker-cluster-libs-1.1.10-14.el6_5.1.x86_64<br>
<br>
Cluster Name: mici<br>
Corosync Nodes:<br>
<br>
Pacemaker Nodes:<br>
mici-admin mici-admin2<br>
<br>
Resources:<br>
Clone: clusterfs-clone<br>
Meta Attrs: interleave=true target-role=Started<br>
Resource: clusterfs (class=ocf provider=heartbeat type=Filesystem)<br>
Attributes: device=/dev/vgha2/lv_clust2 directory=/images fstype=gfs2 options=defaults,noatime,nodiratime<br>
Operations: monitor on-fail=fence interval=30s (clusterfs-monitor-interval-30s)<br>
Clone: clvmd-clone<br>
Meta Attrs: interleave=true ordered=true target-role=Started<br>
Resource: clvmd (class=lsb type=clvmd)<br>
Operations: monitor on-fail=fence interval=30s (clvmd-monitor-interval-30s)<br>
Clone: dlm-clone<br>
Meta Attrs: interleave=true ordered=true<br>
Resource: dlm (class=ocf provider=pacemaker type=controld)<br>
Operations: monitor on-fail=fence interval=30s (dlm-monitor-interval-30s)<br>
<br>
Stonith Devices:<br>
Resource: p_ipmi_fencing_1 (class=stonith type=fence_ipmilan)<br>
Attributes: ipaddr=128.##.##.78 login=XXXXX passwd=XXXXX lanplus=1 action=reboot pcmk_host_check=static-list pcmk_host_list=mici-admin<br>
Meta Attrs: target-role=Started<br>
Operations: monitor start-delay=30 interval=60s timeout=30 (p_ipmi_fencing_1-monitor-60s)<br>
Resource: p_ipmi_fencing_2 (class=stonith type=fence_ipmilan)<br>
Attributes: ipaddr=128.##.##.220 login=XXXXX passwd=XXXXX lanplus=1 action=reboot pcmk_host_check=static-list pcmk_host_list=mici-admin2<br>
Meta Attrs: target-role=Started<br>
Operations: monitor start-delay=30 interval=60s timeout=30 (p_ipmi_fencing_2-monitor-60s)<br>
Fencing Levels:<br>
<br>
Location Constraints:<br>
Resource: p_ipmi_fencing_1<br>
Disabled on: mici-admin (score:-INFINITY) (id:location-p_ipmi_fencing_1-mici-admin--INFINITY)<br>
Resource: p_ipmi_fencing_2<br>
Disabled on: mici-admin2 (score:-INFINITY) (id:location-p_ipmi_fencing_2-mici-admin2--INFINITY)<br>
Ordering Constraints:<br>
start dlm-clone then start clvmd-clone (Mandatory) (id:order-dlm-clone-clvmd-clone-mandatory)<br>
start clvmd-clone then start clusterfs-clone (Mandatory) (id:order-clvmd-clone-clusterfs-clone-mandatory)<br>
Colocation Constraints:<br>
clusterfs-clone with clvmd-clone (INFINITY) (id:colocation-clusterfs-clone-clvmd-clone-INFINITY)<br>
clvmd-clone with dlm-clone (INFINITY) (id:colocation-clvmd-clone-dlm-clone-INFINITY)<br>
<br>
Cluster Properties:<br>
cluster-infrastructure: cman<br>
dc-version: 1.1.10-14.el6_5.1-368c726<br>
last-lrm-refresh: 1388530552<br>
no-quorum-policy: ignore<br>
stonith-enabled: true<br>
Node Attributes:<br>
mici-admin: standby=off<br>
mici-admin2: standby=off<br>
<br>
<br>
Last updated: Tue Dec 31 17:15:55 2013<br>
Last change: Tue Dec 31 16:57:37 2013 via cibadmin on mici-admin<br>
Stack: cman<br>
Current DC: mici-admin2 - partition with quorum<br>
Version: 1.1.10-14.el6_5.1-368c726<br>
2 Nodes configured<br>
8 Resources configured<br>
<br>
Online: [ mici-admin mici-admin2 ]<br>
<br>
Full list of resources:<br>
<br>
p_ipmi_fencing_1 (stonith:fence_ipmilan): Started mici-admin2<br>
p_ipmi_fencing_2 (stonith:fence_ipmilan): Started mici-admin<br>
Clone Set: clusterfs-clone [clusterfs]<br>
Started: [ mici-admin mici-admin2 ]<br>
Clone Set: clvmd-clone [clvmd]<br>
Started: [ mici-admin mici-admin2 ]<br>
Clone Set: dlm-clone [dlm]<br>
Started: [ mici-admin mici-admin2 ]<br>
<br>
Migration summary:<br>
* Node mici-admin:<br>
* Node mici-admin2:<br>
<br>
=====================================================<br>
crm_mon after the fenced node reboots. Shows the failure of clvmd that then<br>
occurs, which in turn triggers a fencing of that nnode<br>
<br>
Last updated: Tue Dec 31 17:06:55 2013<br>
Last change: Tue Dec 31 16:57:37 2013 via cibadmin on mici-admin<br>
Stack: cman<br>
Current DC: mici-admin - partition with quorum<br>
Version: 1.1.10-14.el6_5.1-368c726<br>
2 Nodes configured<br>
8 Resources configured<br>
<br>
Node mici-admin: UNCLEAN (online)<br>
Online: [ mici-admin2 ]<br>
<br>
Full list of resources:<br>
<br>
p_ipmi_fencing_1 (stonith:fence_ipmilan): Stopped<br>
p_ipmi_fencing_2 (stonith:fence_ipmilan): Started mici-admin<br>
Clone Set: clusterfs-clone [clusterfs]<br>
Started: [ mici-admin ]<br>
Stopped: [ mici-admin2 ]<br>
Clone Set: clvmd-clone [clvmd]<br>
clvmd (lsb:clvmd): FAILED mici-admin<br>
Stopped: [ mici-admin2 ]<br>
Clone Set: dlm-clone [dlm]<br>
Started: [ mici-admin mici-admin2 ]<br>
<br>
Migration summary:<br>
* Node mici-admin:<br>
clvmd: migration-threshold=1000000 fail-count=1 last-failure='Tue Dec 31 17:04:29 2013'<br>
* Node mici-admin2:<br>
<br>
Failed actions:<br>
clvmd_monitor_30000 on mici-admin 'unknown error' (1): call=60, status=Timed Out, la<br>
st-rc-change='Tue Dec 31 17:04:29 2013', queued=0ms, exec=0ms<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
</div>
<br>_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>esta es mi vida e me la vivo hasta que dios quiera
</div>