<tt><font size=2>"Users" <users-bounces@clusterlabs.org>
schrieb am 16.07.2018 15:12:14:<br>
<br>
> Von: "Ulrich Windl" <Ulrich.Windl@rz.uni-regensburg.de></font></tt>
<br><tt><font size=2>> An: <users@clusterlabs.org></font></tt>
<br><tt><font size=2>> Datum: 16.07.2018 15:12</font></tt>
<br><tt><font size=2>> Betreff: [ClusterLabs] Antw: Antwort: Antw: Antw:
Antwort: Antw: <br>
> corosync/dlm fencing?</font></tt>
<br><tt><font size=2>> Gesendet von: "Users" <users-bounces@clusterlabs.org></font></tt>
<br><tt><font size=2>> <br>
> >>> Philipp Achmüller <philipp.achmueller@arz.at> schrieb
am 16.07.2018 um<br>
> 14:09 in<br>
> Nachricht <OF217A0E94.FE51C4C9-ONC12582CC.00427BA0-C12582CC.0042CDD6@arz.at>:<br>
> > Hi,<br>
> > <br>
> >> Von: "Ulrich Windl" <Ulrich.Windl@rz.uni-regensburg.de><br>
> >> An: <users@clusterlabs.org><br>
> >> Datum: 16.07.2018 13:46<br>
> >> Betreff: [ClusterLabs] Antw: Antw: Antwort: Antw: corosync/dlm
fencing?<br>
> >> Gesendet von: "Users" <users-bounces@clusterlabs.org><br>
> >> <br>
> >> Hi again!<br>
> >> <br>
> >> Oh, I missed "...maintenance i would like to standby
1 or 2 nodes from<br>
> >> "sitea""...<br>
> >> <br>
> > <br>
> > Yes - sorry for tons of logs - but i think this will catch the
whole <br>
> > situation... <br>
> > <br>
> >> I think some time ago I had asked about the same thing for
SLES11, and <br>
> > the<br>
> >> answer was that with some configurations a standby is not
possible (I <br>
> > thought<br>
> >> it was related to OCFS2, but mybe it was cLVM or DLM). Despite
of quorum<br>
> >> issues, why not shutdown the node completely?<br>
> > <br>
> > i also tried "systemctl stop pacemaker" or "normal"
system shutdown -> <br>
> > same outcome, there is always dlm fencing some other nodes...<br>
> <br>
> Hmm... I looked up my last node shutdown on the three-node cluster.
Essential<br>
> messages are like this (many messages left out, sometimes indicated
by<br>
> "(more)"):<br>
> <br>
> attrd[13169]: notice: attrd_trigger_update: Sending flush op
to all hosts<br>
> for: shutdown (0)<br>
> pengine[13170]: notice: stage6: Scheduling Node h10 for shutdown<br>
> pengine[13170]: notice: LogActions: Stop prm_DLM:1
(h10)<br>
> pengine[13170]: notice: LogActions: Stop prm_cLVMd:1
(h10)<br>
> pengine[13170]: notice: LogActions: Stop prm_LVM_CFS_VMs:1
(h10)<br>
> (more)<br>
> crmd[13171]: notice: te_rsc_command: Initiating action 97:
stop<br>
> prm_CFS_VMs_fs_stop_0 on h10<br>
> cluster-dlm[13901]: log_config: dlm:ls:490B9FCAFA3D4B2F9A586A5893E00730
conf 2<br>
> 0 1 memb 739512321 739512325 join left 739512330<br>
> cluster-dlm[13901]: add_change: 490B9FCAFA3D4B2F9A586A5893E00730 add_change
cg<br>
> 8 remove nodeid 739512330 reason 2<br>
> (more)<br>
> cluster-dlm[13901]: receive_plocks_stored: 490B9FCAFA3D4B2F9A586A5893E00730<br>
> receive_plocks_stored 739512321:8 flags a sig 0 need_plocks 0<br>
> ocfs2_controld[14389]: confchg called<br>
> ocfs2_controld[14389]: node daemon left 739512330<br>
> cluster-dlm[13901]: log_config: dlm:ls:clvmd conf 2 0 1 memb 739512321<br>
> 739512325 join left 739512330<br>
> cluster-dlm[13901]: stop_kernel: clvmd stop_kernel cg 8<br>
> cluster-dlm[13901]: log_config: dlm:controld conf 2 0 1 memb 739512321<br>
> 739512325 join left 739512330<br>
> crmd[13171]: notice: run_graph: Transition 10 (Complete=41,
Pending=0,<br>
> Fired=0, Skipped=0, Incomplete=0,<br>
> Source=/var/lib/pacemaker/pengine/pe-input-26.bz2): Complete<br>
> cib[13166]: notice: crm_update_peer_state: cib_peer_update_callback:
Node<br>
> h10[739512330] - state is now lost (was member)<br>
> corosync[13162]: [CLM ] Members Left:<br>
> corosync[13162]: [CLM ] r(0) ip(172.20.16.10)
r(1) ip(10.2.2.10)<br>
> stonith-ng[13167]: notice: crm_update_peer_state: st_peer_update_callback:<br>
> Node h10[739512330] - state is now lost (was member)<br>
> cluster-dlm[13901]: dlm_process_node: Removed inactive node 739512330:<br>
> born-on=3804, last-seen=3804, this-event=3808, last-event=3804<br>
> kernel: [349683.507286] dlm: closing connection to node 739512330<br>
> crmd[13171]: notice: peer_update_callback: do_shutdown of h10
(op 190) is<br>
> complete<br>
> corosync[13162]: [MAIN ] Completed service synchronization,
ready to provide<br>
> service.<br>
> crmd[13171]: notice: do_state_transition: State transition<br>
> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL<br>
> origin=notify_crmd ]<br>
> <br>
> Another random thought: How do you collect your logs? The node being
fenced<br>
> may not have the last few log messages written to disk (SLES11 syslog
at<br>
> least). Sometimes it's a good idea to "tail -f" the syslogs
on each <br>
> node, so if<br>
> the node goes down, you see what is expected to be in the syslog at
least.<br>
> Chances may be better if you have a remote syslog server via DGRAM
sockets...<br>
> </font></tt>
<br>
<br><tt><font size=2>There are messages in syslog, but notwhing cluster
relevated</font></tt>
<br>
<br><tt><font size=2>regards</font></tt>
<br><tt><font size=2>Philipp</font></tt>
<br><tt><font size=2><br>
> Regards,<br>
> Ulrich<br>
> <br>
> > <br>
> >> <br>
> >> Regards,<br>
> >> Ulrich<br>
> >> <br>
> >> >>> "Ulrich Windl" <Ulrich.Windl@rz.uni-regensburg.de>
schrieb am <br>
> > 16.07.2018<br>
> >> um<br>
> >> 13:35 in Nachricht <5B4C82F0020000A10002C5FA@gwsmtp1.uni-regensburg.de>:<br>
> >> >>>> Philipp Achmüller <philipp.achmueller@arz.at>
schrieb am 16.07.2018 <br>
> > um<br>
> >> > 11:44 in<br>
> >> > Nachricht<br>
> >> <OFE07B0DF6.7DA543A3-ONC12582CC.0031FEDF-C12582CC.0035892B@arz.at>:<br>
> >> >> hi!<br>
> >> >> <br>
> >> >> Thank you for comment.<br>
> >> >> Unfortunatly it is not obvious for me - the "grep
fence" is attached <br>
> > in my<br>
> >> <br>
> >> >> original message.<br>
> >> > <br>
> >> > Hi!<br>
> >> > <br>
> >> > OK, seems I missed finding the needle in all the hay...<br>
> >> > Anyway I think the problem is "Cluster node siteb-1
will be fenced: <br>
> > peer is<br>
> >> <br>
> >> > no<br>
> >> > longer part of the cluster". Looks as if the cluster
noticed an <br>
> > unclean<br>
> >> > shutdown of node siteb-1. The message "Stonith/shutdown
of siteb-1 not <br>
> > <br>
> >> > matched"<br>
> >> > seems to confirm that.<br>
> >> > When shutting down two nodes, did you wait until shutdown
of the first <br>
> > node<br>
> >> > succeeded before shutting down the second?<br>
> >> > <br>
> >> > Regards,<br>
> >> > Ulrich<br>
> >> > <br>
> >> >> <br>
> >> >> i searched older logs with activated dubug information
for dlm: - <br>
> > this is <br>
> >> >> the sequence from syslog (from another timeframe):<br>
> >> >> <br>
> >> >> -------------------<br>
> >> >> Node: siteb-2 (DC):<br>
> >> >> ............<br>
> >> >> 2018-06-28T09:02:23.272415+02:00 siteb-2 crmd[189260]:
notice: <br>
> > State <br>
> >> >> transition S_IDLE -> S_POLICY_ENGINE<br>
> >> >> 2018-06-28T09:02:23.279028+02:00 siteb-2 pengine[189259]:
notice: <br>
> >> >> Watchdog will be used via SBD if fencing is required<br>
> >> >> 2018-06-28T09:02:23.279214+02:00 siteb-2 pengine[189259]:
notice: <br>
> > On <br>
> >> >> loss of CCM Quorum: Ignore<br>
> >> >> 2018-06-28T09:02:23.282153+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Move <br>
> >> >> stonith-sbd#011(Started sitea-1 -> siteb-1)<br>
> >> >> 2018-06-28T09:02:23.282249+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Move <br>
> >> >> cl-info#011(Started sitea-1 -> siteb-2)<br>
> >> >> 2018-06-28T09:02:23.282338+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Move <br>
> >> >> k45RG#011(Started sitea-2 -> siteb-1)<br>
> >> >> 2018-06-28T09:02:23.282422+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> dlm:0#011(sitea-1)<br>
> >> >> 2018-06-28T09:02:23.282505+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> clvm:0#011(sitea-1)<br>
> >> >> 2018-06-28T09:02:23.282588+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> vg1:0#011(sitea-1)<br>
> >> >> 2018-06-28T09:02:23.282670+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> dlm:3#011(sitea-2)<br>
> >> >> 2018-06-28T09:02:23.282752+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> clvm:3#011(sitea-2)<br>
> >> >> 2018-06-28T09:02:23.282833+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> vg1:3#011(sitea-2)<br>
> >> >> 2018-06-28T09:02:23.282916+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> sysinfo:0#011(sitea-1)<br>
> >> >> 2018-06-28T09:02:23.283001+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> sysinfo:3#011(sitea-2)<br>
> >> >> 2018-06-28T09:02:23.283978+02:00 siteb-2 pengine[189259]:
notice: <br>
> >> >> Calculated transition 1056, saving inputs in <br>
> >> >> /var/lib/pacemaker/pengine/pe-input-2321.bz2<br>
> >> >> 2018-06-28T09:02:23.284428+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Processing graph 1056 (ref=pe_calc-dc-1530169343-1339)
derived from <br>
> >> >> /var/lib/pacemaker/pengine/pe-input-2321.bz2<br>
> >> >> 2018-06-28T09:02:23.284575+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation stonith-sbd_stop_0 on
sitea-1<br>
> >> >> 2018-06-28T09:02:23.284659+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation cl-info_stop_0 on sitea-1<br>
> >> >> 2018-06-28T09:02:23.284742+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation k45RG_stop_0 on sitea-2<br>
> >> >> 2018-06-28T09:02:23.284824+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation vg1_stop_0 on sitea-1<br>
> >> >> 2018-06-28T09:02:23.284908+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation vg1_stop_0 on sitea-2<br>
> >> >> 2018-06-28T09:02:23.284990+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation sysinfo_stop_0 on sitea-1<br>
> >> >> 2018-06-28T09:02:23.285072+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation sysinfo_stop_0 on sitea-2<br>
> >> >> 2018-06-28T09:02:23.288254+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating start operation stonith-sbd_start_0 on
siteb-1<br>
> >> >> 2018-06-28T09:02:23.298867+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating start operation cl-info_start_0 locally
on siteb-2<br>
> >> >> 2018-06-28T09:02:23.309272+02:00 siteb-2 lrmd[189257]:
notice: <br>
> > executing<br>
> >> <br>
> >> >> - rsc:cl-info action:start call_id:105<br>
> >> >> 2018-06-28T09:02:23.384074+02:00 siteb-2 lrmd[189257]:
notice: <br>
> > finished <br>
> >> >> - rsc:cl-info action:start call_id:105 pid:253747
exit-code:0 <br>
> >> >> exec-time:75ms queue-time:0ms<br>
> >> >> 2018-06-28T09:02:23.393759+02:00 siteb-2 crmd[189260]:
notice: <br>
> > Result of<br>
> >> <br>
> >> >> start operation for cl-info on siteb-2: 0 (ok)<br>
> >> >> 2018-06-28T09:02:23.395594+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating monitor operation cl-info_monitor_60000
locally on siteb-2<br>
> >> >> 2018-06-28T09:02:24.159586+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation clvm_stop_0 on sitea-2<br>
> >> >> 2018-06-28T09:02:24.193317+02:00 siteb-2 kernel:
[80844.122213] dlm: <br>
> >> >> clvmd: dlm_recover 5<br>
> >> >> 2018-06-28T09:02:24.193349+02:00 siteb-2 kernel:
[80844.122240] dlm: <br>
> >> >> clvmd: dlm_clear_toss 1 done<br>
> >> >> 2018-06-28T09:02:24.193351+02:00 siteb-2 kernel:
[80844.122251] dlm: <br>
> >> >> clvmd: remove member 3<br>
> >> >> 2018-06-28T09:02:24.193352+02:00 siteb-2 kernel:
[80844.122579] dlm: <br>
> >> >> clvmd: dlm_recover_members 3 nodes<br>
> >> >> 2018-06-28T09:02:24.993269+02:00 siteb-2 kernel:
[80844.920751] dlm: <br>
> >> >> clvmd: generation 7 slots 3 1:2 2:4 3:1<br>
> >> >> 2018-06-28T09:02:24.993283+02:00 siteb-2 kernel:
[80844.920755] dlm: <br>
> >> >> clvmd: dlm_recover_directory<br>
> >> >> 2018-06-28T09:02:25.189265+02:00 siteb-2 kernel:
[80845.118173] dlm: <br>
> >> >> clvmd: dlm_recover_directory 1 in 0 new<br>
> >> >> 2018-06-28T09:02:25.604661+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation clvm_stop_0 on sitea-1<br>
> >> >> 2018-06-28T09:02:26.189277+02:00 siteb-2 kernel:
[80846.116630] dlm: <br>
> >> >> clvmd: dlm_recover_directory 0 out 2 messages<br>
> >> >> 2018-06-28T09:02:26.189291+02:00 siteb-2 kernel:
[80846.116635] dlm: <br>
> >> >> clvmd: dlm_recover_masters<br>
> >> >> 2018-06-28T09:02:26.189292+02:00 siteb-2 kernel:
[80846.116648] dlm: <br>
> >> >> clvmd: dlm_recover_masters 0 of 2<br>
> >> >> 2018-06-28T09:02:26.189293+02:00 siteb-2 kernel:
[80846.116649] dlm: <br>
> >> >> clvmd: dlm_recover_locks 0 out<br>
> >> >> 2018-06-28T09:02:26.208948+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation dlm_stop_0 on sitea-2<br>
> >> >> 2018-06-28T09:02:26.789266+02:00 siteb-2 kernel:
[80846.716532] dlm: <br>
> >> >> clvmd: dlm_recover_locks 0 in<br>
> >> >> 2018-06-28T09:02:27.013264+02:00 siteb-2 kernel:
[80846.940071] dlm: <br>
> >> >> clvmd: dlm_recover_done_wait error -4<br>
> >> >> 2018-06-28T09:02:27.013276+02:00 siteb-2 kernel:
[80846.940076] dlm: <br>
> >> >> clvmd: dlm_recover 5 error -4<br>
> >> >> 2018-06-28T09:02:27.093263+02:00 siteb-2 kernel:
[80847.022308] dlm: <br>
> >> >> clvmd: dlm_recover 7<br>
> >> >> 2018-06-28T09:02:27.093273+02:00 siteb-2 kernel:
[80847.022372] dlm: <br>
> >> >> clvmd: prev removed member 3<br>
> >> >> 2018-06-28T09:02:27.093274+02:00 siteb-2 kernel:
[80847.022375] dlm: <br>
> >> >> clvmd: remove member 1<br>
> >> >> 2018-06-28T09:02:27.493266+02:00 siteb-2 kernel:
[80847.420198] dlm: <br>
> >> >> clvmd: dlm_recover_members 2 nodes<br>
> >> >> 2018-06-28T09:02:38.414440+02:00 siteb-2 corosync[189245]:
[TOTEM ] <br>
> > A <br>
> >> >> processor failed, forming new configuration.<br>
> >> >> 2018-06-28T09:02:52.080141+02:00 siteb-2 corosync[189245]:
[TOTEM ] <br>
> > A <br>
> >> >> new membership (192.168.121.55:2012) was formed.
Members left: 2<br>
> >> >> 2018-06-28T09:02:52.080537+02:00 siteb-2 corosync[189245]:
[TOTEM ] <br>
> > <br>
> >> >> Failed to receive the leave message. failed: 2<br>
> >> >> 2018-06-28T09:02:52.083415+02:00 siteb-2 attrd[189258]:
notice: <br>
> > Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.083533+02:00 siteb-2 attrd[189258]:
notice: <br>
> > Removing<br>
> >> <br>
> >> >> all siteb-1 attributes for peer loss<br>
> >> >> 2018-06-28T09:02:52.083617+02:00 siteb-2 attrd[189258]:
notice: <br>
> > Purged 1<br>
> >> <br>
> >> >> peer with id=2 and/or uname=siteb-1 from the membership
cache<br>
> >> >> 2018-06-28T09:02:52.083700+02:00 siteb-2 cib[189255]:
notice: Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.083793+02:00 siteb-2 cib[189255]:
notice: <br>
> > Purged 1 <br>
> >> >> peer with id=2 and/or uname=siteb-1 from the membership
cache<br>
> >> >> 2018-06-28T09:02:52.083875+02:00 siteb-2 stonith-ng[189256]:
notice: <br>
> >> >> Node siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.083968+02:00 siteb-2 stonith-ng[189256]:
notice: <br>
> >> >> Purged 1 peer with id=2 and/or uname=siteb-1 from
the membership <br>
> > cache<br>
> >> >> 2018-06-28T09:02:52.084054+02:00 siteb-2 crmd[189260]:
warning: No <br>
> > reason<br>
> >> <br>
> >> >> to expect node 2 to be down<br>
> >> >> 2018-06-28T09:02:52.084156+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Stonith/shutdown of siteb-1 not matched<br>
> >> >> 2018-06-28T09:02:52.084242+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Transition aborted: Node failure<br>
> >> >> 2018-06-28T09:02:52.084327+02:00 siteb-2 crmd[189260]:
notice: <br>
> > Action 17<br>
> >> <br>
> >> >> (stonith-sbd_start_0) was pending on 2 (offline)<br>
> >> >> 2018-06-28T09:02:52.084409+02:00 siteb-2 corosync[189245]:
[QUORUM] <br>
> > <br>
> >> >> Members[3]: 1 3 4<br>
> >> >> 2018-06-28T09:02:52.084492+02:00 siteb-2 corosync[189245]:
[MAIN ] <br>
> > <br>
> >> >> Completed service synchronization, ready to provide
service.<br>
> >> >> 2018-06-28T09:02:52.084574+02:00 siteb-2 pacemakerd[189252]:
notice: <br>
> >> >> Node siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.084667+02:00 siteb-2 crmd[189260]:
notice: Node <br>
> > <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.084749+02:00 siteb-2 crmd[189260]:
warning: No <br>
> > reason<br>
> >> <br>
> >> >> to expect node 2 to be down<br>
> >> >> 2018-06-28T09:02:52.084831+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Stonith/shutdown of siteb-1 not matched<br>
> >> >> 2018-06-28T09:02:52.085203+02:00 siteb-2 kernel:
[80872.011754] dlm: <br>
> >> >> clvmd: dlm_recover_members_wait error -4<br>
> >> >> 2018-06-28T09:02:52.085209+02:00 siteb-2 kernel:
[80872.011757] dlm: <br>
> >> >> clvmd: dlm_recover 7 error -4<br>
> >> >> 2018-06-28T09:02:52.085210+02:00 siteb-2 kernel:
[80872.012486] dlm: <br>
> >> >> closing connection to node 2<br>
> >> >> 2018-06-28T09:02:52.086174+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Transition 1056 (Complete=18, Pending=0, Fired=0,
Skipped=4, <br>
> > Incomplete=4,<br>
> >> <br>
> >> >> Source=/var/lib/pacemaker/pengine/pe-input-2321.bz2):
Stopped<br>
> >> >> 2018-06-28T09:02:53.095172+02:00 siteb-2 pengine[189259]:
notice: <br>
> >> >> Watchdog will be used via SBD if fencing is required<br>
> >> >> 2018-06-28T09:02:53.095348+02:00 siteb-2 pengine[189259]:
notice: <br>
> > On <br>
> >> >> loss of CCM Quorum: Ignore<br>
> >> >> 2018-06-28T09:02:53.095584+02:00 siteb-2 pengine[189259]:
warning: <br>
> >> >> Cluster node siteb-1 will be fenced: peer is no
longer part of the <br>
> > cluster<br>
> >> >> 2018-06-28T09:02:53.095673+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Node <br>
> >> >> siteb-1 is unclean<br>
> >> >> 2018-06-28T09:02:53.097943+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Action<br>
> >> <br>
> >> >> stonith-sbd_stop_0 on siteb-1 is unrunnable (offline)<br>
> >> >> 2018-06-28T09:02:53.098028+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Action<br>
> >> <br>
> >> >> dlm:2_stop_0 on siteb-1 is unrunnable (offline)<br>
> >> >> 2018-06-28T09:02:53.098110+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Action<br>
> >> <br>
> >> >> dlm:2_stop_0 on siteb-1 is unrunnable (offline)<br>
> >> >> 2018-06-28T09:02:53.098191+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Action<br>
> >> <br>
> >> >> clvm:2_stop_0 on siteb-1 is unrunnable (offline)<br>
> >> >> 2018-06-28T09:02:53.098273+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Action<br>
> >> <br>
> >> >> clvm:2_stop_0 on siteb-1 is unrunnable (offline)<br>
> >> >> 2018-06-28T09:02:53.098354+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Action<br>
> >> <br>
> >> >> vg1:2_stop_0 on siteb-1 is unrunnable (offline)<br>
> >> >> 2018-06-28T09:02:53.098436+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Action<br>
> >> <br>
> >> >> vg1:2_stop_0 on siteb-1 is unrunnable (offline)<br>
> >> >> 2018-06-28T09:02:53.098521+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Action<br>
> >> <br>
> >> >> sysinfo:1_stop_0 on siteb-1 is unrunnable (offline)<br>
> >> >> 2018-06-28T09:02:53.098603+02:00 siteb-2 pengine[189259]:
warning: <br>
> > Action<br>
> >> <br>
> >> >> sysinfo:1_stop_0 on siteb-1 is unrunnable (offline)<br>
> >> >> 2018-06-28T09:02:53.098683+02:00 siteb-2 pengine[189259]:
warning: <br>
> >> >> Scheduling Node siteb-1 for STONITH<br>
> >> >> 2018-06-28T09:02:53.098764+02:00 siteb-2 pengine[189259]:
notice: * <br>
> > <br>
> >> >> Fence siteb-1<br>
> >> >> 2018-06-28T09:02:53.098846+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Move <br>
> >> >> stonith-sbd#011(Started siteb-1 -> siteb-2)<br>
> >> >> 2018-06-28T09:02:53.098928+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> dlm:0#011(sitea-1)<br>
> >> >> 2018-06-28T09:02:53.099010+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> dlm:2#011(siteb-1)<br>
> >> >> 2018-06-28T09:02:53.099094+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> clvm:2#011(siteb-1)<br>
> >> >> 2018-06-28T09:02:53.099177+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> vg1:2#011(siteb-1)<br>
> >> >> 2018-06-28T09:02:53.099258+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> sysinfo:1#011(siteb-1)<br>
> >> >> 2018-06-28T09:02:53.099339+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Start <br>
> >> <br>
> >> >> k45RG#011(siteb-2)<br>
> >> >> 2018-06-28T09:02:53.099715+02:00 siteb-2 pengine[189259]:
warning: <br>
> >> >> Calculated transition 1057 (with warnings), saving
inputs in <br>
> >> >> /var/lib/pacemaker/pengine/pe-warn-6.bz2<br>
> >> >> 2018-06-28T09:02:53.100282+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Processing graph 1057 (ref=pe_calc-dc-1530169373-1354)
derived from <br>
> >> >> /var/lib/pacemaker/pengine/pe-warn-6.bz2<br>
> >> >> 2018-06-28T09:02:53.100407+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating start operation stonith-sbd_start_0 locally
on siteb-2<br>
> >> >> 2018-06-28T09:02:53.112921+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Requesting fencing (reboot) of node siteb-1<br>
> >> >> 2018-06-28T09:02:53.113113+02:00 siteb-2 stonith-ng[189256]:
notice: <br>
> >> >> Client crmd.189260.7d286b46 wants to fence (reboot)
'siteb-1' with <br>
> > device <br>
> >> >> '(any)'<br>
> >> >> 2018-06-28T09:02:53.113308+02:00 siteb-2 stonith-ng[189256]:
notice: <br>
> >> >> Merging stonith action reboot for node siteb-1 originating
from <br>
> > client <br>
> >> >> crmd.189260.7b806ca3 with identical request from
<br>
> >> >> stonith-api.171087@sitea-1.9fe08723 (72s)<br>
> >> >> 2018-06-28T09:03:29.097551+02:00 siteb-2 stonith-ng[189256]:
notice: <br>
> >> >> Operation reboot of siteb-1 by sitea-2 for <br>
> >> >> stonith-api.171087@sitea-1.9fe08723: OK<br>
> >> >> 2018-06-28T09:03:29.097669+02:00 siteb-2 stonith-ng[189256]:
notice: <br>
> >> >> Operation reboot of siteb-1 by sitea-2 for <br>
> > crmd.189260@siteb-2.7b806ca3: <br>
> >> >> OK<br>
> >> >> 2018-06-28T09:03:29.097756+02:00 siteb-2 crmd[189260]:
notice: Peer <br>
> > <br>
> >> >> siteb-1 was terminated (reboot) by sitea-2 on behalf
of <br>
> >> >> stonith-api.171087: OK<br>
> >> >> 2018-06-28T09:03:29.097863+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Transition aborted: External Fencing Operation<br>
> >> >> 2018-06-28T09:03:29.097948+02:00 siteb-2 crmd[189260]:
notice: <br>
> > Stonith <br>
> >> >> operation 2/51:1057:0:1e551e09-fe55-4e9f-9447-a5199f92bf8c:
OK (0)<br>
> >> >> 2018-06-28T09:03:29.098036+02:00 siteb-2 crmd[189260]:
notice: Peer <br>
> > <br>
> >> >> siteb-1 was terminated (reboot) by sitea-2 on behalf
of crmd.189260: <br>
> > OK<br>
> >> >> 2018-06-28T09:03:29.098196+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Transition 1057 (Complete=14, Pending=0, Fired=0,
Skipped=2, <br>
> > Incomplete=6,<br>
> >> <br>
> >> >> Source=/var/lib/pacemaker/pengine/pe-warn-6.bz2):
Stopped<br>
> >> >> 2018-06-28T09:03:30.100556+02:00 siteb-2 dlm_controld[205418]:
80917 <br>
> > fence<br>
> >> <br>
> >> >> status 2 receive 0 from 1 walltime 1530169410 local
80917<br>
> >> >> 2018-06-28T09:03:30.101210+02:00 siteb-2 kernel:
[80910.025422] dlm: <br>
> >> >> clvmd: dlm_recover 9<br>
> >> >> 2018-06-28T09:03:30.101216+02:00 siteb-2 kernel:
[80910.025444] dlm: <br>
> >> >> clvmd: prev removed member 1<br>
> >> >> 2018-06-28T09:03:30.101216+02:00 siteb-2 kernel:
[80910.025446] dlm: <br>
> >> >> clvmd: prev removed member 3<br>
> >> >> 2018-06-28T09:03:30.101217+02:00 siteb-2 kernel:
[80910.025446] dlm: <br>
> >> >> clvmd: remove member 2<br>
> >> >> 2018-06-28T09:03:30.101217+02:00 siteb-2 kernel:
[80910.025448] dlm: <br>
> >> >> clvmd: dlm_recover_members 1 nodes<br>
> >> >> 2018-06-28T09:03:30.101219+02:00 siteb-2 kernel:
[80910.025450] dlm: <br>
> >> >> clvmd: generation 8 slots 1 2:4<br>
> >> >> 2018-06-28T09:03:30.101222+02:00 siteb-2 kernel:
[80910.025452] dlm: <br>
> >> >> clvmd: dlm_recover_directory<br>
> >> >> 2018-06-28T09:03:30.101223+02:00 siteb-2 kernel:
[80910.025452] dlm: <br>
> >> >> clvmd: dlm_recover_directory 0 in 0 new<br>
> >> >> 2018-06-28T09:03:30.101223+02:00 siteb-2 kernel:
[80910.025454] dlm: <br>
> >> >> clvmd: dlm_recover_directory 0 out 0 messages<br>
> >> >> 2018-06-28T09:03:30.101223+02:00 siteb-2 kernel:
[80910.025455] dlm: <br>
> >> >> clvmd: dlm_recover_masters<br>
> >> >> 2018-06-28T09:03:30.101224+02:00 siteb-2 kernel:
[80910.025456] dlm: <br>
> >> >> clvmd: dlm_recover_masters 2 of 2<br>
> >> >> 2018-06-28T09:03:30.101224+02:00 siteb-2 kernel:
[80910.025457] dlm: <br>
> >> >> clvmd: dlm_recover_locks 0 out<br>
> >> >> 2018-06-28T09:03:30.101226+02:00 siteb-2 kernel:
[80910.025458] dlm: <br>
> >> >> clvmd: dlm_recover_locks 0 in<br>
> >> >> 2018-06-28T09:03:30.101227+02:00 siteb-2 kernel:
[80910.025459] dlm: <br>
> >> >> clvmd: dlm_recover_rsbs 2 done<br>
> >> >> 2018-06-28T09:03:30.101227+02:00 siteb-2 kernel:
[80910.025470] dlm: <br>
> >> >> clvmd: dlm_recover 9 generation 8 done: 0 ms<br>
> >> >> 2018-06-28T09:03:30.103542+02:00 siteb-2 pengine[189259]:
notice: <br>
> >> >> Watchdog will be used via SBD if fencing is required<br>
> >> >> 2018-06-28T09:03:30.103660+02:00 siteb-2 pengine[189259]:
notice: <br>
> > On <br>
> >> >> loss of CCM Quorum: Ignore<br>
> >> >> 2018-06-28T09:03:30.106439+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Stop <br>
> >> >> dlm:0#011(sitea-1)<br>
> >> >> 2018-06-28T09:03:30.106528+02:00 siteb-2 pengine[189259]:
notice: <br>
> > Start <br>
> >> <br>
> >> >> k45RG#011(siteb-2)<br>
> >> >> 2018-06-28T09:03:30.107002+02:00 siteb-2 pengine[189259]:
notice: <br>
> >> >> Calculated transition 1058, saving inputs in <br>
> >> >> /var/lib/pacemaker/pengine/pe-input-2322.bz2<br>
> >> >> 2018-06-28T09:03:30.107292+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Processing graph 1058 (ref=pe_calc-dc-1530169410-1356)
derived from <br>
> >> >> /var/lib/pacemaker/pengine/pe-input-2322.bz2<br>
> >> >> 2018-06-28T09:03:30.107384+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating start operation k45RG_start_0 locally
on siteb-2<br>
> >> >> 2018-06-28T09:03:30.117125+02:00 siteb-2 lrmd[189257]:
notice: <br>
> > executing<br>
> >> <br>
> >> >> - rsc:k45RG action:start call_id:108<br>
> >> >> 2018-06-28T09:03:30.117427+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating stop operation dlm_stop_0 on sitea-1<br>
> >> >> 2018-06-28T09:03:44.279235+02:00 siteb-2 lrmd[189257]:
notice: <br>
> > finished <br>
> >> >> - rsc:k45RG action:start call_id:108 pid:254501
exit-code:0 <br>
> >> >> exec-time:14162ms queue-time:0ms<br>
> >> >> 2018-06-28T09:03:44.287804+02:00 siteb-2 crmd[189260]:
notice: <br>
> > Result of<br>
> >> <br>
> >> >> start operation for k45RG on siteb-2: 0 (ok)<br>
> >> >> 2018-06-28T09:03:44.289761+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Initiating monitor operation k45RG_monitor_300000
locally on siteb-2<br>
> >> >> 2018-06-28T09:03:45.952776+02:00 siteb-2 crmd[189260]:
notice: <br>
> >> >> Transition 1058 (Complete=8, Pending=0, Fired=0,
Skipped=0, <br>
> > Incomplete=0, <br>
> >> >> Source=/var/lib/pacemaker/pengine/pe-input-2322.bz2):
Complete<br>
> >> >> 2018-06-28T09:03:45.952976+02:00 siteb-2 crmd[189260]:
notice: <br>
> > State <br>
> >> >> transition S_TRANSITION_ENGINE -> S_IDLE<br>
> >> >> -------------------<br>
> >> >> Node sitea-1:<br>
> >> >> .....<br>
> >> >> 2018-06-28T09:02:23.504692+02:00 sitea-1 LVM(vg1)[170755]:
INFO: <br>
> >> >> Deactivating volume group clustervg<br>
> >> >> 2018-06-28T09:02:23.937007+02:00 sitea-1 multipathd[3567]:
dm-12: <br>
> > remove <br>
> >> >> map (uevent)<br>
> >> >> 2018-06-28T09:02:23.937371+02:00 sitea-1 multipathd[3567]:
dm-12: <br>
> > devmap <br>
> >> >> not registered, can't remove<br>
> >> >> 2018-06-28T09:02:24.193367+02:00 sitea-1 kernel:
[59480.678902] dlm: <br>
> >> >> clvmd: dlm_recover 13<br>
> >> >> 2018-06-28T09:02:24.193379+02:00 sitea-1 kernel:
[59480.678938] dlm: <br>
> >> >> clvmd: remove member 3<br>
> >> >> 2018-06-28T09:02:24.393411+02:00 sitea-1 kernel:
[59480.878296] dlm: <br>
> >> >> clvmd: dlm_recover_members 3 nodes<br>
> >> >> 2018-06-28T09:02:24.593412+02:00 sitea-1 kernel:
[59481.078227] dlm: <br>
> >> >> clvmd: generation 7 slots 3 1:2 2:4 3:1<br>
> >> >> 2018-06-28T09:02:24.593423+02:00 sitea-1 kernel:
[59481.078232] dlm: <br>
> >> >> clvmd: dlm_recover_directory<br>
> >> >> 2018-06-28T09:02:24.793410+02:00 sitea-1 kernel:
[59481.278263] dlm: <br>
> >> >> clvmd: dlm_recover_directory 0 in 0 new<br>
> >> >> 2018-06-28T09:02:25.017465+02:00 sitea-1 multipathd[3567]:
dm-12: <br>
> > remove <br>
> >> >> map (uevent)<br>
> >> >> 2018-06-28T09:02:25.017639+02:00 sitea-1 multipathd[3567]:
dm-13: <br>
> > remove <br>
> >> >> map (uevent)<br>
> >> >> 2018-06-28T09:02:25.017724+02:00 sitea-1 multipathd[3567]:
dm-13: <br>
> > devmap <br>
> >> >> not registered, can't remove<br>
> >> >> 2018-06-28T09:02:25.017801+02:00 sitea-1 multipathd[3567]:
dm-13: <br>
> > remove <br>
> >> >> map (uevent)<br>
> >> >> 2018-06-28T09:02:25.589352+02:00 sitea-1 kernel:
[59482.074179] dlm: <br>
> >> >> clvmd: dlm_recover_directory 1 out 2 messages<br>
> >> >> 2018-06-28T09:02:25.589357+02:00 sitea-1 kernel:
[59482.074192] dlm: <br>
> >> >> clvmd: dlm_recover_masters<br>
> >> >> 2018-06-28T09:02:25.589357+02:00 sitea-1 kernel:
[59482.074201] dlm: <br>
> >> >> clvmd: dlm_recover_masters 1 of 3<br>
> >> >> 2018-06-28T09:02:25.591207+02:00 sitea-1 LVM(vg1)[170755]:
INFO: 0 <br>
> >> >> logical volume(s) in volume group "clustervg"
now active<br>
> >> >> 2018-06-28T09:02:25.596699+02:00 sitea-1 LVM(vg1)[170755]:
INFO: LVM <br>
> >> >> Volume clustervg is not available (stopped)<br>
> >> >> 2018-06-28T09:02:25.600665+02:00 sitea-1 lrmd[9025]:
notice: <br>
> > finished - <br>
> >> >> rsc:vg1 action:stop call_id:80 pid:170755 exit-code:0
<br>
> > exec-time:2315ms <br>
> >> >> queue-time:0ms<br>
> >> >> 2018-06-28T09:02:25.600897+02:00 sitea-1 crmd[9031]:
notice: Result <br>
> > of <br>
> >> >> stop operation for vg1 on sitea-1: 0 (ok)<br>
> >> >> 2018-06-28T09:02:25.604928+02:00 sitea-1 lrmd[9025]:
notice: <br>
> > executing -<br>
> >> <br>
> >> >> rsc:clvm action:stop call_id:83<br>
> >> >> 2018-06-28T09:02:25.617437+02:00 sitea-1 clvmd(clvm)[171004]:
INFO: <br>
> >> >> Stopping clvm<br>
> >> >> 2018-06-28T09:02:25.626327+02:00 sitea-1 clvmd(clvm)[171004]:
INFO: <br>
> >> >> Stopping clvmd<br>
> >> >> 2018-06-28T09:02:25.745414+02:00 sitea-1 kernel:
[59482.230710] dlm: <br>
> >> >> clvmd: dlm_recover_locks 0 out<br>
> >> >> 2018-06-28T09:02:26.389417+02:00 sitea-1 kernel:
[59482.874050] dlm: <br>
> >> >> clvmd: dlm_recover_locks 0 in<br>
> >> >> 2018-06-28T09:02:26.389429+02:00 sitea-1 kernel:
[59482.874056] dlm: <br>
> >> >> clvmd: dlm_recover_rsbs 3 done<br>
> >> >> 2018-06-28T09:02:26.988133+02:00 sitea-1 kernel:
[59483.474046] dlm: <br>
> >> >> clvmd: dlm_recover 13 generation 7 done: 2396 ms<br>
> >> >> 2018-06-28T09:02:26.988142+02:00 sitea-1 kernel:
[59483.474240] dlm: <br>
> >> >> clvmd: leaving the lockspace group...<br>
> >> >> 2018-06-28T09:02:26.988142+02:00 sitea-1 kernel:
[59483.475630] dlm: <br>
> >> >> clvmd: group event done 0 0<br>
> >> >> 2018-06-28T09:02:26.988143+02:00 sitea-1 kernel:
[59483.476068] dlm: <br>
> >> >> clvmd: release_lockspace final free<br>
> >> >> 2018-06-28T09:02:27.652248+02:00 sitea-1 clvmd(clvm)[171004]:
INFO: <br>
> >> >> Stopping cmirrord<br>
> >> >> 2018-06-28T09:02:28.666056+02:00 sitea-1 lrmd[9025]:
notice: </font></tt>
<br><tt><font size=2>> > finished - <br>
> >> >> rsc:clvm action:stop call_id:83 pid:171004 exit-code:0
<br>
> > exec-time:3061ms <br>
> >> >> queue-time:0ms<br>
> >> >> 2018-06-28T09:02:28.666439+02:00 sitea-1 crmd[9031]:
notice: Result <br>
> > of <br>
> >> >> stop operation for clvm on sitea-1: 0 (ok)<br>
> >> >> 2018-06-28T09:02:38.413748+02:00 sitea-1 corosync[6661]:
[TOTEM ] A <br>
> > <br>
> >> >> processor failed, forming new configuration.<br>
> >> >> 2018-06-28T09:02:52.079905+02:00 sitea-1 corosync[6661]:
[TOTEM ] A <br>
> > new <br>
> >> >> membership (192.168.121.55:2012) was formed. Members
left: 2<br>
> >> >> 2018-06-28T09:02:52.080306+02:00 sitea-1 corosync[6661]:
[TOTEM ] <br>
> > Failed<br>
> >> <br>
> >> >> to receive the leave message. failed: 2<br>
> >> >> 2018-06-28T09:02:52.082619+02:00 sitea-1 cib[9021]:
notice: Node <br>
> > siteb-1<br>
> >> <br>
> >> >> state is now lost<br>
> >> >> 2018-06-28T09:02:52.082790+02:00 sitea-1 cib[9021]:
notice: Purged <br>
> > 1 <br>
> >> >> peer with id=2 and/or uname=siteb-1 from the membership
cache<br>
> >> >> 2018-06-28T09:02:52.082895+02:00 sitea-1 attrd[9026]:
notice: Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.082993+02:00 sitea-1 attrd[9026]:
notice: <br>
> > Removing <br>
> >> >> all siteb-1 attributes for peer loss<br>
> >> >> 2018-06-28T09:02:52.083075+02:00 sitea-1 attrd[9026]:
notice: <br>
> > Purged 1 <br>
> >> >> peer with id=2 and/or uname=siteb-1 from the membership
cache<br>
> >> >> 2018-06-28T09:02:52.083156+02:00 sitea-1 stonith-ng[9022]:
notice: <br>
> > Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.083253+02:00 sitea-1 stonith-ng[9022]:
notice: <br>
> >> >> Purged 1 peer with id=2 and/or uname=siteb-1 from
the membership <br>
> > cache<br>
> >> >> 2018-06-28T09:02:52.083335+02:00 sitea-1 pacemakerd[8546]:
notice: <br>
> > Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.083429+02:00 sitea-1 corosync[6661]:
[QUORUM] <br>
> >> >> Members[3]: 1 3 4<br>
> >> >> 2018-06-28T09:02:52.083521+02:00 sitea-1 corosync[6661]:
[MAIN ] <br>
> >> >> Completed service synchronization, ready to provide
service.<br>
> >> >> 2018-06-28T09:02:52.083606+02:00 sitea-1 crmd[9031]:
notice: Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.084290+02:00 sitea-1 dlm_controld[73416]:
59514 <br>
> > fence <br>
> >> >> request 2 pid 171087 nodedown time 1530169372 fence_all
dlm_stonith<br>
> >> >> 2018-06-28T09:02:52.085446+02:00 sitea-1 kernel:
[59508.568940] dlm: <br>
> >> >> closing connection to node 2<br>
> >> >> 2018-06-28T09:02:52.109393+02:00 sitea-1 dlm_stonith:
<br>
> > stonith_api_time: <br>
> >> >> Found 0 entries for 2/(null): 0 in progress, 0 completed<br>
> >> >> 2018-06-28T09:02:52.110167+02:00 sitea-1 stonith-ng[9022]:
notice: <br>
> >> >> Client stonith-api.171087.d3c59fc2 wants to fence
(reboot) '2' with <br>
> > device<br>
> >> <br>
> >> >> '(any)'<br>
> >> >> 2018-06-28T09:02:52.113257+02:00 sitea-1 stonith-ng[9022]:
notice: <br>
> >> >> Requesting peer fencing (reboot) of siteb-1<br>
> >> >> 2018-06-28T09:03:29.096714+02:00 sitea-1 stonith-ng[9022]:
notice: <br>
> >> >> Operation reboot of siteb-1 by sitea-2 for <br>
> >> >> stonith-api.171087@sitea-1.9fe08723: OK<br>
> >> >> 2018-06-28T09:03:29.097152+02:00 sitea-1 stonith-api[171087]:
<br>
> >> >> stonith_api_kick: Node 2/(null) kicked: reboot<br>
> >> >> 2018-06-28T09:03:29.097426+02:00 sitea-1 crmd[9031]:
notice: Peer <br>
> >> >> lnx0361b was terminated (reboot) by sitea-2 on behalf
of <br>
> >> >> stonith-api.171087: OK<br>
> >> >> 2018-06-28T09:03:30.098657+02:00 sitea-1 dlm_controld[73416]:
59552 <br>
> > fence <br>
> >> >> result 2 pid 171087 result 0 exit status<br>
> >> >> 2018-06-28T09:03:30.099730+02:00 sitea-1 dlm_controld[73416]:
59552 <br>
> > fence <br>
> >> >> status 2 receive 0 from 1 walltime 1530169410 local
59552<br>
> >> >> 2018-06-28T09:03:30.118114+02:00 sitea-1 lrmd[9025]:
notice: <br>
> > executing -<br>
> >> <br>
> >> >> rsc:dlm action:stop call_id:85<br>
> >> >> 2018-06-28T09:03:30.149362+02:00 sitea-1 kernel:
[59546.630487] dlm: <br>
> >> >> closing connection to node 4<br>
> >> >> 2018-06-28T09:03:30.149370+02:00 sitea-1 kernel:
[59546.630496] dlm: <br>
> >> >> closing connection to node 3<br>
> >> >> 2018-06-28T09:03:30.149371+02:00 sitea-1 kernel:
[59546.630501] dlm: <br>
> >> >> closing connection to node 1<br>
> >> >> 2018-06-28T09:03:31.168356+02:00 sitea-1 lrmd[9025]:
notice: <br>
> > finished - <br>
> >> >> rsc:dlm action:stop call_id:85 pid:171235 exit-code:0
<br>
> > exec-time:1050ms <br>
> >> >> queue-time:0ms<br>
> >> >> 2018-06-28T09:03:31.168768+02:00 sitea-1 crmd[9031]:
notice: Result <br>
> > of <br>
> >> >> stop operation for dlm on lnx0361a: 0 (ok)<br>
> >> >> <br>
> >> >> -------------------<br>
> >> >> Node sitea-2:<br>
> >> >> ...<br>
> >> >> 2018-06-28T09:02:23.285402+02:00 sitea-2 lrmd[9657]:
notice: <br>
> > executing -<br>
> >> <br>
> >> >> rsc:vg1 action:stop call_id:78<br>
> >> >> 2018-06-28T09:02:23.286028+02:00 sitea-2 lrmd[9657]:
notice: <br>
> > executing -<br>
> >> <br>
> >> >> rsc:sysinfo action:stop call_id:80<br>
> >> >> 2018-06-28T09:02:23.293918+02:00 sitea-2 lrmd[9657]:
notice: <br>
> > finished - <br>
> >> >> rsc:sysinfo action:stop call_id:80 pid:228481 exit-code:0
<br>
> > exec-time:8ms <br>
> >> >> queue-time:0ms<br>
> >> >> 2018-06-28T09:02:23.294070+02:00 sitea-2 crmd[9660]:
notice: Result <br>
> > of <br>
> >> >> stop operation for sysinfo on sitea-2: 0 (ok)<br>
> >> >> 2018-06-28T09:02:23.448868+02:00 sitea-2 LVM(vg1)[228480]:
INFO: <br>
> >> >> Deactivating volume group clustervg<br>
> >> >> 2018-06-28T09:02:23.494608+02:00 sitea-2 multipathd[3459]:
dm-12: <br>
> > remove <br>
> >> >> map (uevent)<br>
> >> >> 2018-06-28T09:02:23.494774+02:00 sitea-2 multipathd[3459]:
dm-12: <br>
> > devmap <br>
> >> >> not registered, can't remove<br>
> >> >> 2018-06-28T09:02:24.147057+02:00 sitea-2 LVM(vg1)[228480]:
INFO: 0 <br>
> >> >> logical volume(s) in volume group "clustervg"
now active<br>
> >> >> 2018-06-28T09:02:24.152209+02:00 sitea-2 LVM(vg1)[228480]:
INFO: LVM <br>
> >> >> Volume clustervg is not available (stopped)<br>
> >> >> 2018-06-28T09:02:24.155824+02:00 sitea-2 lrmd[9657]:
notice: <br>
> > finished - <br>
> >> >> rsc:vg1 action:stop call_id:78 pid:228480 exit-code:0
exec-time:871ms <br>
> > <br>
> >> >> queue-time:0ms<br>
> >> >> 2018-06-28T09:02:24.156133+02:00 sitea-2 crmd[9660]:
notice: Result <br>
> > of <br>
> >> >> stop operation for vg1 on sitea-2: 0 (ok)<br>
> >> >> 2018-06-28T09:02:24.159361+02:00 sitea-2 lrmd[9657]:
notice: <br>
> > executing -<br>
> >> <br>
> >> >> rsc:clvm action:stop call_id:81<br>
> >> >> 2018-06-28T09:02:24.169269+02:00 sitea-2 clvmd(clvm)[229478]:
INFO: <br>
> >> >> Stopping clvm<br>
> >> >> 2018-06-28T09:02:24.177998+02:00 sitea-2 clvmd(clvm)[229478]:
INFO: <br>
> >> >> Stopping clvmd<br>
> >> >> 2018-06-28T09:02:24.187070+02:00 sitea-2 kernel:
[59548.133661] dlm: <br>
> >> >> clvmd: leaving the lockspace group...<br>
> >> >> 2018-06-28T09:02:24.187078+02:00 sitea-2 kernel:
[59548.134502] dlm: <br>
> >> >> clvmd: group event done 0 0<br>
> >> >> 2018-06-28T09:02:24.187078+02:00 sitea-2 kernel:
[59548.134863] dlm: <br>
> >> >> clvmd: release_lockspace final free<br>
> >> >> 2018-06-28T09:02:24.707189+02:00 sitea-2 multipathd[3459]:
dm-12: <br>
> > remove <br>
> >> >> map (uevent)<br>
> >> >> 2018-06-28T09:02:24.707584+02:00 sitea-2 multipathd[3459]:
dm-13: <br>
> > remove <br>
> >> >> map (uevent)<br>
> >> >> 2018-06-28T09:02:24.707672+02:00 sitea-2 multipathd[3459]:
dm-13: <br>
> > devmap <br>
> >> >> not registered, can't remove<br>
> >> >> 2018-06-28T09:02:24.707754+02:00 sitea-2 multipathd[3459]:
dm-13: <br>
> > remove <br>
> >> >> map (uevent)<br>
> >> >> 2018-06-28T09:02:25.193751+02:00 sitea-2 clvmd(clvm)[229478]:
INFO: <br>
> >> >> Stopping cmirrord<br>
> >> >> 2018-06-28T09:02:26.204708+02:00 sitea-2 lrmd[9657]:
notice: <br>
> > finished - <br>
> >> >> rsc:clvm action:stop call_id:81 pid:229478 exit-code:0
<br>
> > exec-time:2046ms <br>
> >> >> queue-time:0ms<br>
> >> >> 2018-06-28T09:02:26.204942+02:00 sitea-2 crmd[9660]:
notice: Result <br>
> > of <br>
> >> >> stop operation for clvm on sitea-2: 0 (ok)<br>
> >> >> 2018-06-28T09:02:26.208588+02:00 sitea-2 lrmd[9657]:
notice: <br>
> > executing -<br>
> >> <br>
> >> >> rsc:dlm action:stop call_id:83<br>
> >> >> 2018-06-28T09:02:26.236329+02:00 sitea-2 kernel:
[59550.182284] dlm: <br>
> >> >> closing connection to node 4<br>
> >> >> 2018-06-28T09:02:26.236336+02:00 sitea-2 kernel:
[59550.182293] dlm: <br>
> >> >> closing connection to node 3<br>
> >> >> 2018-06-28T09:02:26.236337+02:00 sitea-2 kernel:
[59550.182299] dlm: <br>
> >> >> closing connection to node 2<br>
> >> >> 2018-06-28T09:02:26.236337+02:00 sitea-2 kernel:
[59550.182306] dlm: <br>
> >> >> closing connection to node 1<br>
> >> >> 2018-06-28T09:02:27.248204+02:00 sitea-2 lrmd[9657]:
notice: <br>
> > finished - <br>
> >> >> rsc:dlm action:stop call_id:83 pid:230674 exit-code:0
<br>
> > exec-time:1039ms <br>
> >> >> queue-time:0ms<br>
> >> >> 2018-06-28T09:02:27.248462+02:00 sitea-2 crmd[9660]:
notice: Result <br>
> > of <br>
> >> >> stop operation for dlm on sitea-2: 0 (ok)<br>
> >> >> 2018-06-28T09:02:30.454215+02:00 sitea-2 lrmd[9657]:
notice: <br>
> > finished - <br>
> >> >> rsc:k45RG action:stop call_id:76 pid:228479 exit-code:0
<br>
> > exec-time:7169ms <br>
> >> >> queue-time:0ms<br>
> >> >> 2018-06-28T09:02:30.454595+02:00 sitea-2 crmd[9660]:
notice: Result <br>
> > of <br>
> >> >> stop operation for k45RG on sitea-2: 0 (ok)<br>
> >> >> 2018-06-28T09:02:38.412808+02:00 sitea-2 corosync[6570]:
[TOTEM ] A <br>
> > <br>
> >> >> processor failed, forming new configuration.<br>
> >> >> 2018-06-28T09:02:52.078249+02:00 sitea-2 corosync[6570]:
[TOTEM ] A <br>
> > new <br>
> >> >> membership (192.168.121.55:2012) was formed. Members
left: 2<br>
> >> >> 2018-06-28T09:02:52.078359+02:00 sitea-2 corosync[6570]:
[TOTEM ] <br>
> > Failed<br>
> >> <br>
> >> >> to receive the leave message. failed: 2<br>
> >> >> 2018-06-28T09:02:52.081949+02:00 sitea-2 cib[9655]:
notice: Node <br>
> > siteb-1<br>
> >> <br>
> >> >> state is now lost<br>
> >> >> 2018-06-28T09:02:52.082048+02:00 sitea-2 attrd[9658]:
notice: Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.082138+02:00 sitea-2 cib[9655]:
notice: Purged <br>
> > 1 <br>
> >> >> peer with id=2 and/or uname=siteb-1 from the membership
cache<br>
> >> >> 2018-06-28T09:02:52.082220+02:00 sitea-2 attrd[9658]:
notice: <br>
> > Removing <br>
> >> >> all siteb-1 attributes for peer loss<br>
> >> >> 2018-06-28T09:02:52.082300+02:00 sitea-2 stonith-ng[9656]:
notice: <br>
> > Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.082391+02:00 sitea-2 stonith-ng[9656]:
notice: <br>
> >> >> Purged 1 peer with id=2 and/or uname=siteb-1 from
the membership <br>
> > cache<br>
> >> >> 2018-06-28T09:02:52.082470+02:00 sitea-2 attrd[9658]:
notice: <br>
> > Purged 1 <br>
> >> >> peer with id=2 and/or uname=siteb-1 from the membership
cache<br>
> >> >> 2018-06-28T09:02:52.082653+02:00 sitea-2 corosync[6570]:
[QUORUM] <br>
> >> >> Members[3]: 1 3 4<br>
> >> >> 2018-06-28T09:02:52.082739+02:00 sitea-2 corosync[6570]:
[MAIN ] <br>
> >> >> Completed service synchronization, ready to provide
service.<br>
> >> >> 2018-06-28T09:02:52.082819+02:00 sitea-2 pacemakerd[9653]:
notice: <br>
> > Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.082908+02:00 sitea-2 crmd[9660]:
notice: Node <br>
> >> >> siteb-1 state is now lost<br>
> >> >> 2018-06-28T09:02:52.495697+02:00 sitea-2 stonith-ng[9656]:
notice: <br>
> >> >> stonith-sbd can fence (reboot) siteb-1: dynamic-list<br>
> >> >> 2018-06-28T09:02:52.495902+02:00 sitea-2 stonith-ng[9656]:
notice: <br>
> >> >> Delaying reboot on stonith-sbd for 25358ms (timeout=300s)<br>
> >> >> 2018-06-28T09:03:01.357937+02:00 sitea-2 cron[231205]:
<br>
> >> >> pam_unix(crond:session): session opened for user
root by (uid=0)<br>
> >> >> 2018-06-28T09:03:29.093957+02:00 sitea-2 stonith-ng[9656]:
notice: <br>
> >> >> Operation 'reboot' [231293] (call 2 from stonith-api.171087)
for host <br>
> > <br>
> >> >> 'siteb-1' with device 'stonith-sbd' returned: 0
(OK)<br>
> >> >> 2018-06-28T09:03:29.096254+02:00 sitea-2 stonith-ng[9656]:
notice: <br>
> >> >> Operation reboot of siteb-1 by sitea-2 for <br>
> >> >> stonith-api.171087@sitea-1.9fe08723: OK<br>
> >> >> 2018-06-28T09:03:29.096769+02:00 sitea-2 crmd[9660]:
notice: Peer <br>
> >> >> siteb-1 was terminated (reboot) by sitea-2 on behalf
of <br>
> >> >> stonith-api.171087: OK<br>
> >> >> -------------------<br>
> >> >> Node sideb-1 has no entries during this timeframe<br>
> >> >> <br>
> >> >> during standby corosync should be up/running - so
may the "Failed to <br>
> >> >> receive the leave message" will be a Problem?<br>
> >> >> <br>
> >> >> regards<br>
> >> >> Philipp<br>
> >> >> <br>
> >> >>> Von: "Ulrich Windl" <Ulrich.Windl@rz.uni-regensburg.de><br>
> >> >>> An: <users@clusterlabs.org><br>
> >> >>> Datum: 16.07.2018 08:33<br>
> >> >>> Betreff: [ClusterLabs] Antw: corosync/dlm
fencing?<br>
> >> >>> Gesendet von: "Users" <users-bounces@clusterlabs.org><br>
> >> >>> <br>
> >> >>> Hi!<br>
> >> >>> <br>
> >> >>> I don't run SLES 12, but in SLES 11 some RAs
tried to unload kernel <br>
> >> >> modules<br>
> >> >>> during stop, and when you had installed kernel
updates before, the <br>
> >> >> operation<br>
> >> >>> failed, and the cluster paniced (i.e.: started
to fence the node). <br>
> >> >>> In your case<br>
> >> >>> it should be obvious from the logs why the cluster
wants to fence <br>
> > nodes. <br>
> >> >> Maybe<br>
> >> >>> "grep -i fence /var/log/messages"
(or in pacemaker's log)<br>
> >> >>> <br>
> >> >>> Regards,<br>
> >> >>> Ulrich<br>
> >> >>> <br>
> >> >>> >>> Philipp Achmüller <philipp.achmueller@arz.at>
schrieb am <br>
> > 15.07.2018 <br>
> >> >> um<br>
> >> >>> 15:39 in<br>
> >> >>> Nachricht <br>
> >> >> <OF678C848F.04D4334F-ONC12582CB.00473374-C12582CB.004AFD3B@arz.at>:<br>
> >> >>> > hi!<br>
> >> >>> > <br>
> >> >>> > i have a 4 node cluster running on SLES12
SP3<br>
> >> >>> > ‑ pacemaker‑1.1.16‑4.8.x86_64<br>
> >> >>> > ‑ corosync‑2.3.6‑9.5.1.x86_64<br>
> >> >>> > <br>
> >> >>> > following configuration:<br>
> >> >>> > <br>
> >> >>> > Stack: corosync<br>
> >> >>> > Current DC: sitea‑2 (version 1.1.16‑4.8‑77ea74d)
‑ partition <br>
> > with <br>
> >> >> quorum<br>
> >> >>> > Last updated: Sun Jul 15 15:00:55 2018<br>
> >> >>> > Last change: Sat Jul 14 18:54:50 2018 by
root via crm_resource on <br>
> >> >> sitea‑1<br>
> >> >>> > <br>
> >> >>> > 4 nodes configured<br>
> >> >>> > 23 resources configured<br>
> >> >>> > <br>
> >> >>> > Node sitea‑1: online<br>
> >> >>> > 1
(ocf::pacemaker:controld): Active <br>
> >> >>> > 1
(ocf::lvm2:clvmd): Active <br>
> >> >>> > 1
(ocf::pacemaker:SysInfo): Active <br>
> >> >>> > 5
(ocf::heartbeat:VirtualDomain): Active <br>
> >> >>> > 1
(ocf::heartbeat:LVM): Active <br>
> >> >>> > Node siteb‑1: online<br>
> >> >>> > 1
(ocf::pacemaker:controld): Active <br>
> >> >>> > 1
(ocf::lvm2:clvmd): Active <br>
> >> >>> > 1
(ocf::pacemaker:SysInfo): Active <br>
> >> >>> > 1
(ocf::heartbeat:VirtualDomain): Active <br>
> >> >>> > 1
(ocf::heartbeat:LVM): Active <br>
> >> >>> > Node sitea‑2: online<br>
> >> >>> > 1
(ocf::pacemaker:controld): Active <br>
> >> >>> > 1
(ocf::lvm2:clvmd): Active <br>
> >> >>> > 1
(ocf::pacemaker:SysInfo): Active <br>
> >> >>> > 3
(ocf::heartbeat:VirtualDomain): Active <br>
> >> >>> > 1
(ocf::heartbeat:LVM): Active <br>
> >> >>> > Node siteb‑2: online<br>
> >> >>> > 1
(ocf::pacemaker:ClusterMon): Active <br>
> >> >>> > 3
(ocf::heartbeat:VirtualDomain): Active <br>
> >> >>> > 1
(ocf::pacemaker:SysInfo): Active <br>
> >> >>> > 1
(stonith:external/sbd): Active <br>
> >> >>> > 1
(ocf::lvm2:clvmd): Active <br>
> >> >>> > 1
(ocf::heartbeat:LVM): Active <br>
> >> >>> > 1
(ocf::pacemaker:controld): Active <br>
> >> >>> > ‑‑‑‑<br>
> >> >>> > and these ordering:<br>
> >> >>> > ...<br>
> >> >>> > group base‑group dlm clvm vg1<br>
> >> >>> > clone base‑clone base‑group \<br>
> >> >>> > meta interleave=true
target‑role=Started ordered=true<br>
> >> >>> > colocation colocation‑VM‑base‑clone‑INFINITY
inf: VM <br>
> > base‑clone<br>
> >> >>> > order order‑base‑clone‑VM‑mandatory
base‑clone:start VM:start<br>
> >> >>> > ...<br>
> >> >>> > <br>
> >> >>> > for maintenance i would like to standby
1 or 2 nodes from "sitea" <br>
> > so <br>
> >> >> that <br>
> >> >>> > every Resources move off from these 2 images.<br>
> >> >>> > everything works fine until dlm stops as
last resource on these <br>
> > nodes, <br>
> >> >> <br>
> >> >>> > then dlm_controld send fence_request ‑
sometimes to the remaining <br>
> > <br>
> >> >> online <br>
> >> >>> > nodes, so there is online 1 node left in
cluster....<br>
> >> >>> > <br>
> >> >>> > messages:<br>
> >> >>> > <br>
> >> >>> > ....<br>
> >> >>> > 2018‑07‑14T14:38:56.441157+02:00 siteb‑1
dlm_controld[39725]: <br>
> > 678 <br>
> >> >> fence <br>
> >> >>> > request 3 pid 54428 startup time 1531571371
fence_all dlm_stonith<br>
> >> >>> > 2018‑07‑14T14:38:56.445284+02:00 siteb‑1
dlm_stonith: <br>
> >> >> stonith_api_time: <br>
> >> >>> > Found 0 entries for 3/(null): 0 in progress,
0 completed<br>
> >> >>> > 2018‑07‑14T14:38:56.446033+02:00 siteb‑1
stonith‑ng[8085]: <br>
> > notice: <br>
> >> >> <br>
> >> >>> > Client stonith‑api.54428.ee6a7e02 wants
to fence (reboot) '3' <br>
> > with <br>
> >> >> device <br>
> >> >>> > '(any)'<br>
> >> >>> > 2018‑07‑14T14:38:56.446294+02:00 siteb‑1
stonith‑ng[8085]: <br>
> > notice: <br>
> >> >> <br>
> >> >>> > Requesting peer fencing (reboot) of sitea‑2<br>
> >> >>> > ...<br>
> >> >>> > <br>
> >> >>> > # dlm_tool dump_config<br>
> >> >>> > daemon_debug=0<br>
> >> >>> > foreground=0<br>
> >> >>> > log_debug=0<br>
> >> >>> > timewarn=0<br>
> >> >>> > protocol=detect<br>
> >> >>> > debug_logfile=0<br>
> >> >>> > enable_fscontrol=0<br>
> >> >>> > enable_plock=1<br>
> >> >>> > plock_debug=0<br>
> >> >>> > plock_rate_limit=0<br>
> >> >>> > plock_ownership=0<br>
> >> >>> > drop_resources_time=10000<br>
> >> >>> > drop_resources_count=10<br>
> >> >>> > drop_resources_age=10000<br>
> >> >>> > post_join_delay=30<br>
> >> >>> > enable_fencing=1<br>
> >> >>> > enable_concurrent_fencing=0<br>
> >> >>> > enable_startup_fencing=0<br>
> >> >>> > repeat_failed_fencing=1<br>
> >> >>> > enable_quorum_fencing=1<br>
> >> >>> > enable_quorum_lockspace=1<br>
> >> >>> > help=‑1<br>
> >> >>> > version=‑1<br>
> >> >>> > <br>
> >> >>> > how to find out what is happening here?<br>
> >> >>> <br>
> >> >>> <br>
> >> >>> <br>
> >> >>> _______________________________________________<br>
> >> >>> Users mailing list: Users@clusterlabs.org <br>
> >> >>> </font></tt><a href=https://lists.clusterlabs.org/mailman/listinfo/users><tt><font size=2>https://lists.clusterlabs.org/mailman/listinfo/users</font></tt></a><tt><font size=2>
<br>
> >> >>> <br>
> >> >>> Project Home: </font></tt><a href=http://www.clusterlabs.org/><tt><font size=2>http://www.clusterlabs.org</font></tt></a><tt><font size=2>
<br>
> >> >>> Getting started: <br>
> > </font></tt><a href=http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf><tt><font size=2>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</font></tt></a><tt><font size=2>
<br>
> >> >>> Bugs: </font></tt><a href=http://bugs.clusterlabs.org/><tt><font size=2>http://bugs.clusterlabs.org</font></tt></a><tt><font size=2>
<br>
> >> > <br>
> >> > <br>
> >> > <br>
> >> > _______________________________________________<br>
> >> > Users mailing list: Users@clusterlabs.org <br>
> >> > </font></tt><a href=https://lists.clusterlabs.org/mailman/listinfo/users><tt><font size=2>https://lists.clusterlabs.org/mailman/listinfo/users</font></tt></a><tt><font size=2>
<br>
> >> > <br>
> >> > Project Home: </font></tt><a href=http://www.clusterlabs.org/><tt><font size=2>http://www.clusterlabs.org</font></tt></a><tt><font size=2>
<br>
> >> > Getting started: <br>
> > </font></tt><a href=http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf><tt><font size=2>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</font></tt></a><tt><font size=2>
<br>
> >> > Bugs: </font></tt><a href=http://bugs.clusterlabs.org/><tt><font size=2>http://bugs.clusterlabs.org</font></tt></a><tt><font size=2>
<br>
> >> <br>
> >> <br>
> >> <br>
> >> _______________________________________________<br>
> >> Users mailing list: Users@clusterlabs.org <br>
> >> </font></tt><a href=https://lists.clusterlabs.org/mailman/listinfo/users><tt><font size=2>https://lists.clusterlabs.org/mailman/listinfo/users</font></tt></a><tt><font size=2>
<br>
> >> <br>
> >> Project Home: </font></tt><a href=http://www.clusterlabs.org/><tt><font size=2>http://www.clusterlabs.org</font></tt></a><tt><font size=2>
<br>
> >> Getting started: </font></tt><a href=http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf><tt><font size=2>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</font></tt></a><tt><font size=2>
<br>
> >> Bugs: </font></tt><a href=http://bugs.clusterlabs.org/><tt><font size=2>http://bugs.clusterlabs.org</font></tt></a><tt><font size=2>
<br>
> <br>
> <br>
> <br>
> _______________________________________________<br>
> Users mailing list: Users@clusterlabs.org<br>
> </font></tt><a href=https://lists.clusterlabs.org/mailman/listinfo/users><tt><font size=2>https://lists.clusterlabs.org/mailman/listinfo/users</font></tt></a><tt><font size=2><br>
> <br>
> Project Home: </font></tt><a href=http://www.clusterlabs.org/><tt><font size=2>http://www.clusterlabs.org</font></tt></a><tt><font size=2><br>
> Getting started: </font></tt><a href=http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf><tt><font size=2>http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</font></tt></a><tt><font size=2><br>
> Bugs: </font></tt><a href=http://bugs.clusterlabs.org/><tt><font size=2>http://bugs.clusterlabs.org</font></tt></a><tt><font size=2><br>
</font></tt><font size=2 face="sans-serif"><br>
</font>