<div dir="ltr">Hi,<div class="gmail_extra"><br><div class="gmail_quote">On Sun, Sep 11, 2016 at 6:28 PM, Gienek Nowacki <span dir="ltr"><<a href="mailto:nowackig@gmail.com" target="_blank">nowackig@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<br><br>I have a problem with my cluster.<br>There are two nodes: wirt1v and wirt2v with pacemaker, corosync, dlm,<br>drbd (/dev/drbd2) and filesystem mounted as /virtfs2 with gfs2.<br>Each node contains LVM partition on wihich is DRBD.<br><br>The situation is as follow:<br><br>pcs cluster standby wirt2v<br> ...all is OK, there is possible to use /virtfs2 on wirt1v node<br><br>pcs cluster unstandby wirt2v<br> This situation causes restarting/remounting Drbd2/Virtfs2 on the wirt1v node.<br></div></blockquote><div><br></div><div>Looks like you are missing the meta-attribute "interleave=true" for your filesystem clone:</div><div><br></div><div><a href="http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_clone_options">http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_clone_options</a></div><div><br></div><div>Regards,</div><div>Andreas</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br> And my question is as follow - why the resources restart on the wirt1v node?<br><br> I'm going to use /virtfs2 filesystem to install virtual machines,<br> so in such situation they will restart also.<br><br> Couuld you advise me what to do with this problem?<br><br>The logs and configs are below..<br><br>Thanks in advance,<br>Gienek Nowacki<br>==============================<wbr>========<br>#-----------------------------<wbr>----<br>### result: cat /etc/redhat-release ###<br><br>CentOS Linux release 7.2.1511 (Core)<br><br>#-----------------------------<wbr>----<br>### result: uname -a ###<br><br>Linux wirt1v 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux<br><br>#-----------------------------<wbr>----<br>### result: cat /etc/hosts ###<br><br>127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4<br>172.31.0.23 wirt1<br>172.31.0.24 wirt2<br>1.1.1.1 wirt1v<br>1.1.1.2 wirt2v<br><br>#-----------------------------<wbr>----<br>### result: cat /etc/drbd.conf ###<br><br>include "drbd.d/global_common.conf";<br>include "drbd.d/*.res";<br><br>#-----------------------------<wbr>----<br>### result: cat /etc/drbd.d/global_common.<wbr>conf ###<br><br>common {<br> protocol C;<br> syncer {<br> verify-alg sha1;<br> }<br> net {<br> allow-two-primaries;<br> }<br>}<br><br>#-----------------------------<wbr>----<br>### result: cat /etc/drbd.d/drbd2.res ###<br><br>resource drbd2 {<br> meta-disk internal;<br> device /dev/drbd2;<br> on wirt1v {<br> disk /dev/vg02/drbd2;<br> address <a href="http://1.1.1.1:7782" target="_blank">1.1.1.1:7782</a>;<br> }<br> on wirt2v {<br> disk /dev/vg02/drbd2;<br> address <a href="http://1.1.1.2:7782" target="_blank">1.1.1.2:7782</a>;<br> }<br>}<br>#-----------------------------<wbr>----<br>### result: cat /proc/drbd ###<br><br>version: 8.4.7-1 (api:1/proto:86-101)<br>GIT-hash: 3a6a769340ef93b1ba2792c6461250<wbr>790795db49 build by phil@Build64R7, 2016-01-12 14:29:40<br> 2: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----<br> ns:132180 nr:44 dw:132224 dr:301948 al:33 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br><br>#-----------------------------<wbr>----<br>### result: cat /etc/corosync/corosync.conf ###<br><br>totem {<br> version: 2<br> secauth: off<br> cluster_name: klasterek<br> transport: udpu<br>}<br>nodelist {<br> node {<br> ring0_addr: wirt1v<br> nodeid: 1<br> }<br> node {<br> ring0_addr: wirt2v<br> nodeid: 2<br> }<br>}<br>quorum {<br> provider: corosync_votequorum<br> two_node: 1<br>}<br>logging {<br> to_logfile: yes<br> logfile: /var/log/cluster/corosync.log<br> to_syslog: yes<br>}<br><br>#-----------------------------<wbr>----<br>### result: mnt ###<br><br>#-----------------------------<wbr>----<br>### result: mount | grep virtfs2 ###<br><br>/dev/drbd2 on /virtfs2 type gfs2 (rw,relatime,seclabel)<br><br><br>#-----------------------------<wbr>----<br>### result: pcs config ###<br><br>Cluster Name: klasterek<br>Corosync Nodes:<br> wirt1v wirt2v<br>Pacemaker Nodes:<br> wirt1v wirt2v<br>Resources:<br> Clone: dlm-clone<br> Meta Attrs: clone-max=2 clone-node-max=1<br> Resource: dlm (class=ocf provider=pacemaker type=controld)<br> Operations: start interval=0s timeout=90 (dlm-start-interval-0s)<br> stop interval=0s timeout=100 (dlm-stop-interval-0s)<br> monitor interval=60s (dlm-monitor-interval-60s)<br> Master: Drbd2-clone<br> Meta Attrs: master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true<br> Resource: Drbd2 (class=ocf provider=linbit type=drbd)<br> Attributes: drbd_resource=drbd2<br> Operations: start interval=0s timeout=240 (Drbd2-start-interval-0s)<br> promote interval=0s timeout=90 (Drbd2-promote-interval-0s)<br> demote interval=0s timeout=90 (Drbd2-demote-interval-0s)<br> stop interval=0s timeout=100 (Drbd2-stop-interval-0s)<br> monitor interval=60s (Drbd2-monitor-interval-60s)<br> Clone: Virtfs2-clone<br> Resource: Virtfs2 (class=ocf provider=heartbeat type=Filesystem)<br> Attributes: device=/dev/drbd2 directory=/virtfs2 fstype=gfs2<br> Operations: start interval=0s timeout=60 (Virtfs2-start-interval-0s)<br> stop interval=0s timeout=60 (Virtfs2-stop-interval-0s)<br> monitor interval=20 timeout=40 (Virtfs2-monitor-interval-20)<br>Stonith Devices:<br> Resource: fencing-idrac1 (class=stonith type=fence_idrac)<br> Attributes: pcmk_host_list=wirt1v ipaddr=172.31.0.223 lanplus=on login=root passwd=my1secret2password3<br> Operations: monitor interval=60 (fencing-idrac1-monitor-<wbr>interval-60)<br> Resource: fencing-idrac2 (class=stonith type=fence_idrac)<br> Attributes: pcmk_host_list=wirt2v ipaddr=172.31.0.224 lanplus=on login=root passwd=my1secret2password3<br> Operations: monitor interval=60 (fencing-idrac2-monitor-<wbr>interval-60)<br>Fencing Levels:<br>Location Constraints:<br>Ordering Constraints:<br> start dlm-clone then start Virtfs2-clone (kind:Mandatory) (id:order-dlm-clone-Virtfs2-<wbr>clone-mandatory)<br> promote Drbd2-clone then start Virtfs2-clone (kind:Mandatory) (id:order-Drbd2-clone-Virtfs2-<wbr>clone-mandatory)<br>Colocation Constraints:<br> Virtfs2-clone with Drbd2-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-Virtfs2-clone-<wbr>Drbd2-clone-INFINITY)<br> Virtfs2-clone with dlm-clone (score:INFINITY) (id:colocation-Virtfs2-clone-<wbr>dlm-clone-INFINITY)<br>Resources Defaults:<br> resource-stickiness: 100<br>Operations Defaults:<br> No defaults set<br>Cluster Properties:<br> cluster-infrastructure: corosync<br> cluster-name: klasterek<br> dc-version: 1.1.13-10.el7_2.4-44eb2dd<br> have-watchdog: false<br> no-quorum-policy: ignore<br> stonith-enabled: true<br> symmetric-cluster: true<br><br>#-----------------------------<wbr>----<br>### result: pcs status ###<br><br>Cluster name: klasterek<br>Last updated: Sun Sep 11 16:02:15 2016 Last change: Sun Sep 11 15:01:05 2016 by root via crm_attribute on wirt2v<br>Stack: corosync<br>Current DC: wirt1v (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum<br>2 nodes and 8 resources configured<br>Online: [ wirt1v wirt2v ]<br>Full list of resources:<br> fencing-idrac1 (stonith:fence_idrac): Started wirt1v<br> fencing-idrac2 (stonith:fence_idrac): Started wirt1v<br> Clone Set: dlm-clone [dlm]<br> Started: [ wirt1v wirt2v ]<br> Master/Slave Set: Drbd2-clone [Drbd2]<br> Masters: [ wirt1v wirt2v ]<br> Clone Set: Virtfs2-clone [Virtfs2]<br> Started: [ wirt1v wirt2v ]<br>PCSD Status:<br> wirt1v: Online<br> wirt2v: Online<br>Daemon Status:<br> corosync: active/disabled<br> pacemaker: active/disabled<br> pcsd: active/enabled<br><br>#-----------------------------<wbr>----<br>### result: pcs property ###<br><br>Cluster Properties:<br> cluster-infrastructure: corosync<br> cluster-name: klasterek<br> dc-version: 1.1.13-10.el7_2.4-44eb2dd<br> have-watchdog: false<br> no-quorum-policy: ignore<br> stonith-enabled: true<br> symmetric-cluster: true<br><br>#-----------------------------<wbr>----<br>### result: pcs cluster cib ###<br><br><cib crm_feature_set="3.0.10" validate-with="pacemaker-2.3" epoch="105" num_updates="11" admin_epoch="0" cib-last-written="Sun Sep 11 15:01:05 2016" update-origin="wirt2v" update-client="crm_attribute" update-user="root" have-quorum="1" dc-uuid="1"><br> <configuration><br> <crm_config><br> <cluster_property_set id="cib-bootstrap-options"><br> <nvpair id="cib-bootstrap-options-<wbr>have-watchdog" name="have-watchdog" value="false"/><br> <nvpair id="cib-bootstrap-options-dc-<wbr>version" name="dc-version" value="1.1.13-10.el7_2.4-<wbr>44eb2dd"/><br> <nvpair id="cib-bootstrap-options-<wbr>cluster-infrastructure" name="cluster-infrastructure" value="corosync"/><br> <nvpair id="cib-bootstrap-options-<wbr>cluster-name" name="cluster-name" value="klasterek"/><br> <nvpair id="cib-bootstrap-options-<wbr>stonith-enabled" name="stonith-enabled" value="true"/><br> <nvpair id="cib-bootstrap-options-<wbr>symmetric-cluster" name="symmetric-cluster" value="true"/><br> <nvpair id="cib-bootstrap-options-no-<wbr>quorum-policy" name="no-quorum-policy" value="ignore"/><br> </cluster_property_set><br> </crm_config><br> <nodes><br> <node id="1" uname="wirt1v"><br> <instance_attributes id="nodes-1"/><br> </node><br> <node id="2" uname="wirt2v"><br> <instance_attributes id="nodes-2"/><br> </node><br> </nodes><br> <resources><br> <primitive class="stonith" id="fencing-idrac1" type="fence_idrac"><br> <instance_attributes id="fencing-idrac1-instance_<wbr>attributes"><br> <nvpair id="fencing-idrac1-instance_<wbr>attributes-pcmk_host_list" name="pcmk_host_list" value="wirt1v"/><br> <nvpair id="fencing-idrac1-instance_<wbr>attributes-ipaddr" name="ipaddr" value="172.31.0.223"/><br> <nvpair id="fencing-idrac1-instance_<wbr>attributes-lanplus" name="lanplus" value="on"/><br> <nvpair id="fencing-idrac1-instance_<wbr>attributes-login" name="login" value="root"/><br> <nvpair id="fencing-idrac1-instance_<wbr>attributes-passwd" name="passwd" value="my1secret2password3"/><br> </instance_attributes><br> <operations><br> <op id="fencing-idrac1-monitor-<wbr>interval-60" interval="60" name="monitor"/><br> </operations><br> <meta_attributes id="fencing-idrac1-meta_<wbr>attributes"/><br> </primitive><br> <primitive class="stonith" id="fencing-idrac2" type="fence_idrac"><br> <instance_attributes id="fencing-idrac2-instance_<wbr>attributes"><br> <nvpair id="fencing-idrac2-instance_<wbr>attributes-pcmk_host_list" name="pcmk_host_list" value="wirt2v"/><br> <nvpair id="fencing-idrac2-instance_<wbr>attributes-ipaddr" name="ipaddr" value="172.31.0.224"/><br> <nvpair id="fencing-idrac2-instance_<wbr>attributes-lanplus" name="lanplus" value="on"/><br> <nvpair id="fencing-idrac2-instance_<wbr>attributes-login" name="login" value="root"/><br> <nvpair id="fencing-idrac2-instance_<wbr>attributes-passwd" name="passwd" value="my1secret2password3"/><br> </instance_attributes><br> <operations><br> <op id="fencing-idrac2-monitor-<wbr>interval-60" interval="60" name="monitor"/><br> </operations><br> <meta_attributes id="fencing-idrac2-meta_<wbr>attributes"/><br> </primitive><br> <clone id="dlm-clone"><br> <primitive class="ocf" id="dlm" provider="pacemaker" type="controld"><br> <instance_attributes id="dlm-instance_attributes"/><br> <operations><br> <op id="dlm-start-interval-0s" interval="0s" name="start" timeout="90"/><br> <op id="dlm-stop-interval-0s" interval="0s" name="stop" timeout="100"/><br> <op id="dlm-monitor-interval-60s" interval="60s" name="monitor"/><br> </operations><br> </primitive><br> <meta_attributes id="dlm-clone-meta_attributes"<wbr>><br> <nvpair id="dlm-clone-max" name="clone-max" value="2"/><br> <nvpair id="dlm-clone-node-max" name="clone-node-max" value="1"/><br> </meta_attributes><br> </clone><br> <primitive class="ocf" id="Drbd2" provider="linbit" type="drbd"><br> <instance_attributes id="Drbd2-instance_attributes"<wbr>><br> <nvpair id="Drbd2-instance_attributes-<wbr>drbd_resource" name="drbd_resource" value="drbd2"/><br> </instance_attributes><br> <operations><br> <op id="Drbd2-start-interval-0s" interval="0s" name="start" timeout="240"/><br> <op id="Drbd2-promote-interval-0s" interval="0s" name="promote" timeout="90"/><br> <op id="Drbd2-demote-interval-0s" interval="0s" name="demote" timeout="90"/><br> <op id="Drbd2-stop-interval-0s" interval="0s" name="stop" timeout="100"/><br> <op id="Drbd2-monitor-interval-<wbr>60s" interval="60s" name="monitor"/><br> </operations><br> </primitive><br> <meta_attributes id="Drbd2-clone-meta_<wbr>attributes"><br> <nvpair id="Drbd2-clone-meta_<wbr>attributes-master-max" name="master-max" value="2"/><br> <nvpair id="Drbd2-clone-meta_<wbr>attributes-master-node-max" name="master-node-max" value="1"/><br> <nvpair id="Drbd2-clone-meta_<wbr>attributes-clone-max" name="clone-max" value="2"/><br> <nvpair id="Drbd2-clone-meta_<wbr>attributes-clone-node-max" name="clone-node-max" value="1"/><br> <nvpair id="Drbd2-clone-meta_<wbr>attributes-notify" name="notify" value="true"/><br> </meta_attributes><br> </master><br> <clone id="Virtfs2-clone"><br> <primitive class="ocf" id="Virtfs2" provider="heartbeat" type="Filesystem"><br> <instance_attributes id="Virtfs2-instance_<wbr>attributes"><br> <nvpair id="Virtfs2-instance_<wbr>attributes-device" name="device" value="/dev/drbd2"/><br> <nvpair id="Virtfs2-instance_<wbr>attributes-directory" name="directory" value="/virtfs2"/><br> <nvpair id="Virtfs2-instance_<wbr>attributes-fstype" name="fstype" value="gfs2"/><br> </instance_attributes><br> <operations><br> <op id="Virtfs2-start-interval-0s" interval="0s" name="start" timeout="60"/><br> <op id="Virtfs2-stop-interval-0s" interval="0s" name="stop" timeout="60"/><br> <op id="Virtfs2-monitor-interval-<wbr>20" interval="20" name="monitor" timeout="40"/><br> </operations><br> </primitive><br> <meta_attributes id="Virtfs2-clone-meta_<wbr>attributes"/><br> </clone><br> </resources><br> <constraints><br> <rsc_colocation id="colocation-Virtfs2-clone-<wbr>Drbd2-clone-INFINITY" rsc="Virtfs2-clone" score="INFINITY" with-rsc="Drbd2-clone" with-rsc-role="Master"/><br> <rsc_colocation id="colocation-Virtfs2-clone-<wbr>dlm-clone-INFINITY" rsc="Virtfs2-clone" score="INFINITY" with-rsc="dlm-clone"/><br> <rsc_order first="dlm-clone" first-action="start" id="order-dlm-clone-Virtfs2-<wbr>clone-mandatory" then="Virtfs2-clone" then-action="start"/><br> <rsc_order first="Drbd2-clone" first-action="promote" id="order-Drbd2-clone-Virtfs2-<wbr>clone-mandatory" then="Virtfs2-clone" then-action="start"/><br> </constraints><br> <rsc_defaults><br> <meta_attributes id="rsc_defaults-options"><br> <nvpair id="rsc_defaults-options-<wbr>resource-stickiness" name="resource-stickiness" value="100"/><br> </meta_attributes><br> </rsc_defaults><br> </configuration><br> <status><br> <node_state id="1" uname="wirt1v" in_ccm="true" crmd="online" crm-debug-origin="do_update_<wbr>resource" join="member" expected="member"><br> <transient_attributes id="1"><br> <instance_attributes id="status-1"><br> <nvpair id="status-1-shutdown" name="shutdown" value="0"/><br> <nvpair id="status-1-probe_complete" name="probe_complete" value="true"/><br> <nvpair id="status-1-master-Drbd2" name="master-Drbd2" value="10000"/><br> </instance_attributes><br> </transient_attributes><br> <lrm id="1"><br> <lrm_resources><br> <lrm_resource id="Virtfs2" type="Filesystem" class="ocf" provider="heartbeat"><br> <lrm_rsc_op id="Virtfs2_last_0" operation_key="Virtfs2_start_<wbr>0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="55:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;55:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="96" rc-code="0" op-status="0" interval="0" last-run="1473598870" last-rc-change="1473598870" exec-time="639" queue-time="1" op-digest="<wbr>8dbd904c2115508ebcf3dffe8e7c6d<wbr>82"/><br> <lrm_rsc_op id="Virtfs2_monitor_20000" operation_key="Virtfs2_<wbr>monitor_20000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="56:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;56:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="97" rc-code="0" op-status="0" interval="20000" last-rc-change="1473598871" exec-time="41" queue-time="0" op-digest="<wbr>051271837d1a8eccc0af38fbd8c406<wbr>e4"/><br> </lrm_resource><br> <lrm_resource id="fencing-idrac1" type="fence_idrac" class="stonith"><br> <lrm_rsc_op id="fencing-idrac1_last_0" operation_key="fencing-idrac1_<wbr>start_0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="9:58:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;9:58:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="58" rc-code="0" op-status="0" interval="0" last-run="1473588685" last-rc-change="1473588685" exec-time="1045" queue-time="0" op-digest="<wbr>23a748cdf02f6f0fd03ac9823fc9bd<wbr>52" op-secure-params=" passwd " op-secure-digest="<wbr>2a5376722a6d891302b4e811e4de5c<wbr>5a"/><br> <lrm_rsc_op id="fencing-idrac1_monitor_<wbr>60000" operation_key="fencing-idrac1_<wbr>monitor_60000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="11:59:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;11:59:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="59" rc-code="0" op-status="0" interval="60000" last-rc-change="1473588687" exec-time="84" queue-time="0" op-digest="<wbr>592f6bfb8f36e6645a6221de49f6f3<wbr>b3" op-secure-params=" passwd " op-secure-digest="<wbr>2a5376722a6d891302b4e811e4de5c<wbr>5a"/><br> </lrm_resource><br> <lrm_resource id="fencing-idrac2" type="fence_idrac" class="stonith"><br> <lrm_rsc_op id="fencing-idrac2_last_0" operation_key="fencing-idrac2_<wbr>start_0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="14:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;14:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="61" rc-code="0" op-status="0" interval="0" last-run="1473590528" last-rc-change="1473590528" exec-time="80" queue-time="0" op-digest="<wbr>268b7ef79bdf7a09609aa321d3d18a<wbr>61" op-secure-params=" passwd " op-secure-digest="<wbr>f22e287dc4906f866a82eac0ab75d2<wbr>17"/><br> <lrm_rsc_op id="fencing-idrac2_monitor_<wbr>60000" operation_key="fencing-idrac2_<wbr>monitor_60000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="15:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;15:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="62" rc-code="0" op-status="0" interval="60000" last-rc-change="1473590529" exec-time="75" queue-time="1" op-digest="<wbr>40430ed0cd93e10fcba03a5e867b2a<wbr>f3" op-secure-params=" passwd " op-secure-digest="<wbr>f22e287dc4906f866a82eac0ab75d2<wbr>17"/><br> </lrm_resource><br> <lrm_resource id="dlm" type="controld" class="ocf" provider="pacemaker"><br> <lrm_rsc_op id="dlm_last_0" operation_key="dlm_start_0" operation="start" crm-debug-origin="build_<wbr>active_RAs" crm_feature_set="3.0.10" transition-key="14:15:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" transition-magic="0:0;14:15:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" on_node="wirt1v" call-id="25" rc-code="0" op-status="0" interval="0" last-run="1473544487" last-rc-change="1473544487" exec-time="1082" queue-time="0" op-digest="<wbr>f2317cad3d54cec5d7d7aa7d0bf35c<wbr>f8"/><br> <lrm_rsc_op id="dlm_monitor_60000" operation_key="dlm_monitor_<wbr>60000" operation="monitor" crm-debug-origin="build_<wbr>active_RAs" crm_feature_set="3.0.10" transition-key="8:16:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" transition-magic="0:0;8:16:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" on_node="wirt1v" call-id="28" rc-code="0" op-status="0" interval="60000" last-rc-change="1473544489" exec-time="38" queue-time="0" op-digest="<wbr>4811cef7f7f94e3a35a70be7916cb2<wbr>fd"/><br> </lrm_resource><br> <lrm_resource id="Drbd2" type="drbd" class="ocf" provider="linbit"><br> <lrm_rsc_op id="Drbd2_last_0" operation_key="Drbd2_promote_<wbr>0" operation="promote" crm-debug-origin="build_<wbr>active_RAs" crm_feature_set="3.0.10" transition-key="17:16:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" transition-magic="0:0;17:16:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" on_node="wirt1v" call-id="30" rc-code="0" op-status="0" interval="0" last-run="1473544489" last-rc-change="1473544489" exec-time="58" queue-time="0" op-digest="<wbr>d0c8a735862843030d8426a5218ceb<wbr>92"/><br> </lrm_resource><br> </lrm_resources><br> </lrm><br> </node_state><br> <node_state id="2" uname="wirt2v" in_ccm="true" crmd="online" crm-debug-origin="do_update_<wbr>resource" join="member" expected="member"><br> <transient_attributes id="2"><br> <instance_attributes id="status-2"><br> <nvpair id="status-2-shutdown" name="shutdown" value="0"/><br> <nvpair id="status-2-probe_complete" name="probe_complete" value="true"/><br> <nvpair id="status-2-master-Drbd2" name="master-Drbd2" value="10000"/><br> </instance_attributes><br> </transient_attributes><br> <lrm id="2"><br> <lrm_resources><br> <lrm_resource id="fencing-idrac1" type="fence_idrac" class="stonith"><br> <lrm_rsc_op id="fencing-idrac1_last_0" operation_key="fencing-idrac1_<wbr>monitor_0" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="7:1:7:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:7;7:1:7:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="5" rc-code="7" op-status="0" interval="0" last-run="1473544754" last-rc-change="1473544754" exec-time="1" queue-time="0" op-digest="<wbr>23a748cdf02f6f0fd03ac9823fc9bd<wbr>52" op-secure-params=" passwd " op-secure-digest="<wbr>2a5376722a6d891302b4e811e4de5c<wbr>5a"/><br> </lrm_resource><br> <lrm_resource id="fencing-idrac2" type="fence_idrac" class="stonith"><br> <lrm_rsc_op id="fencing-idrac2_last_0" operation_key="fencing-idrac2_<wbr>stop_0" operation="stop" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="13:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;13:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="55" rc-code="0" op-status="0" interval="0" last-run="1473590528" last-rc-change="1473590528" exec-time="0" queue-time="0" op-digest="<wbr>268b7ef79bdf7a09609aa321d3d18a<wbr>61" op-secure-params=" passwd " op-secure-digest="<wbr>f22e287dc4906f866a82eac0ab75d2<wbr>17"/><br> <lrm_rsc_op id="fencing-idrac2_monitor_<wbr>60000" operation_key="fencing-idrac2_<wbr>monitor_60000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="13:59:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;13:59:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="53" rc-code="0" op-status="0" interval="60000" last-rc-change="1473588689" exec-time="63" queue-time="0" op-digest="<wbr>40430ed0cd93e10fcba03a5e867b2a<wbr>f3" op-secure-params=" passwd " op-secure-digest="<wbr>f22e287dc4906f866a82eac0ab75d2<wbr>17"/><br> </lrm_resource><br> <lrm_resource id="dlm" type="controld" class="ocf" provider="pacemaker"><br> <lrm_rsc_op id="dlm_last_0" operation_key="dlm_start_0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="15:83:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;15:83:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="101" rc-code="0" op-status="0" interval="0" last-run="1473598865" last-rc-change="1473598865" exec-time="1116" queue-time="0" op-digest="<wbr>f2317cad3d54cec5d7d7aa7d0bf35c<wbr>f8"/><br> <lrm_rsc_op id="dlm_monitor_60000" operation_key="dlm_monitor_<wbr>60000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="16:84:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;16:84:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="104" rc-code="0" op-status="0" interval="60000" last-rc-change="1473598870" exec-time="47" queue-time="0" op-digest="<wbr>4811cef7f7f94e3a35a70be7916cb2<wbr>fd"/><br> </lrm_resource><br> <lrm_resource id="Drbd2" type="drbd" class="ocf" provider="linbit"><br> <lrm_rsc_op id="Drbd2_last_0" operation_key="Drbd2_promote_<wbr>0" operation="promote" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="27:84:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;27:84:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="106" rc-code="0" op-status="0" interval="0" last-run="1473598870" last-rc-change="1473598870" exec-time="69" queue-time="0" op-digest="<wbr>d0c8a735862843030d8426a5218ceb<wbr>92"/><br> </lrm_resource><br> <lrm_resource id="Virtfs2" type="Filesystem" class="ocf" provider="heartbeat"><br> <lrm_rsc_op id="Virtfs2_last_0" operation_key="Virtfs2_start_<wbr>0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="53:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;53:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="108" rc-code="0" op-status="0" interval="0" last-run="1473598870" last-rc-change="1473598870" exec-time="859" queue-time="0" op-digest="<wbr>8dbd904c2115508ebcf3dffe8e7c6d<wbr>82"/><br> <lrm_rsc_op id="Virtfs2_monitor_20000" operation_key="Virtfs2_<wbr>monitor_20000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="54:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;54:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="109" rc-code="0" op-status="0" interval="20000" last-rc-change="1473598871" exec-time="50" queue-time="0" op-digest="<wbr>051271837d1a8eccc0af38fbd8c406<wbr>e4"/><br> </lrm_resource><br> </lrm_resources><br> </lrm><br> </node_state><br> </status><br></cib><br># ==============================<wbr>==============================<wbr>=========<br># wirt1v-during_pcs-cluster-<wbr>standby-wirt2v.log<br>#<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br>Sep 11 14:58:06 wirt1v pengine[31950]: notice: On loss of CCM Quorum: Ignore<br>Sep 11 14:58:06 wirt1v pengine[31950]: notice: Stop dlm:1#011(wirt2v)<br>Sep 11 14:58:06 wirt1v pengine[31950]: notice: Demote Drbd2:1#011(Master -> Stopped wirt2v)<br>Sep 11 14:58:06 wirt1v pengine[31950]: notice: Stop Virtfs2:1#011(wirt2v)<br>Sep 11 14:58:06 wirt1v pengine[31950]: notice: Calculated Transition 81: /var/lib/pacemaker/pengine/pe-<wbr>input-2852.bz2<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 71: notify Drbd2_pre_notify_demote_0 on wirt1v (local)<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 73: notify Drbd2_pre_notify_demote_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 54: stop Virtfs2_stop_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=86, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: recover generation 2 done<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 17: stop dlm_stop_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 26: demote Drbd2_demote_0 on wirt2v<br>Sep 11 14:58:06 wirt1v kernel: block drbd2: peer( Primary -> Secondary )<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 72: notify Drbd2_post_notify_demote_0 on wirt1v (local)<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 74: notify Drbd2_post_notify_demote_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=87, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 66: notify Drbd2_pre_notify_stop_0 on wirt1v (local)<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 68: notify Drbd2_pre_notify_stop_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=88, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 27: stop Drbd2_stop_0 on wirt2v<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> DUnknown )<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: ack_receiver terminated<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: Terminating drbd_a_drbd2<br>Sep 11 14:58:06 wirt1v kernel: block drbd2: new current UUID 12027A6DEC39CCB7:<wbr>9A297D737BE3FBC7:<wbr>214339307E5385FF:<wbr>214239307E5385FF<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: Connection closed<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: conn( TearDown -> Unconnected )<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: receiver terminated<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: Restarting receiver thread<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: receiver (re)started<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: conn( Unconnected -> WFConnection )<br>Sep 11 14:58:06 wirt1v crmd[31951]: warning: No match for shutdown action on 2<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Transition aborted by deletion of nvpair[@id='status-2-master-<wbr>Drbd2']: Transient attribute change (cib=0.104.3, source=abort_unless_down:333, path=/cib/status/node_state[@<wbr>id='2']/transient_attributes[@<wbr>id='2']/instance_attributes[@<wbr>id='status-2']/nvpair[@id='<wbr>status-2-master-Drbd2'], 0)<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Initiating action 67: notify Drbd2_post_notify_stop_0 on wirt1v (local)<br>Sep 11 14:58:06 wirt1v crmd[31951]: notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=89, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:08 wirt1v crmd[31951]: notice: Transition 81 (Complete=28, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2852.bz2): Complete<br>Sep 11 14:58:08 wirt1v pengine[31950]: notice: On loss of CCM Quorum: Ignore<br>Sep 11 14:58:08 wirt1v pengine[31950]: notice: Calculated Transition 82: /var/lib/pacemaker/pengine/pe-<wbr>input-2853.bz2<br>Sep 11 14:58:08 wirt1v crmd[31951]: notice: Transition 82 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2853.bz2): Complete<br>Sep 11 14:58:08 wirt1v crmd[31951]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br><br># ==============================<wbr>==============================<wbr>=========<br># wirt1v-during_pcs-cluster-<wbr>unstandby-wirt2v.log<br>#<br>Sep 11 15:01:01 wirt1v systemd: Started Session 59 of user root.<br>Sep 11 15:01:01 wirt1v systemd: Starting Session 59 of user root.<br>Sep 11 15:01:05 wirt1v crmd[31951]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br>Sep 11 15:01:05 wirt1v pengine[31950]: notice: On loss of CCM Quorum: Ignore<br>Sep 11 15:01:05 wirt1v pengine[31950]: notice: Start dlm:1#011(wirt2v)<br>Sep 11 15:01:05 wirt1v pengine[31950]: notice: Start Drbd2:1#011(wirt2v)<br>Sep 11 15:01:05 wirt1v pengine[31950]: notice: Restart Virtfs2:0#011(Started wirt1v)<br>Sep 11 15:01:05 wirt1v pengine[31950]: notice: Calculated Transition 83: /var/lib/pacemaker/pengine/pe-<wbr>input-2854.bz2<br>Sep 11 15:01:05 wirt1v crmd[31951]: notice: Initiating action 15: start dlm_start_0 on wirt2v<br>Sep 11 15:01:05 wirt1v crmd[31951]: notice: Initiating action 61: notify Drbd2_pre_notify_start_0 on wirt1v (local)<br>Sep 11 15:01:05 wirt1v crmd[31951]: notice: Initiating action 51: stop Virtfs2_stop_0 on wirt1v (local)<br>Sep 11 15:01:06 wirt1v Filesystem(Virtfs2)[42311]: INFO: Running stop for /dev/drbd2 on /virtfs2<br>Sep 11 15:01:06 wirt1v crmd[31951]: notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=90, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:06 wirt1v crmd[31951]: notice: Initiating action 25: start Drbd2_start_0 on wirt2v<br>Sep 11 15:01:06 wirt1v Filesystem(Virtfs2)[42311]: INFO: Trying to unmount /virtfs2<br>Sep 11 15:01:06 wirt1v Filesystem(Virtfs2)[42311]: ERROR: Couldn't unmount /virtfs2; trying cleanup with TERM<br>Sep 11 15:01:06 wirt1v crmd[31951]: notice: Transition aborted by status-2-master-Drbd2, master-Drbd2=1000: Transient attribute change (create cib=0.105.1, source=abort_unless_down:319, path=/cib/status/node_state[@<wbr>id='2']/transient_attributes[@<wbr>id='2']/instance_attributes[@<wbr>id='status-2'], 0)<br>Sep 11 15:01:06 wirt1v Filesystem(Virtfs2)[42311]: INFO: sending signal TERM to: root 39937 39934 0 14:49 pts/0 Ss+ 0:00 -bash<br>Sep 11 15:01:06 wirt1v crmd[31951]: notice: Initiating action 62: notify Drbd2_post_notify_start_0 on wirt1v (local)<br>Sep 11 15:01:06 wirt1v crmd[31951]: notice: Initiating action 63: notify Drbd2_post_notify_start_0 on wirt2v<br>Sep 11 15:01:06 wirt1v crmd[31951]: notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=93, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:06 wirt1v kernel: drbd drbd2: Handshake successful: Agreed network protocol version 101<br>Sep 11 15:01:06 wirt1v kernel: drbd drbd2: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.<br>Sep 11 15:01:06 wirt1v kernel: drbd drbd2: conn( WFConnection -> WFReportParams )<br>Sep 11 15:01:06 wirt1v kernel: drbd drbd2: Starting ack_recv thread (from drbd_r_drbd2 [32139])<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: drbd_sync_handshake:<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: self 12027A6DEC39CCB7:<wbr>9A297D737BE3FBC7:<wbr>214339307E5385FF:<wbr>214239307E5385FF bits:0 flags:0<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: peer 9A297D737BE3FBC6:<wbr>0000000000000000:<wbr>214339307E5385FE:<wbr>214239307E5385FF bits:0 flags:0<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: uuid_compare()=1 by rule 70<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: helper command: /sbin/drbdadm before-resync-source minor-2<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: helper command: /sbin/drbdadm before-resync-source minor-2 exit code 0 (0x0)<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: Began resync as SyncSource (will sync 0 KB [0 bits set]).<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: updated sync UUID 12027A6DEC39CCB7:<wbr>9A2A7D737BE3FBC7:<wbr>9A297D737BE3FBC7:<wbr>214339307E5385FF<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: Resync done (total 1 sec; paused 0 sec; 0 K/sec)<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: updated UUIDs 12027A6DEC39CCB7:<wbr>0000000000000000:<wbr>9A2A7D737BE3FBC7:<wbr>9A297D737BE3FBC7<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<br>Sep 11 15:01:07 wirt1v Filesystem(Virtfs2)[42311]: ERROR: Couldn't unmount /virtfs2; trying cleanup with TERM<br>Sep 11 15:01:07 wirt1v Filesystem(Virtfs2)[42311]: INFO: sending signal TERM to: root 39937 39934 0 14:49 pts/0 Ss+ 0:00 -bash<br>Sep 11 15:01:08 wirt1v Filesystem(Virtfs2)[42311]: ERROR: Couldn't unmount /virtfs2; trying cleanup with TERM<br>Sep 11 15:01:08 wirt1v Filesystem(Virtfs2)[42311]: INFO: sending signal TERM to: root 39937 39934 0 14:49 pts/0 Ss+ 0:00 -bash<br>Sep 11 15:01:09 wirt1v Filesystem(Virtfs2)[42311]: ERROR: Couldn't unmount /virtfs2; trying cleanup with KILL<br>Sep 11 15:01:09 wirt1v Filesystem(Virtfs2)[42311]: INFO: sending signal KILL to: root 39937 39934 0 14:49 pts/0 Ss+ 0:00 -bash<br>Sep 11 15:01:09 wirt1v systemd-logind: Removed session 58.<br>Sep 11 15:01:10 wirt1v Filesystem(Virtfs2)[42311]: INFO: unmounted /virtfs2 successfully<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ umount: /virtfs2: target is busy. ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ (In some cases useful info about processes that use ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ the device is found by lsof(8) or fuser(1)) ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ ocf-exit-reason:Couldn't unmount /virtfs2; trying cleanup with TERM ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ umount: /virtfs2: target is busy. ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ (In some cases useful info about processes that use ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ the device is found by lsof(8) or fuser(1)) ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ ocf-exit-reason:Couldn't unmount /virtfs2; trying cleanup with TERM ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ umount: /virtfs2: target is busy. ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ (In some cases useful info about processes that use ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ the device is found by lsof(8) or fuser(1)) ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ ocf-exit-reason:Couldn't unmount /virtfs2; trying cleanup with TERM ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ umount: /virtfs2: target is busy. ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ (In some cases useful info about processes that use ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ the device is found by lsof(8) or fuser(1)) ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]: notice: Virtfs2_stop_0:42311:stderr [ ocf-exit-reason:Couldn't unmount /virtfs2; trying cleanup with KILL ]<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Operation Virtfs2_stop_0: ok (node=wirt1v, call=92, rc=0, cib-update=178, confirmed=true)<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Transition 83 (Complete=18, Pending=0, Fired=0, Skipped=3, Incomplete=5, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2854.bz2): Stopped<br>Sep 11 15:01:10 wirt1v pengine[31950]: notice: On loss of CCM Quorum: Ignore<br>Sep 11 15:01:10 wirt1v pengine[31950]: notice: Promote Drbd2:1#011(Slave -> Master wirt2v)<br>Sep 11 15:01:10 wirt1v pengine[31950]: notice: Start Virtfs2:0#011(wirt1v)<br>Sep 11 15:01:10 wirt1v pengine[31950]: notice: Start Virtfs2:1#011(wirt2v)<br>Sep 11 15:01:10 wirt1v pengine[31950]: notice: Calculated Transition 84: /var/lib/pacemaker/pengine/pe-<wbr>input-2855.bz2<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Initiating action 16: monitor dlm_monitor_60000 on wirt2v<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Initiating action 69: notify Drbd2_pre_notify_promote_0 on wirt1v (local)<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Initiating action 71: notify Drbd2_pre_notify_promote_0 on wirt2v<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=94, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Initiating action 27: promote Drbd2_promote_0 on wirt2v<br>Sep 11 15:01:10 wirt1v kernel: block drbd2: peer( Secondary -> Primary )<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Initiating action 70: notify Drbd2_post_notify_promote_0 on wirt1v (local)<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Initiating action 72: notify Drbd2_post_notify_promote_0 on wirt2v<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Transition aborted by status-2-master-Drbd2, master-Drbd2=10000: Transient attribute change (modify cib=0.105.7, source=abort_unless_down:319, path=/cib/status/node_state[@<wbr>id='2']/transient_attributes[@<wbr>id='2']/instance_attributes[@<wbr>id='status-2']/nvpair[@id='<wbr>status-2-master-Drbd2'], 0)<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=95, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Transition 84 (Complete=13, Pending=0, Fired=0, Skipped=2, Incomplete=5, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2855.bz2): Stopped<br>Sep 11 15:01:10 wirt1v pengine[31950]: notice: On loss of CCM Quorum: Ignore<br>Sep 11 15:01:10 wirt1v pengine[31950]: notice: Start Virtfs2:0#011(wirt2v)<br>Sep 11 15:01:10 wirt1v pengine[31950]: notice: Start Virtfs2:1#011(wirt1v)<br>Sep 11 15:01:10 wirt1v pengine[31950]: notice: Calculated Transition 85: /var/lib/pacemaker/pengine/pe-<wbr>input-2856.bz2<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Initiating action 53: start Virtfs2_start_0 on wirt2v<br>Sep 11 15:01:10 wirt1v crmd[31951]: notice: Initiating action 55: start Virtfs2:1_start_0 on wirt1v (local)<br>Sep 11 15:01:10 wirt1v Filesystem(Virtfs2)[42615]: INFO: Running start for /dev/drbd2 on /virtfs2<br>Sep 11 15:01:10 wirt1v kernel: GFS2: fsid=klasterek:drbd2: Trying to join cluster "lock_dlm", "klasterek:drbd2"<br>Sep 11 15:01:10 wirt1v kernel: dlm: Using TCP for communications<br>Sep 11 15:01:10 wirt1v kernel: dlm: connecting to 2<br>Sep 11 15:01:10 wirt1v kernel: dlm: got connection from 2<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2: first mounter control generation 0<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2: Joined cluster. Now mounting FS...<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=0, already locked for use<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=0: Looking at journal...<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=0: Done<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=1: Trying to acquire journal lock...<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=1: Looking at journal...<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=1: Done<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: first mount done, others may mount<br>Sep 11 15:01:11 wirt1v crmd[31951]: notice: Operation Virtfs2_start_0: ok (node=wirt1v, call=96, rc=0, cib-update=181, confirmed=true)<br>Sep 11 15:01:11 wirt1v crmd[31951]: notice: Initiating action 56: monitor Virtfs2:1_monitor_20000 on wirt1v (local)<br>Sep 11 15:01:11 wirt1v crmd[31951]: notice: Initiating action 54: monitor Virtfs2_monitor_20000 on wirt2v<br>Sep 11 15:01:11 wirt1v crmd[31951]: notice: Transition 85 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2856.bz2): Complete<br>Sep 11 15:01:11 wirt1v crmd[31951]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>Sep 11 15:01:18 wirt1v systemd-logind: New session 60 of user root.<br>Sep 11 15:01:18 wirt1v systemd: Started Session 60 of user root.<br>Sep 11 15:01:18 wirt1v systemd: Starting Session 60 of user root.<br><br># ==============================<wbr>==============================<wbr>=========<br># wirt2v-during_pcs-cluster-<wbr>standby-wirt2v.log<br>#<br>Sep 11 14:58:06 wirt2v crmd[27038]: notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=92, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt2v Filesystem(Virtfs2)[28208]: INFO: Running stop for /dev/drbd2 on /virtfs2<br>Sep 11 14:58:06 wirt2v Filesystem(Virtfs2)[28208]: INFO: Trying to unmount /virtfs2<br>Sep 11 14:58:06 wirt2v Filesystem(Virtfs2)[28208]: INFO: unmounted /virtfs2 successfully<br>Sep 11 14:58:06 wirt2v crmd[27038]: notice: Operation Virtfs2_stop_0: ok (node=wirt2v, call=94, rc=0, cib-update=58, confirmed=true)<br>Sep 11 14:58:06 wirt2v kernel: dlm: closing connection to node 2<br>Sep 11 14:58:06 wirt2v kernel: dlm: closing connection to node 1<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: role( Primary -> Secondary )<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: bitmap WRITE of 0 pages took 0 jiffies<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>Sep 11 14:58:06 wirt2v systemd-udevd: error: /dev/drbd2: Wrong medium type<br>Sep 11 14:58:06 wirt2v crmd[27038]: error: pcmkRegisterNode: Triggered assert at xml.c:594 : node->type == XML_ELEMENT_NODE<br>Sep 11 14:58:06 wirt2v crmd[27038]: notice: Operation Drbd2_demote_0: ok (node=wirt2v, call=97, rc=0, cib-update=59, confirmed=true)<br>Sep 11 14:58:06 wirt2v systemd-udevd: error: /dev/drbd2: Wrong medium type<br>Sep 11 14:58:06 wirt2v crmd[27038]: notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=98, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt2v crmd[27038]: notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=99, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: peer( Primary -> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown )<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: ack_receiver terminated<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: Terminating drbd_a_drbd2<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: Connection closed<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: conn( Disconnecting -> StandAlone )<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: receiver terminated<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: Terminating drbd_r_drbd2<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: disk( UpToDate -> Failed )<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: bitmap WRITE of 0 pages took 0 jiffies<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: disk( Failed -> Diskless )<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: Terminating drbd_w_drbd2<br>Sep 11 14:58:06 wirt2v crmd[27038]: notice: Operation Drbd2_stop_0: ok (node=wirt2v, call=100, rc=0, cib-update=60, confirmed=true)<br>Sep 11 14:58:08 wirt2v crmd[27038]: notice: Operation dlm_stop_0: ok (node=wirt2v, call=96, rc=0, cib-update=61, confirmed=true)<br><br># ==============================<wbr>==============================<wbr>=========<br># wirt2v-during_pcs-cluster-<wbr>unstandby-wirt2v.log<br>#<br>Sep 11 15:01:01 wirt2v systemd: Started Session 51 of user root.<br>Sep 11 15:01:01 wirt2v systemd: Starting Session 51 of user root.<br>Sep 11 15:01:06 wirt2v dlm_controld[28577]: 62718 dlm_controld 4.0.2 started<br>Sep 11 15:01:06 wirt2v systemd-udevd: error: /dev/drbd2: Wrong medium type<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Starting worker thread (from drbdsetup-84 [28610])<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: disk( Diskless -> Attaching )<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Method to ensure write ordering: flush<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: max BIO size = 262144<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: drbd_bm_resize called with capacity == 104854328<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: resync bitmap: bits=<a href="tel:13106791" value="+13106791" target="_blank">13106791</a> words=204794 pages=400<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: size = 50 GB (52427164 KB)<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: recounting of set bits took additional 1 jiffies<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: disk( Attaching -> UpToDate )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: attached to UUIDs 9A297D737BE3FBC6:<wbr>0000000000000000:<wbr>214339307E5385FE:<wbr>214239307E5385FF<br>Sep 11 15:01:06 wirt2v systemd-udevd: error: /dev/drbd2: Wrong medium type<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: conn( StandAlone -> Unconnected )<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Starting receiver thread (from drbd_w_drbd2 [28612])<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: receiver (re)started<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: conn( Unconnected -> WFConnection )<br>Sep 11 15:01:06 wirt2v crmd[27038]: error: pcmkRegisterNode: Triggered assert at xml.c:594 : node->type == XML_ELEMENT_NODE<br>Sep 11 15:01:06 wirt2v crmd[27038]: notice: Operation Drbd2_start_0: ok (node=wirt2v, call=102, rc=0, cib-update=62, confirmed=true)<br>Sep 11 15:01:06 wirt2v crmd[27038]: notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=103, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Handshake successful: Agreed network protocol version 101<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: conn( WFConnection -> WFReportParams )<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Starting ack_recv thread (from drbd_r_drbd2 [28622])<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: drbd_sync_handshake:<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: self 9A297D737BE3FBC6:<wbr>0000000000000000:<wbr>214339307E5385FE:<wbr>214239307E5385FF bits:0 flags:0<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: peer 12027A6DEC39CCB7:<wbr>9A297D737BE3FBC7:<wbr>214339307E5385FF:<wbr>214239307E5385FF bits:0 flags:0<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: uuid_compare()=-1 by rule 50<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: updated sync uuid 9A2A7D737BE3FBC6:<wbr>0000000000000000:<wbr>214339307E5385FE:<wbr>214239307E5385FF<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: Began resync as SyncTarget (will sync 0 KB [0 bits set]).<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: Resync done (total 1 sec; paused 0 sec; 0 K/sec)<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: updated UUIDs 12027A6DEC39CCB6:<wbr>0000000000000000:<wbr>9A2A7D737BE3FBC6:<wbr>9A297D737BE3FBC7<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: helper command: /sbin/drbdadm after-resync-target minor-2<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: helper command: /sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)<br>Sep 11 15:01:07 wirt2v crmd[27038]: notice: Operation dlm_start_0: ok (node=wirt2v, call=101, rc=0, cib-update=63, confirmed=true)<br>Sep 11 15:01:10 wirt2v crmd[27038]: notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=105, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:10 wirt2v kernel: block drbd2: role( Secondary -> Primary )<br>Sep 11 15:01:10 wirt2v crmd[27038]: error: pcmkRegisterNode: Triggered assert at xml.c:594 : node->type == XML_ELEMENT_NODE<br>Sep 11 15:01:10 wirt2v crmd[27038]: notice: Operation Drbd2_promote_0: ok (node=wirt2v, call=106, rc=0, cib-update=65, confirmed=true)<br>Sep 11 15:01:10 wirt2v crmd[27038]: notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=107, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:10 wirt2v Filesystem(Virtfs2)[28772]: INFO: Running start for /dev/drbd2 on /virtfs2<br>Sep 11 15:01:10 wirt2v kernel: GFS2: fsid=klasterek:drbd2: Trying to join cluster "lock_dlm", "klasterek:drbd2"<br>Sep 11 15:01:10 wirt2v kernel: dlm: Using TCP for communications<br>Sep 11 15:01:10 wirt2v kernel: dlm: connecting to 1<br>Sep 11 15:01:10 wirt2v kernel: dlm: got connection from 1<br>Sep 11 15:01:11 wirt2v kernel: GFS2: fsid=klasterek:drbd2: Joined cluster. Now mounting FS...<br>Sep 11 15:01:11 wirt2v kernel: GFS2: fsid=klasterek:drbd2.1: jid=1, already locked for use<br>Sep 11 15:01:11 wirt2v kernel: GFS2: fsid=klasterek:drbd2.1: jid=1: Looking at journal...<br>Sep 11 15:01:11 wirt2v kernel: GFS2: fsid=klasterek:drbd2.1: jid=1: Done<br>Sep 11 15:01:11 wirt2v crmd[27038]: notice: Operation Virtfs2_start_0: ok (node=wirt2v, call=108, rc=0, cib-update=66, confirmed=true)<br><br>==============================<wbr>==============================<wbr>== <wbr> <wbr> <br><br><br></div>
<br>______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br></div></div>