<div dir="ltr">Hi,<div class="gmail_extra"><br><div class="gmail_quote">On Sun, Sep 11, 2016 at 6:28 PM, Gienek Nowacki <span dir="ltr"><<a href="mailto:nowackig@gmail.com" target="_blank">nowackig@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<br><br>I have a problem with my cluster.<br>There are two nodes: wirt1v and wirt2v with pacemaker, corosync, dlm,<br>drbd (/dev/drbd2) and filesystem mounted as /virtfs2 with gfs2.<br>Each node contains LVM partition on wihich is DRBD.<br><br>The situation is as follow:<br><br>pcs cluster standby wirt2v<br> ...all is OK, there is possible to use /virtfs2 on wirt1v node<br><br>pcs cluster unstandby wirt2v<br> This situation causes restarting/remounting Drbd2/Virtfs2 on the wirt1v node.<br></div></blockquote><div><br></div><div>Looks like you are missing the meta-attribute "interleave=true" for your filesystem clone:</div><div><br></div><div><a href="http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_clone_options">http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_clone_options</a></div><div><br></div><div>Regards,</div><div>Andreas</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br> And my question is as follow - why the resources restart on the wirt1v node?<br><br> I'm going to use /virtfs2 filesystem to install virtual machines,<br> so in such situation they will restart also.<br><br> Couuld you advise me what to do with this problem?<br><br>The logs and configs are below..<br><br>Thanks in advance,<br>Gienek Nowacki<br>==============================<wbr>========<br>#-----------------------------<wbr>----<br>### result:  cat /etc/redhat-release  ###<br><br>CentOS Linux release 7.2.1511 (Core)<br><br>#-----------------------------<wbr>----<br>### result:  uname -a  ###<br><br>Linux wirt1v 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux<br><br>#-----------------------------<wbr>----<br>### result:  cat /etc/hosts  ###<br><br>127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4<br>172.31.0.23     wirt1<br>172.31.0.24     wirt2<br>1.1.1.1         wirt1v<br>1.1.1.2         wirt2v<br><br>#-----------------------------<wbr>----<br>### result:  cat /etc/drbd.conf  ###<br><br>include "drbd.d/global_common.conf";<br>include "drbd.d/*.res";<br><br>#-----------------------------<wbr>----<br>### result:  cat /etc/drbd.d/global_common.<wbr>conf  ###<br><br>common {<br>        protocol C;<br>        syncer {<br>                verify-alg sha1;<br>        }<br>        net {<br>                allow-two-primaries;<br>        }<br>}<br><br>#-----------------------------<wbr>----<br>### result:  cat /etc/drbd.d/drbd2.res  ###<br><br>resource drbd2 {<br>        meta-disk internal;<br>        device /dev/drbd2;<br>        on wirt1v {<br>                disk /dev/vg02/drbd2;<br>                address <a href="http://1.1.1.1:7782" target="_blank">1.1.1.1:7782</a>;<br>        }<br>        on wirt2v {<br>                disk /dev/vg02/drbd2;<br>                address <a href="http://1.1.1.2:7782" target="_blank">1.1.1.2:7782</a>;<br>        }<br>}<br>#-----------------------------<wbr>----<br>### result:  cat /proc/drbd  ###<br><br>version: 8.4.7-1 (api:1/proto:86-101)<br>GIT-hash: 3a6a769340ef93b1ba2792c6461250<wbr>790795db49 build by phil@Build64R7, 2016-01-12 14:29:40<br> 2: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----<br>    ns:132180 nr:44 dw:132224 dr:301948 al:33 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0<br><br>#-----------------------------<wbr>----<br>### result:  cat /etc/corosync/corosync.conf  ###<br><br>totem {<br>    version: 2<br>    secauth: off<br>    cluster_name: klasterek<br>    transport: udpu<br>}<br>nodelist {<br>    node {<br>        ring0_addr: wirt1v<br>        nodeid: 1<br>    }<br>    node {<br>        ring0_addr: wirt2v<br>        nodeid: 2<br>    }<br>}<br>quorum {<br>    provider: corosync_votequorum<br>    two_node: 1<br>}<br>logging {<br>    to_logfile: yes<br>    logfile: /var/log/cluster/corosync.log<br>    to_syslog: yes<br>}<br><br>#-----------------------------<wbr>----<br>### result:  mnt  ###<br><br>#-----------------------------<wbr>----<br>### result:  mount | grep virtfs2  ###<br><br>/dev/drbd2 on /virtfs2 type gfs2 (rw,relatime,seclabel)<br><br><br>#-----------------------------<wbr>----<br>### result:  pcs config  ###<br><br>Cluster Name: klasterek<br>Corosync Nodes:<br> wirt1v wirt2v<br>Pacemaker Nodes:<br> wirt1v wirt2v<br>Resources:<br> Clone: dlm-clone<br>  Meta Attrs: clone-max=2 clone-node-max=1<br>  Resource: dlm (class=ocf provider=pacemaker type=controld)<br>   Operations: start interval=0s timeout=90 (dlm-start-interval-0s)<br>               stop interval=0s timeout=100 (dlm-stop-interval-0s)<br>               monitor interval=60s (dlm-monitor-interval-60s)<br> Master: Drbd2-clone<br>  Meta Attrs: master-max=2 master-node-max=1 clone-max=2 clone-node-max=1 notify=true<br>  Resource: Drbd2 (class=ocf provider=linbit type=drbd)<br>   Attributes: drbd_resource=drbd2<br>   Operations: start interval=0s timeout=240 (Drbd2-start-interval-0s)<br>               promote interval=0s timeout=90 (Drbd2-promote-interval-0s)<br>               demote interval=0s timeout=90 (Drbd2-demote-interval-0s)<br>               stop interval=0s timeout=100 (Drbd2-stop-interval-0s)<br>               monitor interval=60s (Drbd2-monitor-interval-60s)<br> Clone: Virtfs2-clone<br>  Resource: Virtfs2 (class=ocf provider=heartbeat type=Filesystem)<br>   Attributes: device=/dev/drbd2 directory=/virtfs2 fstype=gfs2<br>   Operations: start interval=0s timeout=60 (Virtfs2-start-interval-0s)<br>               stop interval=0s timeout=60 (Virtfs2-stop-interval-0s)<br>               monitor interval=20 timeout=40 (Virtfs2-monitor-interval-20)<br>Stonith Devices:<br> Resource: fencing-idrac1 (class=stonith type=fence_idrac)<br>  Attributes: pcmk_host_list=wirt1v ipaddr=172.31.0.223 lanplus=on login=root passwd=my1secret2password3<br>  Operations: monitor interval=60 (fencing-idrac1-monitor-<wbr>interval-60)<br> Resource: fencing-idrac2 (class=stonith type=fence_idrac)<br>  Attributes: pcmk_host_list=wirt2v ipaddr=172.31.0.224 lanplus=on login=root passwd=my1secret2password3<br>  Operations: monitor interval=60 (fencing-idrac2-monitor-<wbr>interval-60)<br>Fencing Levels:<br>Location Constraints:<br>Ordering Constraints:<br>  start dlm-clone then start Virtfs2-clone (kind:Mandatory) (id:order-dlm-clone-Virtfs2-<wbr>clone-mandatory)<br>  promote Drbd2-clone then start Virtfs2-clone (kind:Mandatory) (id:order-Drbd2-clone-Virtfs2-<wbr>clone-mandatory)<br>Colocation Constraints:<br>  Virtfs2-clone with Drbd2-clone (score:INFINITY) (with-rsc-role:Master) (id:colocation-Virtfs2-clone-<wbr>Drbd2-clone-INFINITY)<br>  Virtfs2-clone with dlm-clone (score:INFINITY) (id:colocation-Virtfs2-clone-<wbr>dlm-clone-INFINITY)<br>Resources Defaults:<br> resource-stickiness: 100<br>Operations Defaults:<br> No defaults set<br>Cluster Properties:<br> cluster-infrastructure: corosync<br> cluster-name: klasterek<br> dc-version: 1.1.13-10.el7_2.4-44eb2dd<br> have-watchdog: false<br> no-quorum-policy: ignore<br> stonith-enabled: true<br> symmetric-cluster: true<br><br>#-----------------------------<wbr>----<br>### result:  pcs status  ###<br><br>Cluster name: klasterek<br>Last updated: Sun Sep 11 16:02:15 2016          Last change: Sun Sep 11 15:01:05 2016 by root via crm_attribute on wirt2v<br>Stack: corosync<br>Current DC: wirt1v (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum<br>2 nodes and 8 resources configured<br>Online: [ wirt1v wirt2v ]<br>Full list of resources:<br> fencing-idrac1 (stonith:fence_idrac):  Started wirt1v<br> fencing-idrac2 (stonith:fence_idrac):  Started wirt1v<br> Clone Set: dlm-clone [dlm]<br>     Started: [ wirt1v wirt2v ]<br> Master/Slave Set: Drbd2-clone [Drbd2]<br>     Masters: [ wirt1v wirt2v ]<br> Clone Set: Virtfs2-clone [Virtfs2]<br>     Started: [ wirt1v wirt2v ]<br>PCSD Status:<br>  wirt1v: Online<br>  wirt2v: Online<br>Daemon Status:<br>  corosync: active/disabled<br>  pacemaker: active/disabled<br>  pcsd: active/enabled<br><br>#-----------------------------<wbr>----<br>### result:  pcs property  ###<br><br>Cluster Properties:<br> cluster-infrastructure: corosync<br> cluster-name: klasterek<br> dc-version: 1.1.13-10.el7_2.4-44eb2dd<br> have-watchdog: false<br> no-quorum-policy: ignore<br> stonith-enabled: true<br> symmetric-cluster: true<br><br>#-----------------------------<wbr>----<br>### result:  pcs cluster cib  ###<br><br><cib crm_feature_set="3.0.10" validate-with="pacemaker-2.3" epoch="105" num_updates="11" admin_epoch="0" cib-last-written="Sun Sep 11 15:01:05 2016" update-origin="wirt2v" update-client="crm_attribute" update-user="root" have-quorum="1" dc-uuid="1"><br>  <configuration><br>    <crm_config><br>      <cluster_property_set id="cib-bootstrap-options"><br>        <nvpair id="cib-bootstrap-options-<wbr>have-watchdog" name="have-watchdog" value="false"/><br>        <nvpair id="cib-bootstrap-options-dc-<wbr>version" name="dc-version" value="1.1.13-10.el7_2.4-<wbr>44eb2dd"/><br>        <nvpair id="cib-bootstrap-options-<wbr>cluster-infrastructure" name="cluster-infrastructure" value="corosync"/><br>        <nvpair id="cib-bootstrap-options-<wbr>cluster-name" name="cluster-name" value="klasterek"/><br>        <nvpair id="cib-bootstrap-options-<wbr>stonith-enabled" name="stonith-enabled" value="true"/><br>        <nvpair id="cib-bootstrap-options-<wbr>symmetric-cluster" name="symmetric-cluster" value="true"/><br>        <nvpair id="cib-bootstrap-options-no-<wbr>quorum-policy" name="no-quorum-policy" value="ignore"/><br>      </cluster_property_set><br>    </crm_config><br>    <nodes><br>      <node id="1" uname="wirt1v"><br>        <instance_attributes id="nodes-1"/><br>      </node><br>      <node id="2" uname="wirt2v"><br>        <instance_attributes id="nodes-2"/><br>      </node><br>    </nodes><br>    <resources><br>      <primitive class="stonith" id="fencing-idrac1" type="fence_idrac"><br>        <instance_attributes id="fencing-idrac1-instance_<wbr>attributes"><br>          <nvpair id="fencing-idrac1-instance_<wbr>attributes-pcmk_host_list" name="pcmk_host_list" value="wirt1v"/><br>          <nvpair id="fencing-idrac1-instance_<wbr>attributes-ipaddr" name="ipaddr" value="172.31.0.223"/><br>          <nvpair id="fencing-idrac1-instance_<wbr>attributes-lanplus" name="lanplus" value="on"/><br>          <nvpair id="fencing-idrac1-instance_<wbr>attributes-login" name="login" value="root"/><br>          <nvpair id="fencing-idrac1-instance_<wbr>attributes-passwd" name="passwd" value="my1secret2password3"/><br>        </instance_attributes><br>        <operations><br>          <op id="fencing-idrac1-monitor-<wbr>interval-60" interval="60" name="monitor"/><br>        </operations><br>        <meta_attributes id="fencing-idrac1-meta_<wbr>attributes"/><br>      </primitive><br>      <primitive class="stonith" id="fencing-idrac2" type="fence_idrac"><br>        <instance_attributes id="fencing-idrac2-instance_<wbr>attributes"><br>          <nvpair id="fencing-idrac2-instance_<wbr>attributes-pcmk_host_list" name="pcmk_host_list" value="wirt2v"/><br>          <nvpair id="fencing-idrac2-instance_<wbr>attributes-ipaddr" name="ipaddr" value="172.31.0.224"/><br>          <nvpair id="fencing-idrac2-instance_<wbr>attributes-lanplus" name="lanplus" value="on"/><br>          <nvpair id="fencing-idrac2-instance_<wbr>attributes-login" name="login" value="root"/><br>          <nvpair id="fencing-idrac2-instance_<wbr>attributes-passwd" name="passwd" value="my1secret2password3"/><br>        </instance_attributes><br>        <operations><br>          <op id="fencing-idrac2-monitor-<wbr>interval-60" interval="60" name="monitor"/><br>        </operations><br>        <meta_attributes id="fencing-idrac2-meta_<wbr>attributes"/><br>      </primitive><br>      <clone id="dlm-clone"><br>        <primitive class="ocf" id="dlm" provider="pacemaker" type="controld"><br>          <instance_attributes id="dlm-instance_attributes"/><br>          <operations><br>            <op id="dlm-start-interval-0s" interval="0s" name="start" timeout="90"/><br>            <op id="dlm-stop-interval-0s" interval="0s" name="stop" timeout="100"/><br>            <op id="dlm-monitor-interval-60s" interval="60s" name="monitor"/><br>          </operations><br>        </primitive><br>        <meta_attributes id="dlm-clone-meta_attributes"<wbr>><br>          <nvpair id="dlm-clone-max" name="clone-max" value="2"/><br>          <nvpair id="dlm-clone-node-max" name="clone-node-max" value="1"/><br>        </meta_attributes><br>      </clone><br>        <primitive class="ocf" id="Drbd2" provider="linbit" type="drbd"><br>          <instance_attributes id="Drbd2-instance_attributes"<wbr>><br>            <nvpair id="Drbd2-instance_attributes-<wbr>drbd_resource" name="drbd_resource" value="drbd2"/><br>          </instance_attributes><br>          <operations><br>            <op id="Drbd2-start-interval-0s" interval="0s" name="start" timeout="240"/><br>            <op id="Drbd2-promote-interval-0s" interval="0s" name="promote" timeout="90"/><br>            <op id="Drbd2-demote-interval-0s" interval="0s" name="demote" timeout="90"/><br>            <op id="Drbd2-stop-interval-0s" interval="0s" name="stop" timeout="100"/><br>            <op id="Drbd2-monitor-interval-<wbr>60s" interval="60s" name="monitor"/><br>          </operations><br>        </primitive><br>        <meta_attributes id="Drbd2-clone-meta_<wbr>attributes"><br>          <nvpair id="Drbd2-clone-meta_<wbr>attributes-master-max" name="master-max" value="2"/><br>          <nvpair id="Drbd2-clone-meta_<wbr>attributes-master-node-max" name="master-node-max" value="1"/><br>          <nvpair id="Drbd2-clone-meta_<wbr>attributes-clone-max" name="clone-max" value="2"/><br>          <nvpair id="Drbd2-clone-meta_<wbr>attributes-clone-node-max" name="clone-node-max" value="1"/><br>          <nvpair id="Drbd2-clone-meta_<wbr>attributes-notify" name="notify" value="true"/><br>        </meta_attributes><br>      </master><br>      <clone id="Virtfs2-clone"><br>        <primitive class="ocf" id="Virtfs2" provider="heartbeat" type="Filesystem"><br>          <instance_attributes id="Virtfs2-instance_<wbr>attributes"><br>            <nvpair id="Virtfs2-instance_<wbr>attributes-device" name="device" value="/dev/drbd2"/><br>            <nvpair id="Virtfs2-instance_<wbr>attributes-directory" name="directory" value="/virtfs2"/><br>            <nvpair id="Virtfs2-instance_<wbr>attributes-fstype" name="fstype" value="gfs2"/><br>          </instance_attributes><br>          <operations><br>            <op id="Virtfs2-start-interval-0s" interval="0s" name="start" timeout="60"/><br>            <op id="Virtfs2-stop-interval-0s" interval="0s" name="stop" timeout="60"/><br>            <op id="Virtfs2-monitor-interval-<wbr>20" interval="20" name="monitor" timeout="40"/><br>          </operations><br>        </primitive><br>        <meta_attributes id="Virtfs2-clone-meta_<wbr>attributes"/><br>      </clone><br>    </resources><br>    <constraints><br>      <rsc_colocation id="colocation-Virtfs2-clone-<wbr>Drbd2-clone-INFINITY" rsc="Virtfs2-clone" score="INFINITY" with-rsc="Drbd2-clone" with-rsc-role="Master"/><br>      <rsc_colocation id="colocation-Virtfs2-clone-<wbr>dlm-clone-INFINITY" rsc="Virtfs2-clone" score="INFINITY" with-rsc="dlm-clone"/><br>      <rsc_order first="dlm-clone" first-action="start" id="order-dlm-clone-Virtfs2-<wbr>clone-mandatory" then="Virtfs2-clone" then-action="start"/><br>      <rsc_order first="Drbd2-clone" first-action="promote" id="order-Drbd2-clone-Virtfs2-<wbr>clone-mandatory" then="Virtfs2-clone" then-action="start"/><br>    </constraints><br>    <rsc_defaults><br>      <meta_attributes id="rsc_defaults-options"><br>        <nvpair id="rsc_defaults-options-<wbr>resource-stickiness" name="resource-stickiness" value="100"/><br>      </meta_attributes><br>    </rsc_defaults><br>  </configuration><br>  <status><br>    <node_state id="1" uname="wirt1v" in_ccm="true" crmd="online" crm-debug-origin="do_update_<wbr>resource" join="member" expected="member"><br>      <transient_attributes id="1"><br>        <instance_attributes id="status-1"><br>          <nvpair id="status-1-shutdown" name="shutdown" value="0"/><br>          <nvpair id="status-1-probe_complete" name="probe_complete" value="true"/><br>          <nvpair id="status-1-master-Drbd2" name="master-Drbd2" value="10000"/><br>        </instance_attributes><br>      </transient_attributes><br>      <lrm id="1"><br>        <lrm_resources><br>          <lrm_resource id="Virtfs2" type="Filesystem" class="ocf" provider="heartbeat"><br>            <lrm_rsc_op id="Virtfs2_last_0" operation_key="Virtfs2_start_<wbr>0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="55:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;55:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="96" rc-code="0" op-status="0" interval="0" last-run="1473598870" last-rc-change="1473598870" exec-time="639" queue-time="1" op-digest="<wbr>8dbd904c2115508ebcf3dffe8e7c6d<wbr>82"/><br>            <lrm_rsc_op id="Virtfs2_monitor_20000" operation_key="Virtfs2_<wbr>monitor_20000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="56:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;56:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="97" rc-code="0" op-status="0" interval="20000" last-rc-change="1473598871" exec-time="41" queue-time="0" op-digest="<wbr>051271837d1a8eccc0af38fbd8c406<wbr>e4"/><br>          </lrm_resource><br>          <lrm_resource id="fencing-idrac1" type="fence_idrac" class="stonith"><br>            <lrm_rsc_op id="fencing-idrac1_last_0" operation_key="fencing-idrac1_<wbr>start_0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="9:58:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;9:58:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="58" rc-code="0" op-status="0" interval="0" last-run="1473588685" last-rc-change="1473588685" exec-time="1045" queue-time="0" op-digest="<wbr>23a748cdf02f6f0fd03ac9823fc9bd<wbr>52" op-secure-params=" passwd " op-secure-digest="<wbr>2a5376722a6d891302b4e811e4de5c<wbr>5a"/><br>            <lrm_rsc_op id="fencing-idrac1_monitor_<wbr>60000" operation_key="fencing-idrac1_<wbr>monitor_60000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="11:59:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;11:59:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="59" rc-code="0" op-status="0" interval="60000" last-rc-change="1473588687" exec-time="84" queue-time="0" op-digest="<wbr>592f6bfb8f36e6645a6221de49f6f3<wbr>b3" op-secure-params=" passwd " op-secure-digest="<wbr>2a5376722a6d891302b4e811e4de5c<wbr>5a"/><br>          </lrm_resource><br>          <lrm_resource id="fencing-idrac2" type="fence_idrac" class="stonith"><br>            <lrm_rsc_op id="fencing-idrac2_last_0" operation_key="fencing-idrac2_<wbr>start_0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="14:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;14:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="61" rc-code="0" op-status="0" interval="0" last-run="1473590528" last-rc-change="1473590528" exec-time="80" queue-time="0" op-digest="<wbr>268b7ef79bdf7a09609aa321d3d18a<wbr>61" op-secure-params=" passwd " op-secure-digest="<wbr>f22e287dc4906f866a82eac0ab75d2<wbr>17"/><br>            <lrm_rsc_op id="fencing-idrac2_monitor_<wbr>60000" operation_key="fencing-idrac2_<wbr>monitor_60000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="15:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;15:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt1v" call-id="62" rc-code="0" op-status="0" interval="60000" last-rc-change="1473590529" exec-time="75" queue-time="1" op-digest="<wbr>40430ed0cd93e10fcba03a5e867b2a<wbr>f3" op-secure-params=" passwd " op-secure-digest="<wbr>f22e287dc4906f866a82eac0ab75d2<wbr>17"/><br>          </lrm_resource><br>          <lrm_resource id="dlm" type="controld" class="ocf" provider="pacemaker"><br>            <lrm_rsc_op id="dlm_last_0" operation_key="dlm_start_0" operation="start" crm-debug-origin="build_<wbr>active_RAs" crm_feature_set="3.0.10" transition-key="14:15:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" transition-magic="0:0;14:15:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" on_node="wirt1v" call-id="25" rc-code="0" op-status="0" interval="0" last-run="1473544487" last-rc-change="1473544487" exec-time="1082" queue-time="0" op-digest="<wbr>f2317cad3d54cec5d7d7aa7d0bf35c<wbr>f8"/><br>            <lrm_rsc_op id="dlm_monitor_60000" operation_key="dlm_monitor_<wbr>60000" operation="monitor" crm-debug-origin="build_<wbr>active_RAs" crm_feature_set="3.0.10" transition-key="8:16:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" transition-magic="0:0;8:16:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" on_node="wirt1v" call-id="28" rc-code="0" op-status="0" interval="60000" last-rc-change="1473544489" exec-time="38" queue-time="0" op-digest="<wbr>4811cef7f7f94e3a35a70be7916cb2<wbr>fd"/><br>          </lrm_resource><br>          <lrm_resource id="Drbd2" type="drbd" class="ocf" provider="linbit"><br>            <lrm_rsc_op id="Drbd2_last_0" operation_key="Drbd2_promote_<wbr>0" operation="promote" crm-debug-origin="build_<wbr>active_RAs" crm_feature_set="3.0.10" transition-key="17:16:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" transition-magic="0:0;17:16:0:<wbr>29cf445c-f17d-4274-89e9-<wbr>a869e4783c46" on_node="wirt1v" call-id="30" rc-code="0" op-status="0" interval="0" last-run="1473544489" last-rc-change="1473544489" exec-time="58" queue-time="0" op-digest="<wbr>d0c8a735862843030d8426a5218ceb<wbr>92"/><br>          </lrm_resource><br>        </lrm_resources><br>      </lrm><br>    </node_state><br>    <node_state id="2" uname="wirt2v" in_ccm="true" crmd="online" crm-debug-origin="do_update_<wbr>resource" join="member" expected="member"><br>      <transient_attributes id="2"><br>        <instance_attributes id="status-2"><br>          <nvpair id="status-2-shutdown" name="shutdown" value="0"/><br>          <nvpair id="status-2-probe_complete" name="probe_complete" value="true"/><br>          <nvpair id="status-2-master-Drbd2" name="master-Drbd2" value="10000"/><br>        </instance_attributes><br>      </transient_attributes><br>      <lrm id="2"><br>        <lrm_resources><br>          <lrm_resource id="fencing-idrac1" type="fence_idrac" class="stonith"><br>            <lrm_rsc_op id="fencing-idrac1_last_0" operation_key="fencing-idrac1_<wbr>monitor_0" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="7:1:7:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:7;7:1:7:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="5" rc-code="7" op-status="0" interval="0" last-run="1473544754" last-rc-change="1473544754" exec-time="1" queue-time="0" op-digest="<wbr>23a748cdf02f6f0fd03ac9823fc9bd<wbr>52" op-secure-params=" passwd " op-secure-digest="<wbr>2a5376722a6d891302b4e811e4de5c<wbr>5a"/><br>          </lrm_resource><br>          <lrm_resource id="fencing-idrac2" type="fence_idrac" class="stonith"><br>            <lrm_rsc_op id="fencing-idrac2_last_0" operation_key="fencing-idrac2_<wbr>stop_0" operation="stop" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="13:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;13:62:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="55" rc-code="0" op-status="0" interval="0" last-run="1473590528" last-rc-change="1473590528" exec-time="0" queue-time="0" op-digest="<wbr>268b7ef79bdf7a09609aa321d3d18a<wbr>61" op-secure-params=" passwd " op-secure-digest="<wbr>f22e287dc4906f866a82eac0ab75d2<wbr>17"/><br>            <lrm_rsc_op id="fencing-idrac2_monitor_<wbr>60000" operation_key="fencing-idrac2_<wbr>monitor_60000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="13:59:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;13:59:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="53" rc-code="0" op-status="0" interval="60000" last-rc-change="1473588689" exec-time="63" queue-time="0" op-digest="<wbr>40430ed0cd93e10fcba03a5e867b2a<wbr>f3" op-secure-params=" passwd " op-secure-digest="<wbr>f22e287dc4906f866a82eac0ab75d2<wbr>17"/><br>          </lrm_resource><br>          <lrm_resource id="dlm" type="controld" class="ocf" provider="pacemaker"><br>            <lrm_rsc_op id="dlm_last_0" operation_key="dlm_start_0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="15:83:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;15:83:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="101" rc-code="0" op-status="0" interval="0" last-run="1473598865" last-rc-change="1473598865" exec-time="1116" queue-time="0" op-digest="<wbr>f2317cad3d54cec5d7d7aa7d0bf35c<wbr>f8"/><br>            <lrm_rsc_op id="dlm_monitor_60000" operation_key="dlm_monitor_<wbr>60000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="16:84:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;16:84:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="104" rc-code="0" op-status="0" interval="60000" last-rc-change="1473598870" exec-time="47" queue-time="0" op-digest="<wbr>4811cef7f7f94e3a35a70be7916cb2<wbr>fd"/><br>          </lrm_resource><br>          <lrm_resource id="Drbd2" type="drbd" class="ocf" provider="linbit"><br>            <lrm_rsc_op id="Drbd2_last_0" operation_key="Drbd2_promote_<wbr>0" operation="promote" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="27:84:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;27:84:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="106" rc-code="0" op-status="0" interval="0" last-run="1473598870" last-rc-change="1473598870" exec-time="69" queue-time="0" op-digest="<wbr>d0c8a735862843030d8426a5218ceb<wbr>92"/><br>          </lrm_resource><br>          <lrm_resource id="Virtfs2" type="Filesystem" class="ocf" provider="heartbeat"><br>            <lrm_rsc_op id="Virtfs2_last_0" operation_key="Virtfs2_start_<wbr>0" operation="start" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="53:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;53:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="108" rc-code="0" op-status="0" interval="0" last-run="1473598870" last-rc-change="1473598870" exec-time="859" queue-time="0" op-digest="<wbr>8dbd904c2115508ebcf3dffe8e7c6d<wbr>82"/><br>            <lrm_rsc_op id="Virtfs2_monitor_20000" operation_key="Virtfs2_<wbr>monitor_20000" operation="monitor" crm-debug-origin="do_update_<wbr>resource" crm_feature_set="3.0.10" transition-key="54:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" transition-magic="0:0;54:85:0:<wbr>bfa57406-efca-4ca9-bdb5-<wbr>01a121d172d8" on_node="wirt2v" call-id="109" rc-code="0" op-status="0" interval="20000" last-rc-change="1473598871" exec-time="50" queue-time="0" op-digest="<wbr>051271837d1a8eccc0af38fbd8c406<wbr>e4"/><br>          </lrm_resource><br>        </lrm_resources><br>      </lrm><br>    </node_state><br>  </status><br></cib><br># ==============================<wbr>==============================<wbr>=========<br># wirt1v-during_pcs-cluster-<wbr>standby-wirt2v.log<br>#<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br>Sep 11 14:58:06 wirt1v pengine[31950]:  notice: On loss of CCM Quorum: Ignore<br>Sep 11 14:58:06 wirt1v pengine[31950]:  notice: Stop    dlm:1#011(wirt2v)<br>Sep 11 14:58:06 wirt1v pengine[31950]:  notice: Demote  Drbd2:1#011(Master -> Stopped wirt2v)<br>Sep 11 14:58:06 wirt1v pengine[31950]:  notice: Stop    Virtfs2:1#011(wirt2v)<br>Sep 11 14:58:06 wirt1v pengine[31950]:  notice: Calculated Transition 81: /var/lib/pacemaker/pengine/pe-<wbr>input-2852.bz2<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 71: notify Drbd2_pre_notify_demote_0 on wirt1v (local)<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 73: notify Drbd2_pre_notify_demote_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 54: stop Virtfs2_stop_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=86, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: recover generation 2 done<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 17: stop dlm_stop_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 26: demote Drbd2_demote_0 on wirt2v<br>Sep 11 14:58:06 wirt1v kernel: block drbd2: peer( Primary -> Secondary )<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 72: notify Drbd2_post_notify_demote_0 on wirt1v (local)<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 74: notify Drbd2_post_notify_demote_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=87, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 66: notify Drbd2_pre_notify_stop_0 on wirt1v (local)<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 68: notify Drbd2_pre_notify_stop_0 on wirt2v<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=88, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 27: stop Drbd2_stop_0 on wirt2v<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: peer( Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate -> DUnknown )<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: ack_receiver terminated<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: Terminating drbd_a_drbd2<br>Sep 11 14:58:06 wirt1v kernel: block drbd2: new current UUID 12027A6DEC39CCB7:<wbr>9A297D737BE3FBC7:<wbr>214339307E5385FF:<wbr>214239307E5385FF<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: Connection closed<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: conn( TearDown -> Unconnected )<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: receiver terminated<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: Restarting receiver thread<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: receiver (re)started<br>Sep 11 14:58:06 wirt1v kernel: drbd drbd2: conn( Unconnected -> WFConnection )<br>Sep 11 14:58:06 wirt1v crmd[31951]: warning: No match for shutdown action on 2<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Transition aborted by deletion of nvpair[@id='status-2-master-<wbr>Drbd2']: Transient attribute change (cib=0.104.3, source=abort_unless_down:333, path=/cib/status/node_state[@<wbr>id='2']/transient_attributes[@<wbr>id='2']/instance_attributes[@<wbr>id='status-2']/nvpair[@id='<wbr>status-2-master-Drbd2'], 0)<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Initiating action 67: notify Drbd2_post_notify_stop_0 on wirt1v (local)<br>Sep 11 14:58:06 wirt1v crmd[31951]:  notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=89, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:08 wirt1v crmd[31951]:  notice: Transition 81 (Complete=28, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2852.bz2): Complete<br>Sep 11 14:58:08 wirt1v pengine[31950]:  notice: On loss of CCM Quorum: Ignore<br>Sep 11 14:58:08 wirt1v pengine[31950]:  notice: Calculated Transition 82: /var/lib/pacemaker/pengine/pe-<wbr>input-2853.bz2<br>Sep 11 14:58:08 wirt1v crmd[31951]:  notice: Transition 82 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2853.bz2): Complete<br>Sep 11 14:58:08 wirt1v crmd[31951]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br><br># ==============================<wbr>==============================<wbr>=========<br># wirt1v-during_pcs-cluster-<wbr>unstandby-wirt2v.log<br>#<br>Sep 11 15:01:01 wirt1v systemd: Started Session 59 of user root.<br>Sep 11 15:01:01 wirt1v systemd: Starting Session 59 of user root.<br>Sep 11 15:01:05 wirt1v crmd[31951]:  notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br>Sep 11 15:01:05 wirt1v pengine[31950]:  notice: On loss of CCM Quorum: Ignore<br>Sep 11 15:01:05 wirt1v pengine[31950]:  notice: Start   dlm:1#011(wirt2v)<br>Sep 11 15:01:05 wirt1v pengine[31950]:  notice: Start   Drbd2:1#011(wirt2v)<br>Sep 11 15:01:05 wirt1v pengine[31950]:  notice: Restart Virtfs2:0#011(Started wirt1v)<br>Sep 11 15:01:05 wirt1v pengine[31950]:  notice: Calculated Transition 83: /var/lib/pacemaker/pengine/pe-<wbr>input-2854.bz2<br>Sep 11 15:01:05 wirt1v crmd[31951]:  notice: Initiating action 15: start dlm_start_0 on wirt2v<br>Sep 11 15:01:05 wirt1v crmd[31951]:  notice: Initiating action 61: notify Drbd2_pre_notify_start_0 on wirt1v (local)<br>Sep 11 15:01:05 wirt1v crmd[31951]:  notice: Initiating action 51: stop Virtfs2_stop_0 on wirt1v (local)<br>Sep 11 15:01:06 wirt1v Filesystem(Virtfs2)[42311]: INFO: Running stop for /dev/drbd2 on /virtfs2<br>Sep 11 15:01:06 wirt1v crmd[31951]:  notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=90, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:06 wirt1v crmd[31951]:  notice: Initiating action 25: start Drbd2_start_0 on wirt2v<br>Sep 11 15:01:06 wirt1v Filesystem(Virtfs2)[42311]: INFO: Trying to unmount /virtfs2<br>Sep 11 15:01:06 wirt1v Filesystem(Virtfs2)[42311]: ERROR: Couldn't unmount /virtfs2; trying cleanup with TERM<br>Sep 11 15:01:06 wirt1v crmd[31951]:  notice: Transition aborted by status-2-master-Drbd2, master-Drbd2=1000: Transient attribute change (create cib=0.105.1, source=abort_unless_down:319, path=/cib/status/node_state[@<wbr>id='2']/transient_attributes[@<wbr>id='2']/instance_attributes[@<wbr>id='status-2'], 0)<br>Sep 11 15:01:06 wirt1v Filesystem(Virtfs2)[42311]: INFO: sending signal TERM to: root     39937 39934  0 14:49 pts/0    Ss+    0:00 -bash<br>Sep 11 15:01:06 wirt1v crmd[31951]:  notice: Initiating action 62: notify Drbd2_post_notify_start_0 on wirt1v (local)<br>Sep 11 15:01:06 wirt1v crmd[31951]:  notice: Initiating action 63: notify Drbd2_post_notify_start_0 on wirt2v<br>Sep 11 15:01:06 wirt1v crmd[31951]:  notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=93, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:06 wirt1v kernel: drbd drbd2: Handshake successful: Agreed network protocol version 101<br>Sep 11 15:01:06 wirt1v kernel: drbd drbd2: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.<br>Sep 11 15:01:06 wirt1v kernel: drbd drbd2: conn( WFConnection -> WFReportParams )<br>Sep 11 15:01:06 wirt1v kernel: drbd drbd2: Starting ack_recv thread (from drbd_r_drbd2 [32139])<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: drbd_sync_handshake:<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: self 12027A6DEC39CCB7:<wbr>9A297D737BE3FBC7:<wbr>214339307E5385FF:<wbr>214239307E5385FF bits:0 flags:0<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: peer 9A297D737BE3FBC6:<wbr>0000000000000000:<wbr>214339307E5385FE:<wbr>214239307E5385FF bits:0 flags:0<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: uuid_compare()=1 by rule 70<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: helper command: /sbin/drbdadm before-resync-source minor-2<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: helper command: /sbin/drbdadm before-resync-source minor-2 exit code 0 (0x0)<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: Began resync as SyncSource (will sync 0 KB [0 bits set]).<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: updated sync UUID 12027A6DEC39CCB7:<wbr>9A2A7D737BE3FBC7:<wbr>9A297D737BE3FBC7:<wbr>214339307E5385FF<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: Resync done (total 1 sec; paused 0 sec; 0 K/sec)<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: updated UUIDs 12027A6DEC39CCB7:<wbr>0000000000000000:<wbr>9A2A7D737BE3FBC7:<wbr>9A297D737BE3FBC7<br>Sep 11 15:01:06 wirt1v kernel: block drbd2: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<br>Sep 11 15:01:07 wirt1v Filesystem(Virtfs2)[42311]: ERROR: Couldn't unmount /virtfs2; trying cleanup with TERM<br>Sep 11 15:01:07 wirt1v Filesystem(Virtfs2)[42311]: INFO: sending signal TERM to: root     39937 39934  0 14:49 pts/0    Ss+    0:00 -bash<br>Sep 11 15:01:08 wirt1v Filesystem(Virtfs2)[42311]: ERROR: Couldn't unmount /virtfs2; trying cleanup with TERM<br>Sep 11 15:01:08 wirt1v Filesystem(Virtfs2)[42311]: INFO: sending signal TERM to: root     39937 39934  0 14:49 pts/0    Ss+    0:00 -bash<br>Sep 11 15:01:09 wirt1v Filesystem(Virtfs2)[42311]: ERROR: Couldn't unmount /virtfs2; trying cleanup with KILL<br>Sep 11 15:01:09 wirt1v Filesystem(Virtfs2)[42311]: INFO: sending signal KILL to: root     39937 39934  0 14:49 pts/0    Ss+    0:00 -bash<br>Sep 11 15:01:09 wirt1v systemd-logind: Removed session 58.<br>Sep 11 15:01:10 wirt1v Filesystem(Virtfs2)[42311]: INFO: unmounted /virtfs2 successfully<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [ umount: /virtfs2: target is busy. ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [         (In some cases useful info about processes that use ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [          the device is found by lsof(8) or fuser(1)) ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [ ocf-exit-reason:Couldn't unmount /virtfs2; trying cleanup with TERM ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [ umount: /virtfs2: target is busy. ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [         (In some cases useful info about processes that use ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [          the device is found by lsof(8) or fuser(1)) ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [ ocf-exit-reason:Couldn't unmount /virtfs2; trying cleanup with TERM ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [ umount: /virtfs2: target is busy. ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [         (In some cases useful info about processes that use ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [          the device is found by lsof(8) or fuser(1)) ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [ ocf-exit-reason:Couldn't unmount /virtfs2; trying cleanup with TERM ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [ umount: /virtfs2: target is busy. ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [         (In some cases useful info about processes that use ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [          the device is found by lsof(8) or fuser(1)) ]<br>Sep 11 15:01:10 wirt1v lrmd[31948]:  notice: Virtfs2_stop_0:42311:stderr [ ocf-exit-reason:Couldn't unmount /virtfs2; trying cleanup with KILL ]<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Operation Virtfs2_stop_0: ok (node=wirt1v, call=92, rc=0, cib-update=178, confirmed=true)<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Transition 83 (Complete=18, Pending=0, Fired=0, Skipped=3, Incomplete=5, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2854.bz2): Stopped<br>Sep 11 15:01:10 wirt1v pengine[31950]:  notice: On loss of CCM Quorum: Ignore<br>Sep 11 15:01:10 wirt1v pengine[31950]:  notice: Promote Drbd2:1#011(Slave -> Master wirt2v)<br>Sep 11 15:01:10 wirt1v pengine[31950]:  notice: Start   Virtfs2:0#011(wirt1v)<br>Sep 11 15:01:10 wirt1v pengine[31950]:  notice: Start   Virtfs2:1#011(wirt2v)<br>Sep 11 15:01:10 wirt1v pengine[31950]:  notice: Calculated Transition 84: /var/lib/pacemaker/pengine/pe-<wbr>input-2855.bz2<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Initiating action 16: monitor dlm_monitor_60000 on wirt2v<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Initiating action 69: notify Drbd2_pre_notify_promote_0 on wirt1v (local)<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Initiating action 71: notify Drbd2_pre_notify_promote_0 on wirt2v<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=94, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Initiating action 27: promote Drbd2_promote_0 on wirt2v<br>Sep 11 15:01:10 wirt1v kernel: block drbd2: peer( Secondary -> Primary )<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Initiating action 70: notify Drbd2_post_notify_promote_0 on wirt1v (local)<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Initiating action 72: notify Drbd2_post_notify_promote_0 on wirt2v<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Transition aborted by status-2-master-Drbd2, master-Drbd2=10000: Transient attribute change (modify cib=0.105.7, source=abort_unless_down:319, path=/cib/status/node_state[@<wbr>id='2']/transient_attributes[@<wbr>id='2']/instance_attributes[@<wbr>id='status-2']/nvpair[@id='<wbr>status-2-master-Drbd2'], 0)<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Operation Drbd2_notify_0: ok (node=wirt1v, call=95, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Transition 84 (Complete=13, Pending=0, Fired=0, Skipped=2, Incomplete=5, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2855.bz2): Stopped<br>Sep 11 15:01:10 wirt1v pengine[31950]:  notice: On loss of CCM Quorum: Ignore<br>Sep 11 15:01:10 wirt1v pengine[31950]:  notice: Start   Virtfs2:0#011(wirt2v)<br>Sep 11 15:01:10 wirt1v pengine[31950]:  notice: Start   Virtfs2:1#011(wirt1v)<br>Sep 11 15:01:10 wirt1v pengine[31950]:  notice: Calculated Transition 85: /var/lib/pacemaker/pengine/pe-<wbr>input-2856.bz2<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Initiating action 53: start Virtfs2_start_0 on wirt2v<br>Sep 11 15:01:10 wirt1v crmd[31951]:  notice: Initiating action 55: start Virtfs2:1_start_0 on wirt1v (local)<br>Sep 11 15:01:10 wirt1v Filesystem(Virtfs2)[42615]: INFO: Running start for /dev/drbd2 on /virtfs2<br>Sep 11 15:01:10 wirt1v kernel: GFS2: fsid=klasterek:drbd2: Trying to join cluster "lock_dlm", "klasterek:drbd2"<br>Sep 11 15:01:10 wirt1v kernel: dlm: Using TCP for communications<br>Sep 11 15:01:10 wirt1v kernel: dlm: connecting to 2<br>Sep 11 15:01:10 wirt1v kernel: dlm: got connection from 2<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2: first mounter control generation 0<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2: Joined cluster. Now mounting FS...<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=0, already locked for use<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=0: Looking at journal...<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=0: Done<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=1: Trying to acquire journal lock...<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=1: Looking at journal...<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: jid=1: Done<br>Sep 11 15:01:11 wirt1v kernel: GFS2: fsid=klasterek:drbd2.0: first mount done, others may mount<br>Sep 11 15:01:11 wirt1v crmd[31951]:  notice: Operation Virtfs2_start_0: ok (node=wirt1v, call=96, rc=0, cib-update=181, confirmed=true)<br>Sep 11 15:01:11 wirt1v crmd[31951]:  notice: Initiating action 56: monitor Virtfs2:1_monitor_20000 on wirt1v (local)<br>Sep 11 15:01:11 wirt1v crmd[31951]:  notice: Initiating action 54: monitor Virtfs2_monitor_20000 on wirt2v<br>Sep 11 15:01:11 wirt1v crmd[31951]:  notice: Transition 85 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/<wbr>pengine/pe-input-2856.bz2): Complete<br>Sep 11 15:01:11 wirt1v crmd[31951]:  notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>Sep 11 15:01:18 wirt1v systemd-logind: New session 60 of user root.<br>Sep 11 15:01:18 wirt1v systemd: Started Session 60 of user root.<br>Sep 11 15:01:18 wirt1v systemd: Starting Session 60 of user root.<br><br># ==============================<wbr>==============================<wbr>=========<br># wirt2v-during_pcs-cluster-<wbr>standby-wirt2v.log<br>#<br>Sep 11 14:58:06 wirt2v crmd[27038]:  notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=92, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt2v Filesystem(Virtfs2)[28208]: INFO: Running stop for /dev/drbd2 on /virtfs2<br>Sep 11 14:58:06 wirt2v Filesystem(Virtfs2)[28208]: INFO: Trying to unmount /virtfs2<br>Sep 11 14:58:06 wirt2v Filesystem(Virtfs2)[28208]: INFO: unmounted /virtfs2 successfully<br>Sep 11 14:58:06 wirt2v crmd[27038]:  notice: Operation Virtfs2_stop_0: ok (node=wirt2v, call=94, rc=0, cib-update=58, confirmed=true)<br>Sep 11 14:58:06 wirt2v kernel: dlm: closing connection to node 2<br>Sep 11 14:58:06 wirt2v kernel: dlm: closing connection to node 1<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: role( Primary -> Secondary )<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: bitmap WRITE of 0 pages took 0 jiffies<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>Sep 11 14:58:06 wirt2v systemd-udevd: error: /dev/drbd2: Wrong medium type<br>Sep 11 14:58:06 wirt2v crmd[27038]:   error: pcmkRegisterNode: Triggered assert at xml.c:594 : node->type == XML_ELEMENT_NODE<br>Sep 11 14:58:06 wirt2v crmd[27038]:  notice: Operation Drbd2_demote_0: ok (node=wirt2v, call=97, rc=0, cib-update=59, confirmed=true)<br>Sep 11 14:58:06 wirt2v systemd-udevd: error: /dev/drbd2: Wrong medium type<br>Sep 11 14:58:06 wirt2v crmd[27038]:  notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=98, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt2v crmd[27038]:  notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=99, rc=0, cib-update=0, confirmed=true)<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: peer( Primary -> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate -> DUnknown )<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: ack_receiver terminated<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: Terminating drbd_a_drbd2<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: Connection closed<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: conn( Disconnecting -> StandAlone )<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: receiver terminated<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: Terminating drbd_r_drbd2<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: disk( UpToDate -> Failed )<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: bitmap WRITE of 0 pages took 0 jiffies<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>Sep 11 14:58:06 wirt2v kernel: block drbd2: disk( Failed -> Diskless )<br>Sep 11 14:58:06 wirt2v kernel: drbd drbd2: Terminating drbd_w_drbd2<br>Sep 11 14:58:06 wirt2v crmd[27038]:  notice: Operation Drbd2_stop_0: ok (node=wirt2v, call=100, rc=0, cib-update=60, confirmed=true)<br>Sep 11 14:58:08 wirt2v crmd[27038]:  notice: Operation dlm_stop_0: ok (node=wirt2v, call=96, rc=0, cib-update=61, confirmed=true)<br><br># ==============================<wbr>==============================<wbr>=========<br># wirt2v-during_pcs-cluster-<wbr>unstandby-wirt2v.log<br>#<br>Sep 11 15:01:01 wirt2v systemd: Started Session 51 of user root.<br>Sep 11 15:01:01 wirt2v systemd: Starting Session 51 of user root.<br>Sep 11 15:01:06 wirt2v dlm_controld[28577]: 62718 dlm_controld 4.0.2 started<br>Sep 11 15:01:06 wirt2v systemd-udevd: error: /dev/drbd2: Wrong medium type<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Starting worker thread (from drbdsetup-84 [28610])<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: disk( Diskless -> Attaching )<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Method to ensure write ordering: flush<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: max BIO size = 262144<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: drbd_bm_resize called with capacity == 104854328<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: resync bitmap: bits=<a href="tel:13106791" value="+13106791" target="_blank">13106791</a> words=204794 pages=400<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: size = 50 GB (52427164 KB)<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: recounting of set bits took additional 1 jiffies<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: disk( Attaching -> UpToDate )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: attached to UUIDs 9A297D737BE3FBC6:<wbr>0000000000000000:<wbr>214339307E5385FE:<wbr>214239307E5385FF<br>Sep 11 15:01:06 wirt2v systemd-udevd: error: /dev/drbd2: Wrong medium type<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: conn( StandAlone -> Unconnected )<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Starting receiver thread (from drbd_w_drbd2 [28612])<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: receiver (re)started<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: conn( Unconnected -> WFConnection )<br>Sep 11 15:01:06 wirt2v crmd[27038]:   error: pcmkRegisterNode: Triggered assert at xml.c:594 : node->type == XML_ELEMENT_NODE<br>Sep 11 15:01:06 wirt2v crmd[27038]:  notice: Operation Drbd2_start_0: ok (node=wirt2v, call=102, rc=0, cib-update=62, confirmed=true)<br>Sep 11 15:01:06 wirt2v crmd[27038]:  notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=103, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Handshake successful: Agreed network protocol version 101<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: conn( WFConnection -> WFReportParams )<br>Sep 11 15:01:06 wirt2v kernel: drbd drbd2: Starting ack_recv thread (from drbd_r_drbd2 [28622])<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: drbd_sync_handshake:<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: self 9A297D737BE3FBC6:<wbr>0000000000000000:<wbr>214339307E5385FE:<wbr>214239307E5385FF bits:0 flags:0<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: peer 12027A6DEC39CCB7:<wbr>9A297D737BE3FBC7:<wbr>214339307E5385FF:<wbr>214239307E5385FF bits:0 flags:0<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: uuid_compare()=-1 by rule 50<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( UpToDate -> Outdated ) pdsk( DUnknown -> UpToDate )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23; compression: 100.0%<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: conn( WFBitMapT -> WFSyncUUID )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: updated sync uuid 9A2A7D737BE3FBC6:<wbr>0000000000000000:<wbr>214339307E5385FE:<wbr>214239307E5385FF<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: Began resync as SyncTarget (will sync 0 KB [0 bits set]).<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: Resync done (total 1 sec; paused 0 sec; 0 K/sec)<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: updated UUIDs 12027A6DEC39CCB6:<wbr>0000000000000000:<wbr>9A2A7D737BE3FBC6:<wbr>9A297D737BE3FBC7<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: helper command: /sbin/drbdadm after-resync-target minor-2<br>Sep 11 15:01:06 wirt2v kernel: block drbd2: helper command: /sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)<br>Sep 11 15:01:07 wirt2v crmd[27038]:  notice: Operation dlm_start_0: ok (node=wirt2v, call=101, rc=0, cib-update=63, confirmed=true)<br>Sep 11 15:01:10 wirt2v crmd[27038]:  notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=105, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:10 wirt2v kernel: block drbd2: role( Secondary -> Primary )<br>Sep 11 15:01:10 wirt2v crmd[27038]:   error: pcmkRegisterNode: Triggered assert at xml.c:594 : node->type == XML_ELEMENT_NODE<br>Sep 11 15:01:10 wirt2v crmd[27038]:  notice: Operation Drbd2_promote_0: ok (node=wirt2v, call=106, rc=0, cib-update=65, confirmed=true)<br>Sep 11 15:01:10 wirt2v crmd[27038]:  notice: Operation Drbd2_notify_0: ok (node=wirt2v, call=107, rc=0, cib-update=0, confirmed=true)<br>Sep 11 15:01:10 wirt2v Filesystem(Virtfs2)[28772]: INFO: Running start for /dev/drbd2 on /virtfs2<br>Sep 11 15:01:10 wirt2v kernel: GFS2: fsid=klasterek:drbd2: Trying to join cluster "lock_dlm", "klasterek:drbd2"<br>Sep 11 15:01:10 wirt2v kernel: dlm: Using TCP for communications<br>Sep 11 15:01:10 wirt2v kernel: dlm: connecting to 1<br>Sep 11 15:01:10 wirt2v kernel: dlm: got connection from 1<br>Sep 11 15:01:11 wirt2v kernel: GFS2: fsid=klasterek:drbd2: Joined cluster. Now mounting FS...<br>Sep 11 15:01:11 wirt2v kernel: GFS2: fsid=klasterek:drbd2.1: jid=1, already locked for use<br>Sep 11 15:01:11 wirt2v kernel: GFS2: fsid=klasterek:drbd2.1: jid=1: Looking at journal...<br>Sep 11 15:01:11 wirt2v kernel: GFS2: fsid=klasterek:drbd2.1: jid=1: Done<br>Sep 11 15:01:11 wirt2v crmd[27038]:  notice: Operation Virtfs2_start_0: ok (node=wirt2v, call=108, rc=0, cib-update=66, confirmed=true)<br><br>==============================<wbr>==============================<wbr>==                               <wbr>                              <wbr>                          <br><br><br></div>
<br>______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br></div></div>