[ClusterLabs] Parallel adding of resources

Arjun Pandey apandepublic at gmail.com
Thu Jan 14 02:19:55 EST 2016


Please ignore this mail.  I have sent out the  email with the correct
topic  separately :)

My apologies.

Regards
Arjun

On Thu, Jan 14, 2016 at 12:43 PM, Arjun Pandey <apandepublic at gmail.com> wrote:
> Hi
>
> I am running a 2 node cluster with this config on centos 6.6
>
> Master/Slave Set: foo-master [foo]
>    Masters: [ kamet ]
>    Slaves: [ orana ]
> fence-uc-orana (stonith:fence_ilo4): Started kamet
> fence-uc-kamet (stonith:fence_ilo4): Started orana
> C-3 (ocf::pw:IPaddr): Started kamet
> C-FLT (ocf::pw:IPaddr): Started kamet
> C-FLT2 (ocf::pw:IPaddr): Started kamet
> E-3 (ocf::pw:IPaddr): Started kamet
> MGMT-FLT (ocf::pw:IPaddr): Started kamet
> M-FLT (ocf::pw:IPaddr): Started kamet
> M-FLT2 (ocf::pw:IPaddr): Started kamet
> S-FLT (ocf::pw:IPaddr): Started kamet
> S-FLT2 (ocf::pw:IPaddr): Started kamet
>
>
> where i have a multi-state resource foo being run in master/slave mode
> and  IPaddr RA is just modified IPAddr2 RA. Additionally i have a
> collocation constraint for the IP addr to be collocated with the master.
> I have additionally configured fencing and when i plug out the
> redundancy interface fencing gets triggered correctly. However once
> the fenced node(kamet) is rejoining i see all my floating IP resources
> are deleted
> and system looks to be in this state. Also if i log into kamet i see
> that the floating ip addresses are actually available.
>
> Master/Slave Set: foo-master [foo]
>    Masters: [ orana ]
>    Slaves: [ kamet ]
> fence-uc-orana (stonith:fence_ilo4): Started orana
> fence-uc-kamet (stonith:fence_ilo4): Started orana
>
> CIB state post fencing of kamet.
> <cib admin_epoch="0" cib-last-written="Wed Jan 13 19:28:22 2016"
> crm_feature_set="3.0.9" epoch="72" num_updates="7"
> validate-with="pacemaker-2.0" have-quorum="1" dc-uuid="orana">
>   <configuration>
>     <crm_config>
>       <cluster_property_set id="cib-bootstrap-options">
>         <nvpair id="cib-bootstrap-options-no-quorum-policy"
> name="no-quorum-policy" value="ignore"/>
>         <nvpair id="cib-bootstrap-options-cluster-recheck-interval"
> name="cluster-recheck-interval" value="30s"/>
>         <nvpair id="cib-bootstrap-options-stonith-enabled"
> name="stonith-enabled" value="true"/>
>         <nvpair id="cib-bootstrap-options-dc-version"
> name="dc-version" value="1.1.11-97629de"/>
>         <nvpair id="cib-bootstrap-options-cluster-infrastructure"
> name="cluster-infrastructure" value="cman"/>
>         <nvpair id="cib-bootstrap-options-last-lrm-refresh"
> name="last-lrm-refresh" value="1452665061"/>
>       </cluster_property_set>
>     </crm_config>
>     <nodes>
>       <node id="kamet" uname="kamet"/>
>       <node id="orana" uname="orana"/>
>     </nodes>
>     <resources>
>       <master id="foo-master">
>         <primitive class="ocf" id="foo" provider="pw" type="uc">
>           <instance_attributes id="foo-instance_attributes">
>             <nvpair id="foo-instance_attributes-state" name="state"
> value="/var/run/uc/role"/>
>           </instance_attributes>
>           <operations>
>             <op id="foo-start-interval-0s" interval="0s" name="start"
> on-fail="restart" timeout="100s"/>
>             <op id="foo-monitor-interval-10s-role-Master"
> interval="10s" name="monitor" on-fail="restart" role="Master"
> timeout="100s"/>
>             <op id="foo-monitor-interval-11s-role-Slave"
> interval="11s" name="monitor" on-fail="restart" role="Slave"
> timeout="100s"/>
>             <op id="foo-promote-interval-0s" interval="0s"
> name="promote" on-fail="restart" timeout="100s"/>
>             <op id="foo-demote-interval-0s" interval="0s"
> name="demote" on-fail="restart" timeout="100s"/>
>             <op id="foo-stop-interval-0s" interval="0s" name="stop"
> on-fail="restart" timeout="100s"/>
>           </operations>
>         </primitive>
>         <meta_attributes id="foo-master-meta_attributes">
>           <nvpair id="foo-master-meta_attributes-master-max"
> name="master-max" value="1"/>
>           <nvpair id="foo-master-meta_attributes-master-node-max"
> name="master-node-max" value="1"/>
>           <nvpair id="foo-master-meta_attributes-clone-max"
> name="clone-max" value="2"/>
>           <nvpair id="foo-master-meta_attributes-clone-node-max"
> name="clone-node-max" value="1"/>
>           <nvpair id="foo-master-meta_attributes-notify" name="notify"
> value="true"/>
>           <nvpair id="foo-master-meta_attributes-ordered"
> name="ordered" value="true"/>
>         </meta_attributes>
>       </master>
>       <primitive class="stonith" id="fence-uc-orana" type="fence_ilo4">
>         <instance_attributes id="fence-uc-orana-instance_attributes">
>           <nvpair id="fence-uc-orana-instance_attributes-login"
> name="login" value="parallel"/>
>           <nvpair id="fence-uc-orana-instance_attributes-passwd"
> name="passwd" value="wireless"/>
>           <nvpair id="fence-uc-orana-instance_attributes-ipaddr"
> name="ipaddr" value="10.11.10.30"/>
>           <nvpair
> id="fence-uc-orana-instance_attributes-pcmk_host_list"
> name="pcmk_host_list" value="orana"/>
>           <nvpair id="fence-uc-orana-instance_attributes-action"
> name="action" value="off"/>
>           <nvpair id="fence-uc-orana-instance_attributes-lanplus"
> name="lanplus" value="1"/>
>           <nvpair id="fence-uc-orana-instance_attributes-delay"
> name="delay" value="10"/>
>         </instance_attributes>
>         <operations>
>           <op id="fence-uc-orana-start-interval-0s" interval="0s"
> name="start" on-fail="restart"/>
>           <op id="fence-uc-orana-monitor-interval-5s" interval="5s"
> name="monitor" on-fail="restart"/>
>         </operations>
>         <meta_attributes id="fence-uc-orana-meta_attributes">
>           <nvpair id="fence-uc-orana-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>         </meta_attributes>
>       </primitive>
>       <primitive class="stonith" id="fence-uc-kamet" type="fence_ilo4">
>         <instance_attributes id="fence-uc-kamet-instance_attributes">
>           <nvpair id="fence-uc-kamet-instance_attributes-login"
> name="login" value="parallel"/>
>           <nvpair id="fence-uc-kamet-instance_attributes-passwd"
> name="passwd" value="wireless"/>
>           <nvpair id="fence-uc-kamet-instance_attributes-ipaddr"
> name="ipaddr" value="10.11.10.21"/>
>           <nvpair
> id="fence-uc-kamet-instance_attributes-pcmk_host_list"
> name="pcmk_host_list" value="kamet"/>
>           <nvpair id="fence-uc-kamet-instance_attributes-action"
> name="action" value="off"/>
>           <nvpair id="fence-uc-kamet-instance_attributes-lanplus"
> name="lanplus" value="1"/>
>           <nvpair id="fence-uc-kamet-instance_attributes-delay"
> name="delay" value="0"/>
>         </instance_attributes>
>         <operations>
>           <op id="fence-uc-kamet-start-interval-0s" interval="0s"
> name="start" on-fail="restart"/>
>           <op id="fence-uc-kamet-monitor-interval-5s" interval="5s"
> name="monitor" on-fail="restart"/>
>         </operations>
>         <meta_attributes id="fence-uc-kamet-meta_attributes"/>
>       </primitive>
>       <primitive class="ocf" id="C-3" provider="pw" type="IPaddr">
>         <instance_attributes id="C-3-instance_attributes">
>           <nvpair id="C-3-instance_attributes-ip" name="ip" value="10.41.1.22"/>
>           <nvpair id="C-3-instance_attributes-nic" name="nic" value="v40"/>
>           <nvpair id="C-3-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="32"/>
>           <nvpair id="C-3-instance_attributes-iflabel" name="iflabel"
> value="C-3"/>
>         </instance_attributes>
>         <operations>
>           <op id="C-3-start-timeout-20s" interval="0s" name="start"
> timeout="20s"/>
>           <op id="C-3-stop-timeout-20s" interval="0s" name="stop"
> timeout="20s"/>
>           <op id="C-3-monitor-interval-200ms" interval="200ms" name="monitor"/>
>         </operations>
>         <meta_attributes id="C-3-meta_attributes">
>           <nvpair id="C-3-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>           <nvpair id="C-3-meta_attributes-migration-threshold"
> name="migration-threshold" value="2"/>
>         </meta_attributes>
>       </primitive>
>       <primitive class="ocf" id="C-FLT" provider="pw" type="IPaddr">
>         <instance_attributes id="C-FLT-instance_attributes">
>           <nvpair id="C-FLT-instance_attributes-ip" name="ip"
> value="10.41.0.108"/>
>           <nvpair id="C-FLT-instance_attributes-nic" name="nic" value="v40"/>
>           <nvpair id="C-FLT-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="32"/>
>           <nvpair id="C-FLT-instance_attributes-iflabel"
> name="iflabel" value="C-FLT"/>
>         </instance_attributes>
>         <operations>
>           <op id="C-FLT-start-timeout-20s" interval="0s" name="start"
> timeout="20s"/>
>           <op id="C-FLT-stop-timeout-20s" interval="0s" name="stop"
> timeout="20s"/>
>           <op id="C-FLT-monitor-interval-200ms" interval="200ms"
> name="monitor"/>
>         </operations>
>         <meta_attributes id="C-FLT-meta_attributes">
>           <nvpair id="C-FLT-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>           <nvpair id="C-FLT-meta_attributes-migration-threshold"
> name="migration-threshold" value="2"/>
>         </meta_attributes>
>       </primitive>
>       <primitive class="ocf" id="C-FLT2" provider="pw" type="IPaddr">
>         <instance_attributes id="C-FLT2-instance_attributes">
>           <nvpair id="C-FLT2-instance_attributes-ip" name="ip"
> value="10.41.1.15"/>
>           <nvpair id="C-FLT2-instance_attributes-nic" name="nic" value="v40"/>
>           <nvpair id="C-FLT2-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="32"/>
>           <nvpair id="C-FLT2-instance_attributes-iflabel"
> name="iflabel" value="C-FLT2"/>
>         </instance_attributes>
>         <operations>
>           <op id="C-FLT2-start-timeout-20s" interval="0s" name="start"
> timeout="20s"/>
>           <op id="C-FLT2-stop-timeout-20s" interval="0s" name="stop"
> timeout="20s"/>
>           <op id="C-FLT2-monitor-interval-200ms" interval="200ms"
> name="monitor"/>
>         </operations>
>         <meta_attributes id="C-FLT2-meta_attributes">
>           <nvpair id="C-FLT2-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>           <nvpair id="C-FLT2-meta_attributes-migration-threshold"
> name="migration-threshold" value="2"/>
>         </meta_attributes>
>       </primitive>
>       <primitive class="ocf" id="E-3" provider="pw" type="IPaddr">
>         <instance_attributes id="E-3-instance_attributes">
>           <nvpair id="E-3-instance_attributes-ip" name="ip" value="10.21.1.22"/>
>           <nvpair id="E-3-instance_attributes-nic" name="nic" value="v20"/>
>           <nvpair id="E-3-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="32"/>
>           <nvpair id="E-3-instance_attributes-iflabel" name="iflabel"
> value="E-3"/>
>         </instance_attributes>
>         <operations>
>           <op id="E-3-start-timeout-20s" interval="0s" name="start"
> timeout="20s"/>
>           <op id="E-3-stop-timeout-20s" interval="0s" name="stop"
> timeout="20s"/>
>           <op id="E-3-monitor-interval-200ms" interval="200ms" name="monitor"/>
>         </operations>
>         <meta_attributes id="E-3-meta_attributes">
>           <nvpair id="E-3-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>           <nvpair id="E-3-meta_attributes-migration-threshold"
> name="migration-threshold" value="2"/>
>         </meta_attributes>
>       </primitive>
>       <primitive class="ocf" id="MGMT-FLT" provider="pw" type="IPaddr">
>         <instance_attributes id="MGMT-FLT-instance_attributes">
>           <nvpair id="MGMT-FLT-instance_attributes-ip" name="ip"
> value="10.61.0.227"/>
>           <nvpair id="MGMT-FLT-instance_attributes-nic" name="nic"
> value="eth0"/>
>           <nvpair id="MGMT-FLT-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="32"/>
>           <nvpair id="MGMT-FLT-instance_attributes-iflabel"
> name="iflabel" value="MGMT-FLT"/>
>         </instance_attributes>
>         <operations>
>           <op id="MGMT-FLT-start-timeout-20s" interval="0s"
> name="start" timeout="20s"/>
>           <op id="MGMT-FLT-stop-timeout-20s" interval="0s" name="stop"
> timeout="20s"/>
>           <op id="MGMT-FLT-monitor-interval-200ms" interval="200ms"
> name="monitor"/>
>         </operations>
>         <meta_attributes id="MGMT-FLT-meta_attributes">
>           <nvpair id="MGMT-FLT-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>           <nvpair id="MGMT-FLT-meta_attributes-migration-threshold"
> name="migration-threshold" value="2"/>
>         </meta_attributes>
>       </primitive>
>       <primitive class="ocf" id="M-FLT" provider="pw" type="IPaddr">
>         <instance_attributes id="M-FLT-instance_attributes">
>           <nvpair id="M-FLT-instance_attributes-ip" name="ip"
> value="10.21.0.108"/>
>           <nvpair id="M-FLT-instance_attributes-nic" name="nic" value="v20"/>
>           <nvpair id="M-FLT-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="32"/>
>           <nvpair id="M-FLT-instance_attributes-iflabel"
> name="iflabel" value="M-FLT"/>
>         </instance_attributes>
>         <operations>
>           <op id="M-FLT-start-timeout-20s" interval="0s" name="start"
> timeout="20s"/>
>           <op id="M-FLT-stop-timeout-20s" interval="0s" name="stop"
> timeout="20s"/>
>           <op id="M-FLT-monitor-interval-200ms" interval="200ms"
> name="monitor"/>
>         </operations>
>         <meta_attributes id="M-FLT-meta_attributes">
>           <nvpair id="M-FLT-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>           <nvpair id="M-FLT-meta_attributes-migration-threshold"
> name="migration-threshold" value="2"/>
>         </meta_attributes>
>       </primitive>
>       <primitive class="ocf" id="M-FLT2" provider="pw" type="IPaddr">
>         <instance_attributes id="M-FLT2-instance_attributes">
>           <nvpair id="M-FLT2-instance_attributes-ip" name="ip"
> value="10.21.1.15"/>
>           <nvpair id="M-FLT2-instance_attributes-nic" name="nic" value="v20"/>
>           <nvpair id="M-FLT2-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="32"/>
>           <nvpair id="M-FLT2-instance_attributes-iflabel"
> name="iflabel" value="M-FLT2"/>
>         </instance_attributes>
>         <operations>
>           <op id="M-FLT2-start-timeout-20s" interval="0s" name="start"
> timeout="20s"/>
>           <op id="M-FLT2-stop-timeout-20s" interval="0s" name="stop"
> timeout="20s"/>
>           <op id="M-FLT2-monitor-interval-200ms" interval="200ms"
> name="monitor"/>
>         </operations>
>         <meta_attributes id="M-FLT2-meta_attributes">
>           <nvpair id="M-FLT2-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>           <nvpair id="M-FLT2-meta_attributes-migration-threshold"
> name="migration-threshold" value="2"/>
>         </meta_attributes>
>       </primitive>
>       <primitive class="ocf" id="S-FLT" provider="pw" type="IPaddr">
>         <instance_attributes id="S-FLT-instance_attributes">
>           <nvpair id="S-FLT-instance_attributes-ip" name="ip"
> value="10.31.0.108"/>
>           <nvpair id="S-FLT-instance_attributes-nic" name="nic" value="v30"/>
>           <nvpair id="S-FLT-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="32"/>
>           <nvpair id="S-FLT-instance_attributes-iflabel"
> name="iflabel" value="S-FLT"/>
>         </instance_attributes>
>         <operations>
>           <op id="S-FLT-start-timeout-20s" interval="0s" name="start"
> timeout="20s"/>
>           <op id="S-FLT-stop-timeout-20s" interval="0s" name="stop"
> timeout="20s"/>
>           <op id="S-FLT-monitor-interval-200ms" interval="200ms"
> name="monitor"/>
>         </operations>
>         <meta_attributes id="S-FLT-meta_attributes">
>           <nvpair id="S-FLT-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>           <nvpair id="S-FLT-meta_attributes-migration-threshold"
> name="migration-threshold" value="2"/>
>         </meta_attributes>
>       </primitive>
>       <primitive class="ocf" id="S-FLT2" provider="pw" type="IPaddr">
>         <instance_attributes id="S-FLT2-instance_attributes">
>           <nvpair id="S-FLT2-instance_attributes-ip" name="ip"
> value="10.31.1.15"/>
>           <nvpair id="S-FLT2-instance_attributes-nic" name="nic" value="v30"/>
>           <nvpair id="S-FLT2-instance_attributes-cidr_netmask"
> name="cidr_netmask" value="32"/>
>           <nvpair id="S-FLT2-instance_attributes-iflabel"
> name="iflabel" value="S-FLT2"/>
>         </instance_attributes>
>         <operations>
>           <op id="S-FLT2-start-timeout-20s" interval="0s" name="start"
> timeout="20s"/>
>           <op id="S-FLT2-stop-timeout-20s" interval="0s" name="stop"
> timeout="20s"/>
>           <op id="S-FLT2-monitor-interval-200ms" interval="200ms"
> name="monitor"/>
>         </operations>
>         <meta_attributes id="S-FLT2-meta_attributes">
>           <nvpair id="S-FLT2-meta_attributes-failure-timeout"
> name="failure-timeout" value="3s"/>
>           <nvpair id="S-FLT2-meta_attributes-migration-threshold"
> name="migration-threshold" value="2"/>
>         </meta_attributes>
>       </primitive>
>     </resources>
>     <constraints>
>       <rsc_colocation id="colocation-C-3-foo-master-INFINITY"
> rsc="C-3" rsc-role="Started" score="INFINITY" with-rsc="foo-master"
> with-rsc-role="Master"/>
>       <rsc_order first="C-3" first-action="start"
> id="order-C-3-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_colocation id="colocation-C-FLT-foo-master-INFINITY"
> rsc="C-FLT" rsc-role="Started" score="INFINITY" with-rsc="foo-master"
> with-rsc-role="Master"/>
>       <rsc_order first="C-FLT" first-action="start"
> id="order-C-FLT-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_colocation id="colocation-C-FLT2-foo-master-INFINITY"
> rsc="C-FLT2" rsc-role="Started" score="INFINITY" with-rsc="foo-master"
> with-rsc-role="Master"/>
>       <rsc_order first="C-FLT2" first-action="start"
> id="order-C-FLT2-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_colocation id="colocation-E-3-foo-master-INFINITY"
> rsc="E-3" rsc-role="Started" score="INFINITY" with-rsc="foo-master"
> with-rsc-role="Master"/>
>       <rsc_order first="E-3" first-action="start"
> id="order-E-3-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_colocation id="colocation-MGMT-FLT-foo-master-INFINITY"
> rsc="MGMT-FLT" rsc-role="Started" score="INFINITY"
> with-rsc="foo-master" with-rsc-role="Master"/>
>       <rsc_order first="MGMT-FLT" first-action="start"
> id="order-MGMT-FLT-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_colocation id="colocation-M-FLT-foo-master-INFINITY"
> rsc="M-FLT" rsc-role="Started" score="INFINITY" with-rsc="foo-master"
> with-rsc-role="Master"/>
>       <rsc_order first="M-FLT" first-action="start"
> id="order-M-FLT-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_colocation id="colocation-M-FLT2-foo-master-INFINITY"
> rsc="M-FLT2" rsc-role="Started" score="INFINITY" with-rsc="foo-master"
> with-rsc-role="Master"/>
>       <rsc_order first="M-FLT2" first-action="start"
> id="order-M-FLT2-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_colocation id="colocation-S-FLT-foo-master-INFINITY"
> rsc="S-FLT" rsc-role="Started" score="INFINITY" with-rsc="foo-master"
> with-rsc-role="Master"/>
>       <rsc_order first="S-FLT" first-action="start"
> id="order-S-FLT-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_colocation id="colocation-S-FLT2-foo-master-INFINITY"
> rsc="S-FLT2" rsc-role="Started" score="INFINITY" with-rsc="foo-master"
> with-rsc-role="Master"/>
>       <rsc_order first="S-FLT2" first-action="start"
> id="order-S-FLT2-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_colocation
> id="colocation-fence-uc-orana-foo-master-INFINITY"
> rsc="fence-uc-orana" rsc-role="Started" score="INFINITY"
> with-rsc="foo-master" with-rsc-role="Master"/>
>       <rsc_colocation
> id="colocation-fence-uc-kamet-foo-master-INFINITY"
> rsc="fence-uc-kamet" rsc-role="Started" score="INFINITY"
> with-rsc="foo-master" with-rsc-role="Master"/>
>       <rsc_order first="fence-uc-kamet" first-action="start"
> id="order-fence-uc-kamet-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>       <rsc_order first="fence-uc-orana" first-action="start"
> id="order-fence-uc-orana-foo-master-mandatory" then="foo-master"
> then-action="promote"/>
>     </constraints>
>   </configuration>
>   <status>
>     <node_state id="kamet" uname="kamet" in_ccm="false" crmd="offline"
> crm-debug-origin="send_stonith_update" join="down" expected="down"/>
>     <node_state id="orana" uname="orana" in_ccm="true" crmd="online"
> crm-debug-origin="do_update_resource" join="member" expected="member">
>       <lrm id="orana">
>         <lrm_resources>
>           <lrm_resource id="fence-uc-kamet" type="fence_ilo4" class="stonith">
>             <lrm_rsc_op id="fence-uc-kamet_last_0"
> operation_key="fence-uc-kamet_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="13:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;13:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="131" rc-code="0" op-status="0" interval="0"
> last-run="1452731302" last-rc-change="1452731302" exec-time="80"
> queue-time="0" op-digest="93013a04ac64e00066a4f7b080b31515"
> on_node="orana"/>
>             <lrm_rsc_op id="fence-uc-kamet_monitor_5000"
> operation_key="fence-uc-kamet_monitor_5000" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="1:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;1:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="132" rc-code="0" op-status="0" interval="5000"
> last-rc-change="1452731303" exec-time="128" queue-time="0"
> op-digest="e884a8e46b98e8563278b8117eb4db82" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="foo" type="uc" class="ocf" provider="pw">
>             <lrm_rsc_op id="foo_last_0" operation_key="foo_promote_0"
> operation="promote" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.9"
> transition-key="19:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;19:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="133" rc-code="0" op-status="0" interval="0"
> last-run="1452731303" last-rc-change="1452731303" exec-time="51"
> queue-time="0" op-digest="96d68f528fd950fa93acae8f44e75df5"
> on_node="orana"/>
>             <lrm_rsc_op id="foo_monitor_10000"
> operation_key="foo_monitor_10000" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="20:2304:8:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:8;20:2304:8:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="135" rc-code="8" op-status="0" interval="10000"
> last-rc-change="1452731303" exec-time="39" queue-time="1"
> op-digest="f4e31338d1a8837389f1948c9c05d8a8" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="C-3" type="IPaddr" class="ocf" provider="pw">
>             <lrm_rsc_op id="C-3_last_0" operation_key="C-3_start_0"
> operation="start" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.9"
> transition-key="36:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;36:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="96" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="123"
> queue-time="0" op-digest="8dd2987e2ec7dfb4c2955839593b8e9f"
> on_node="orana"/>
>             <lrm_rsc_op id="C-3_monitor_200"
> operation_key="C-3_monitor_200" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="37:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;37:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="113" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="93" queue-time="0"
> op-digest="8ede7902b002a044cd67ffc05a799bdc" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="C-FLT" type="IPaddr" class="ocf" provider="pw">
>             <lrm_rsc_op id="C-FLT_last_0"
> operation_key="C-FLT_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="38:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;38:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="97" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="111"
> queue-time="0" op-digest="789f55f594a2d2a911e2c1b9a41e103d"
> on_node="orana"/>
>             <lrm_rsc_op id="C-FLT_monitor_200"
> operation_key="C-FLT_monitor_200" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="39:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;39:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="108" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="75" queue-time="0"
> op-digest="b9db07a00f66e57a44226b503d43889f" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="C-FLT2" type="IPaddr" class="ocf" provider="pw">
>             <lrm_rsc_op id="C-FLT2_last_0"
> operation_key="C-FLT2_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="40:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;40:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="98" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="113"
> queue-time="0" op-digest="236ab7fb7ace80f0eebd8edf4bd9224b"
> on_node="orana"/>
>             <lrm_rsc_op id="C-FLT2_monitor_200"
> operation_key="C-FLT2_monitor_200" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="41:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;41:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="109" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="89" queue-time="0"
> op-digest="b6c51881a4e04213406b712d950fe34c" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="E-3" type="IPaddr" class="ocf" provider="pw">
>             <lrm_rsc_op id="E-3_last_0" operation_key="E-3_start_0"
> operation="start" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.9"
> transition-key="42:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;42:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="99" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="126"
> queue-time="0" op-digest="f3631c868f7539a7fb24804bfccd48f8"
> on_node="orana"/>
>             <lrm_rsc_op id="E-3_monitor_200"
> operation_key="E-3_monitor_200" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="43:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;43:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="114" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="75" queue-time="0"
> op-digest="a99e7c4d084a719699ad5bd04237e8d2" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="MGMT-FLT" type="IPaddr" class="ocf" provider="pw">
>             <lrm_rsc_op id="MGMT-FLT_last_0"
> operation_key="MGMT-FLT_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="44:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;44:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="100" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="114"
> queue-time="0" op-digest="a78c07f01d90a62d8f139658bc7ac3c9"
> on_node="orana"/>
>             <lrm_rsc_op id="MGMT-FLT_monitor_200"
> operation_key="MGMT-FLT_monitor_200" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="45:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;45:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="110" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="94" queue-time="0"
> op-digest="72c18513de4f6b9105d664dd90d265e8" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="M-FLT" type="IPaddr" class="ocf" provider="pw">
>             <lrm_rsc_op id="M-FLT_last_0"
> operation_key="M-FLT_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="46:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;46:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="101" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="122"
> queue-time="0" op-digest="77947e7e39ef2e8073312201deb6d7e7"
> on_node="orana"/>
>             <lrm_rsc_op id="M-FLT_monitor_200"
> operation_key="M-FLT_monitor_200" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="47:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;47:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="115" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="80" queue-time="0"
> op-digest="a61b31bfa2661c96d41eed752b748f8a" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="M-FLT2" type="IPaddr" class="ocf" provider="pw">
>             <lrm_rsc_op id="M-FLT2_last_0"
> operation_key="M-FLT2_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="48:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;48:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="102" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="112"
> queue-time="0" op-digest="72fcbee579c9168ec7a8b329c0b1e8cc"
> on_node="orana"/>
>             <lrm_rsc_op id="M-FLT2_monitor_200"
> operation_key="M-FLT2_monitor_200" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="49:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;49:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="111" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="89" queue-time="0"
> op-digest="8ee1fce96d23a284b277c3b8fdee8b95" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="S-FLT" type="IPaddr" class="ocf" provider="pw">
>             <lrm_rsc_op id="S-FLT_last_0"
> operation_key="S-FLT_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="50:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;50:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="103" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="123"
> queue-time="0" op-digest="6d479331421af055b010c182b992957c"
> on_node="orana"/>
>             <lrm_rsc_op id="S-FLT_monitor_200"
> operation_key="S-FLT_monitor_200" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="51:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;51:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="116" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="91" queue-time="0"
> op-digest="a88069f185532a1f62424b120846ff71" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="S-FLT2" type="IPaddr" class="ocf" provider="pw">
>             <lrm_rsc_op id="S-FLT2_last_0"
> operation_key="S-FLT2_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="52:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;52:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="104" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="112"
> queue-time="0" op-digest="d0aa64d1a80785f83fca3ed7a7b58527"
> on_node="orana"/>
>             <lrm_rsc_op id="S-FLT2_monitor_200"
> operation_key="S-FLT2_monitor_200" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="53:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;53:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="112" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="92" queue-time="0"
> op-digest="5843ced8789230e395f5cb1f81e27e92" on_node="orana"/>
>           </lrm_resource>
>           <lrm_resource id="fence-uc-orana" type="fence_ilo4" class="stonith">
>             <lrm_rsc_op id="fence-uc-orana_last_0"
> operation_key="fence-uc-orana_start_0" operation="start"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="14:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;14:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="123" rc-code="0" op-status="0" interval="0"
> last-run="1452731302" last-rc-change="1452731302" exec-time="108"
> queue-time="0" op-digest="363cbd5d9ec840db46e5e25e2f8b6652"
> on_node="orana"/>
>             <lrm_rsc_op id="fence-uc-orana_monitor_5000"
> operation_key="fence-uc-orana_monitor_5000" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="12:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;12:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="124" rc-code="0" op-status="0" interval="5000"
> last-rc-change="1452731302" exec-time="94" queue-time="0"
> op-digest="8024392825e4a1e81b7f628e6dee69a6" on_node="orana"/>
>           </lrm_resource>
>         </lrm_resources>
>       </lrm>
>       <transient_attributes id="orana">
>         <instance_attributes id="status-orana">
>           <nvpair id="status-orana-shutdown" name="shutdown" value="0"/>
>           <nvpair id="status-orana-probe_complete"
> name="probe_complete" value="true"/>
>           <nvpair id="status-orana-master-foo" name="master-foo" value="10"/>
>         </instance_attributes>
>       </transient_attributes>
>     </node_state>
>   </status>
> </cib>
>
>
> Attaching full corosync.log from orana.
>
> Mentioning the interesting parts in the log here.
>
> Jan 13 19:32:44 corosync [TOTEM ] A processor joined or left the
> membership and a new membership was formed.
> Jan 13 19:32:44 corosync [QUORUM] Members[2]: 1 2
> Jan 13 19:32:44 corosync [QUORUM] Members[2]: 1 2
> Jan 13 19:32:44 [4296] orana       crmd:     info:
> cman_event_callback: Membership 7044: quorum retained
> Jan 13 19:32:44 [4296] orana       crmd:   notice:
> crm_update_peer_state: cman_event_callback: Node kamet[2] - state is
> now member (was lost)
> Jan 13 19:32:44 [4296] orana       crmd:     info:
> peer_update_callback: kamet is now member (was lost)
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/crmd/2482)
> Jan 13 19:32:44 [4296] orana       crmd:     info: crm_cs_flush: Sent
> 0 CPG messages  (1 remaining, last=122): Try again (6)
> Jan 13 19:32:44 [4291] orana        cib:     info: crm_cs_flush: Sent
> 0 CPG messages  (1 remaining, last=240): Try again (6)
> Jan 13 19:32:44 [4296] orana       crmd:     info:
> cman_event_callback: Membership 7044: quorum retained
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section nodes
> to master (origin=local/crmd/2483)
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/crmd/2484)
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section nodes
> to master (origin=local/crmd/2485)
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/crmd/2486)
> Jan 13 19:32:44 corosync [CPG   ] chosen downlist: sender r(0)
> ip(7.7.7.1) ; members(old:1 left:0)
> Jan 13 19:32:44 corosync [MAIN  ] Completed service synchronization,
> ready to provide service.
> Jan 13 19:32:44 [4291] orana        cib:     info: crm_cs_flush: Sent
> 5 CPG messages  (0 remaining, last=245): OK (1)
> Jan 13 19:32:44 [4296] orana       crmd:     info: crm_cs_flush: Sent
> 2 CPG messages  (0 remaining, last=124): OK (1)
> Jan 13 19:32:44 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.72.7 2
> Jan 13 19:32:44 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.72.8 (null)
> Jan 13 19:32:44 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=8
> Jan 13 19:32:44 [4291] orana        cib:     info: cib_perform_op: +
> /cib/status/node_state[@id='kamet']:
> @crm-debug-origin=peer_update_callback
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/crmd/2482, version=0.72.8)
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section nodes:
> OK (rc=0, origin=orana/crmd/2483, version=0.72.8)
> Jan 13 19:32:44 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.72.8 2
> Jan 13 19:32:44 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.72.9 (null)
> Jan 13 19:32:44 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=9
> Jan 13 19:32:44 [4291] orana        cib:     info: cib_perform_op: +
> /cib/status/node_state[@id='kamet']:  @in_ccm=true,
> @crm-debug-origin=post_cache_update
> Jan 13 19:32:44 [4291] orana        cib:     info: cib_perform_op: +
> /cib/status/node_state[@id='orana']:
> @crm-debug-origin=post_cache_update
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/crmd/2484, version=0.72.9)
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section nodes:
> OK (rc=0, origin=orana/crmd/2485, version=0.72.9)
> Jan 13 19:32:44 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/crmd/2486, version=0.72.9)
> Jan 13 19:32:49 [4291] orana        cib:     info: cib_process_ping:
> Reporting our current digest to orana:
> 9cc493f69fca04421704855b29514829 for 0.72.9 (0x147cad0 0)
> Jan 13 19:32:53 [4296] orana       crmd:     info: crm_timer_popped:
> PEngine Recheck Timer (I_PE_CALC) just popped (30000ms)
> Jan 13 19:32:53 [4296] orana       crmd:   notice:
> do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [
> input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Jan 13 19:32:53 [4296] orana       crmd:     info:
> do_state_transition: Progressed to state S_POLICY_ENGINE after
> C_TIMER_POPPED
> Jan 13 19:32:53 [4295] orana    pengine:   notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 13 19:32:53 [4295] orana    pengine:     info:
> determine_online_status_fencing: - Node kamet is not ready to run
> resources
> Jan 13 19:32:53 [4295] orana    pengine:     info:
> determine_online_status: Node kamet is pending
> Jan 13 19:32:53 [4295] orana    pengine:     info:
> determine_online_status_fencing: Node orana is active
> Jan 13 19:32:53 [4295] orana    pengine:     info:
> determine_online_status: Node orana is online
> Jan 13 19:32:53 [4295] orana    pengine:     info: clone_print:
> Master/Slave Set: foo-master [foo]
> Jan 13 19:32:53 [4295] orana    pengine:     info: short_print:
> Masters: [ orana ]
> Jan 13 19:32:53 [4295] orana    pengine:     info: short_print:
> Stopped: [ kamet ]
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> fence-uc-orana (stonith:fence_ilo4): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> fence-uc-kamet (stonith:fence_ilo4): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> CWS-3G (ocf::pw:IPaddr): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> CWS-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> CWS-FLT2 (ocf::pw:IPaddr): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> EPC-3G (ocf::pw:IPaddr): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> MGMT-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> MME-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> MME-FLT2 (ocf::pw:IPaddr): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> SGW-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_print:
> SGW-FLT2 (ocf::pw:IPaddr): Started orana
> Jan 13 19:32:53 [4295] orana    pengine:     info: native_color:
> Resource foo:1 cannot run anywhere
> Jan 13 19:32:53 [4295] orana    pengine:     info: master_color:
> Promoting foo:0 (Master orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: master_color:
> foo-master: Promoted 1 instances of a possible 1 to master
> Jan 13 19:32:53 [4295] orana    pengine:     info: probe_resources:
> Action probe_complete-kamet on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action foo:0_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action fence-uc-orana_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action fence-uc-kamet_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action CWS-3G_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action CWS-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action CWS-FLT2_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action EPC-3G_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action MGMT-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action MME-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action MME-FLT2_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action SGW-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:  warning: custom_action:
> Action SGW-FLT2_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> foo:0 (Master orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> foo:1 (Stopped)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> fence-uc-orana (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> fence-uc-kamet (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> CWS-3G (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> CWS-FLT (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> CWS-FLT2 (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> EPC-3G (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> MGMT-FLT (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> MME-FLT (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> MME-FLT2 (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> SGW-FLT (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:     info: LogActions: Leave
> SGW-FLT2 (Started orana)
> Jan 13 19:32:53 [4295] orana    pengine:   notice: process_pe_message:
> Calculated Transition 2313:
> /var/lib/pacemaker/pengine/pe-input-1448.bz2
> Jan 13 19:32:53 [4296] orana       crmd:     info:
> do_state_transition: State transition S_POLICY_ENGINE ->
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
> origin=handle_response ]
> Jan 13 19:32:53 [4296] orana       crmd:     info: do_te_invoke:
> Processing graph 2313 (ref=pe_calc-dc-1452731573-2544) derived from
> /var/lib/pacemaker/pengine/pe-input-1448.bz2
> Jan 13 19:32:53 [4296] orana       crmd:   notice: run_graph:
> Transition 2313 (Complete=0, Pending=0, Fired=0, Skipped=0,
> Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1448.bz2):
> Complete
> Jan 13 19:32:53 [4296] orana       crmd:     info: do_log: FSA: Input
> I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
> Jan 13 19:32:53 [4296] orana       crmd:   notice:
> do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [
> input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
> Jan 13 19:32:59 [4292] orana stonith-ng:     info:
> pcmk_cpg_membership: Joined[2.0] stonith-ng.2
> Jan 13 19:32:59 [4292] orana stonith-ng:     info:
> pcmk_cpg_membership: Member[2.0] stonith-ng.1
> Jan 13 19:32:59 [4292] orana stonith-ng:     info:
> pcmk_cpg_membership: Member[2.1] stonith-ng.2
> Jan 13 19:32:59 [4292] orana stonith-ng:     info:
> crm_update_peer_proc: pcmk_cpg_membership: Node kamet[2] -
> corosync-cpg is now online
> Jan 13 19:32:59 [4291] orana        cib:     info:
> pcmk_cpg_membership: Joined[2.0] cib.2
> Jan 13 19:32:59 [4291] orana        cib:     info:
> pcmk_cpg_membership: Member[2.0] cib.1
> Jan 13 19:32:59 [4291] orana        cib:     info:
> pcmk_cpg_membership: Member[2.1] cib.2
> Jan 13 19:32:59 [4291] orana        cib:     info:
> crm_update_peer_proc: pcmk_cpg_membership: Node kamet[2] -
> corosync-cpg is now online
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.72.9 2
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.73.0 (null)
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-cluster-recheck-interval']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-stonith-enabled']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-dc-version']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-cluster-infrastructure']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-last-lrm-refresh']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @epoch=73, @num_updates=0
> Jan 13 19:33:00 [4291] orana        cib:     info:
> cib_process_request: Completed cib_replace operation for section
> configuration: OK (rc=0, origin=kamet/cibadmin/2, version=0.73.0)
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> abort_transition_graph: Transition aborted by deletion of
> nvpair[@id='cib-bootstrap-options-cluster-recheck-interval']:
> Non-status change (cib=0.73.0, source=te_update_diff:383,
> path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']/nvpair[@id='cib-bootstrap-options-cluster-recheck-interval'],
> 1)
> Jan 13 19:33:00 [4296] orana       crmd:   notice:
> do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [
> input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Jan 13 19:33:00 [4291] orana        cib:     info: write_cib_contents:
> Archived previous version as /var/lib/pacemaker/cib/cib-83.raw
> Jan 13 19:33:00 [4291] orana        cib:     info: write_cib_contents:
> Wrote version 0.73.0 of the CIB to disk (digest:
> 3e63c3f9772e422777137db356f7e41d)
> Jan 13 19:33:00 [4291] orana        cib:     info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.K8qdX0
> (digest: /var/lib/pacemaker/cib/cib.3INvqH)
> Jan 13 19:33:00 [4295] orana    pengine:   notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 13 19:33:00 [4295] orana    pengine:     info:
> determine_online_status_fencing: - Node kamet is not ready to run
> resources
> Jan 13 19:33:00 [4295] orana    pengine:     info:
> determine_online_status: Node kamet is pending
> Jan 13 19:33:00 [4295] orana    pengine:     info:
> determine_online_status_fencing: Node orana is active
> Jan 13 19:33:00 [4295] orana    pengine:     info:
> determine_online_status: Node orana is online
> Jan 13 19:33:00 [4295] orana    pengine:     info: clone_print:
> Master/Slave Set: foo-master [foo]
> Jan 13 19:33:00 [4295] orana    pengine:     info: short_print:
> Masters: [ orana ]
> Jan 13 19:33:00 [4295] orana    pengine:     info: short_print:
> Stopped: [ kamet ]
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> fence-uc-orana (stonith:fence_ilo4): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> fence-uc-kamet (stonith:fence_ilo4): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> CWS-3G (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> CWS-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> CWS-FLT2 (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> EPC-3G (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> MGMT-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> MME-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> MME-FLT2 (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> SGW-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> SGW-FLT2 (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_color:
> Resource foo:1 cannot run anywhere
> Jan 13 19:33:00 [4295] orana    pengine:     info: master_color:
> Promoting foo:0 (Master orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: master_color:
> foo-master: Promoted 1 instances of a possible 1 to master
> Jan 13 19:33:00 [4295] orana    pengine:     info: probe_resources:
> Action probe_complete-kamet on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action foo:0_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action fence-uc-orana_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action fence-uc-kamet_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action CWS-3G_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action CWS-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action CWS-FLT2_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action EPC-3G_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action MGMT-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action MME-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action MME-FLT2_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action SGW-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action SGW-FLT2_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> foo:0 (Master orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> foo:1 (Stopped)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> fence-uc-orana (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> fence-uc-kamet (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> CWS-3G (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> CWS-FLT (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> CWS-FLT2 (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> EPC-3G (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> MGMT-FLT (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> MME-FLT (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> MME-FLT2 (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> SGW-FLT (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> SGW-FLT2 (Started orana)
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> do_state_transition: State transition S_POLICY_ENGINE ->
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
> origin=handle_response ]
> Jan 13 19:33:00 [4296] orana       crmd:     info: do_te_invoke:
> Processing graph 2314 (ref=pe_calc-dc-1452731580-2545) derived from
> /var/lib/pacemaker/pengine/pe-input-1449.bz2
> Jan 13 19:33:00 [4296] orana       crmd:   notice: run_graph:
> Transition 2314 (Complete=0, Pending=0, Fired=0, Skipped=0,
> Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1449.bz2):
> Complete
> Jan 13 19:33:00 [4296] orana       crmd:     info: do_log: FSA: Input
> I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
> Jan 13 19:33:00 [4296] orana       crmd:   notice:
> do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [
> input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
> Jan 13 19:33:00 [4295] orana    pengine:   notice: process_pe_message:
> Calculated Transition 2314:
> /var/lib/pacemaker/pengine/pe-input-1449.bz2
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.73.0 2
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.74.0 (null)
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @epoch=74
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']:
>  <nvpair id="cib-bootstrap-options-cluster-recheck-interval"
> name="cluster-recheck-interval" value="30s"/>
> Jan 13 19:33:00 [4291] orana        cib:     info:
> cib_process_request: Completed cib_replace operation for section
> configuration: OK (rc=0, origin=kamet/cibadmin/2, version=0.74.0)
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> abort_transition_graph: Transition aborted by
> cib-bootstrap-options-cluster-recheck-interval,
> cluster-recheck-interval=30s: Non-status change (create cib=0.74.0,
> source=te_update_diff:383,
> path=/cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options'],
> 1)
> Jan 13 19:33:00 [4296] orana       crmd:   notice:
> do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [
> input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Jan 13 19:33:00 [4291] orana        cib:     info: write_cib_contents:
> Archived previous version as /var/lib/pacemaker/cib/cib-84.raw
> Jan 13 19:33:00 [4291] orana        cib:     info: write_cib_contents:
> Wrote version 0.74.0 of the CIB to disk (digest:
> ac133341560d33343b2410d8833adfb8)
> Jan 13 19:33:00 [4291] orana        cib:     info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.yXjrc1
> (digest: /var/lib/pacemaker/cib/cib.zRGbVH)
> Jan 13 19:33:00 [4295] orana    pengine:   notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 13 19:33:00 [4295] orana    pengine:     info:
> determine_online_status_fencing: - Node kamet is not ready to run
> resources
> Jan 13 19:33:00 [4295] orana    pengine:     info:
> determine_online_status: Node kamet is pending
> Jan 13 19:33:00 [4295] orana    pengine:     info:
> determine_online_status_fencing: Node orana is active
> Jan 13 19:33:00 [4295] orana    pengine:     info:
> determine_online_status: Node orana is online
> Jan 13 19:33:00 [4295] orana    pengine:     info: clone_print:
> Master/Slave Set: foo-master [foo]
> Jan 13 19:33:00 [4295] orana    pengine:     info: short_print:
> Masters: [ orana ]
> Jan 13 19:33:00 [4295] orana    pengine:     info: short_print:
> Stopped: [ kamet ]
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> fence-uc-orana (stonith:fence_ilo4): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> fence-uc-kamet (stonith:fence_ilo4): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> CWS-3G (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> CWS-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> CWS-FLT2 (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> EPC-3G (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> MGMT-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> MME-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> MME-FLT2 (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> SGW-FLT (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_print:
> SGW-FLT2 (ocf::pw:IPaddr): Started orana
> Jan 13 19:33:00 [4295] orana    pengine:     info: native_color:
> Resource foo:1 cannot run anywhere
> Jan 13 19:33:00 [4295] orana    pengine:     info: master_color:
> Promoting foo:0 (Master orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: master_color:
> foo-master: Promoted 1 instances of a possible 1 to master
> Jan 13 19:33:00 [4295] orana    pengine:     info: probe_resources:
> Action probe_complete-kamet on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action foo:0_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action fence-uc-orana_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action fence-uc-kamet_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action CWS-3G_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action CWS-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action CWS-FLT2_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action EPC-3G_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action MGMT-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action MME-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action MME-FLT2_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action SGW-FLT_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:  warning: custom_action:
> Action SGW-FLT2_monitor_0 on kamet is unrunnable (pending)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> foo:0 (Master orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> foo:1 (Stopped)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> fence-uc-orana (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> fence-uc-kamet (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> CWS-3G (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> CWS-FLT (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> CWS-FLT2 (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> EPC-3G (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> MGMT-FLT (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> MME-FLT (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> MME-FLT2 (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> SGW-FLT (Started orana)
> Jan 13 19:33:00 [4295] orana    pengine:     info: LogActions: Leave
> SGW-FLT2 (Started orana)
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> do_state_transition: State transition S_POLICY_ENGINE ->
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
> origin=handle_response ]
> Jan 13 19:33:00 [4296] orana       crmd:     info: do_te_invoke:
> Processing graph 2315 (ref=pe_calc-dc-1452731580-2546) derived from
> /var/lib/pacemaker/pengine/pe-input-1450.bz2
> Jan 13 19:33:00 [4296] orana       crmd:   notice: run_graph:
> Transition 2315 (Complete=0, Pending=0, Fired=0, Skipped=0,
> Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1450.bz2):
> Complete
> Jan 13 19:33:00 [4296] orana       crmd:     info: do_log: FSA: Input
> I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
> Jan 13 19:33:00 [4296] orana       crmd:   notice:
> do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [
> input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
> Jan 13 19:33:00 [4295] orana    pengine:   notice: process_pe_message:
> Calculated Transition 2315:
> /var/lib/pacemaker/pengine/pe-input-1450.bz2
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> pcmk_cpg_membership: Joined[2.0] crmd.2
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> pcmk_cpg_membership: Member[2.0] crmd.1
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> pcmk_cpg_membership: Member[2.1] crmd.2
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> crm_update_peer_proc: pcmk_cpg_membership: Node kamet[2] -
> corosync-cpg is now online
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> peer_update_callback: Client kamet/peer now has status [online]
> (DC=true, changed=4000000)
> Jan 13 19:33:00 [4296] orana       crmd:   notice:
> do_state_transition: State transition S_IDLE -> S_INTEGRATION [
> input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=peer_update_callback ]
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> do_dc_join_offer_one: An unknown node joined - (re-)offer to any
> unconfirmed nodes
> Jan 13 19:33:00 [4296] orana       crmd:     info: join_make_offer:
> Making join offers based on membership 7044
> Jan 13 19:33:00 [4296] orana       crmd:     info: join_make_offer:
> join-2: Sending offer to kamet
> Jan 13 19:33:00 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/crmd/2492)
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> crm_update_peer_join: join_make_offer: Node kamet[2] - join-2 phase 0
> -> 1
> Jan 13 19:33:00 [4296] orana       crmd:     info: join_make_offer:
> Skipping orana: already known 4
> Jan 13 19:33:00 [4296] orana       crmd:     info:
> abort_transition_graph: Transition aborted: Peer Halt
> (source=do_te_invoke:158, 1)
> Jan 13 19:33:00 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section nodes:
> OK (rc=0, origin=kamet/crmd/3, version=0.74.0)
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.74.0 2
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.74.1 (null)
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=1
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: +
> /cib/status/node_state[@id='kamet']:  @crmd=online,
> @crm-debug-origin=peer_update_callback
> Jan 13 19:33:00 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/crmd/2492, version=0.74.1)
> Jan 13 19:33:00 [4291] orana        cib:     info:
> cib_process_replace: Replacement 0.4.0 from kamet not applied to
> 0.74.1: current epoch is greater than the replacement
> Jan 13 19:33:00 [4291] orana        cib:  warning:
> cib_process_request: Completed cib_replace operation for section
> 'all': Update was older than existing configuration (rc=-205,
> origin=kamet/cibadmin/2, version=0.74.1)
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.74.1 2
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.75.0 (null)
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/nodes/node[@id='kamet']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/nodes/node[@id='orana']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='fence-uc-orana']/meta_attributes[@id='fence-uc-orana-meta_attributes']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='fence-uc-kamet']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='CWS-3G']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='CWS-FLT']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='CWS-FLT2']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='EPC-3G']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='MGMT-FLT']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='MME-FLT']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='MME-FLT2']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='SGW-FLT']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/resources/primitive[@id='SGW-FLT2']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-CWS-3G-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-CWS-3G-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-CWS-FLT-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-CWS-FLT-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-CWS-FLT2-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-CWS-FLT2-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-EPC-3G-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-EPC-3G-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-MGMT-FLT-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-MGMT-FLT-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-MME-FLT-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-MME-FLT-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-MME-FLT2-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-MME-FLT2-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-SGW-FLT-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-SGW-FLT-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-SGW-FLT2-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-SGW-FLT2-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-fence-uc-orana-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_colocation[@id='colocation-fence-uc-kamet-foo-master-INFINITY']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-fence-uc-kamet-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
> /cib/configuration/constraints/rsc_order[@id='order-fence-uc-orana-foo-master-mandatory']
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @epoch=75, @num_updates=0
> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: +
> /cib/configuration/resources/primitive[@id='fence-uc-orana']/instance_attributes[@id='fence-uc-orana-instance_attributes']/nvpair[@id='fence-uc-orana-instance_attributes-delay']:
>  @value=0
> Jan 13 19:33:00 [4291] orana        cib:     info:
> cib_process_request: Completed cib_replace operation for section
> configuration: OK (rc=0, origin=kamet/cibadmin/2, version=0.75.0)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_remove: Device 'CWS-3G' not found (2 active devices)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_remove: Device 'CWS-FLT' not found (2 active devices)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_remove: Device 'CWS-FLT2' not found (2 active devices)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_remove: Device 'EPC-3G' not found (2 active devices)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_remove: Device 'MGMT-FLT' not found (2 active devices)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_remove: Device 'MME-FLT' not found (2 active devices)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_remove: Device 'MME-FLT2' not found (2 active devices)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_remove: Device 'SGW-FLT' not found (2 active devices)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_remove: Device 'SGW-FLT2' not found (2 active devices)
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> update_cib_stonith_devices_v2: Updating device list from the cib:
> modify nvpair[@id='fence-uc-orana-instance_attributes-delay']
> Jan 13 19:33:00 [4292] orana stonith-ng:     info: cib_devices_update:
> Updating devices to version 0.75.0
> Jan 13 19:33:00 [4292] orana stonith-ng:   notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 13 19:33:00 [4292] orana stonith-ng:     info: unpack_nodes:
> Creating a fake local node
> Jan 13 19:33:00 [4292] orana stonith-ng:     info:
> stonith_device_register: Overwriting an existing entry for
> fence-uc-orana from the cib
> Jan 13 19:33:00 [4292] orana stonith-ng:   notice:
> stonith_device_register: Added 'fence-uc-orana' to the device list (2
> active devices)
> Jan 13 19:33:00 [4291] orana        cib:     info: write_cib_contents:
> Archived previous version as /var/lib/pacemaker/cib/cib-85.raw
> Jan 13 19:33:00 [4291] orana        cib:     info: write_cib_contents:
> Wrote version 0.75.0 of the CIB to disk (digest:
> 4fb4a8ef2f8cde3a07fb30eb706e7e9c)
> Jan 13 19:33:00 [4291] orana        cib:     info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.HyXAc3
> (digest: /var/lib/pacemaker/cib/cib.lOa3UL)
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.75.0 2
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.76.0 (null)
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @epoch=76
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/configuration/constraints:  <rsc_colocation
> id="colocation-fence-uc-orana-foo-master-INFINITY"
> rsc="fence-uc-orana" score="INFINITY" with-rsc="foo-master"/>
> Jan 13 19:33:01 [4291] orana        cib:     info:
> cib_process_request: Completed cib_replace operation for section
> configuration: OK (rc=0, origin=kamet/cibadmin/2, version=0.76.0)
> Jan 13 19:33:01 [4291] orana        cib:     info: write_cib_contents:
> Archived previous version as /var/lib/pacemaker/cib/cib-86.raw
> Jan 13 19:33:01 [4291] orana        cib:     info: write_cib_contents:
> Wrote version 0.76.0 of the CIB to disk (digest:
> df07ff6cbef5a35891d43b89b9ba4371)
> Jan 13 19:33:01 [4291] orana        cib:     info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.Pi8ov3
> (digest: /var/lib/pacemaker/cib/cib.kYeIwM)
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.76.0 2
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.77.0 (null)
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @epoch=77
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/configuration/resources:  <primitive class="stonith"
> id="fence-uc-kamet" type="fence_ilo4"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                 <instance_attributes
> id="fence-uc-kamet-instance_attributes">
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                   <nvpair
> id="fence-uc-kamet-instance_attributes-login" name="login"
> value="parallel"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                   <nvpair
> id="fence-uc-kamet-instance_attributes-passwd" name="passwd"
> value="wireless"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                   <nvpair
> id="fence-uc-kamet-instance_attributes-ipaddr" name="ipaddr"
> value="10.11.10.21"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                   <nvpair
> id="fence-uc-kamet-instance_attributes-pcmk_host_list"
> name="pcmk_host_list" value="kamet"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                   <nvpair
> id="fence-uc-kamet-instance_attributes-action" name="action"
> value="off"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                   <nvpair
> id="fence-uc-kamet-instance_attributes-lanplus" name="lanplus"
> value="1"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                   <nvpair
> id="fence-uc-kamet-instance_attributes-delay" name="delay"
> value="10"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                 </instance_attributes>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                 <operations>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                   <op
> id="fence-uc-kamet-start-interval-0s" interval="0s" name="start"
> on-fail="restart"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                   <op
> id="fence-uc-kamet-monitor-interval-5s" interval="5s" name="monitor"
> on-fail="restart"/>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                                 </operations>
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
>                               </primitive>
> Jan 13 19:33:01 [4291] orana        cib:     info:
> cib_process_request: Completed cib_replace operation for section
> configuration: OK (rc=0, origin=kamet/cibadmin/2, version=0.77.0)
> Jan 13 19:33:01 [4292] orana stonith-ng:     info:
> update_cib_stonith_devices_v2: Updating device list from the cib:
> create resources
> Jan 13 19:33:01 [4292] orana stonith-ng:     info: cib_devices_update:
> Updating devices to version 0.77.0
> Jan 13 19:33:01 [4292] orana stonith-ng:   notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 13 19:33:01 [4292] orana stonith-ng:     info: unpack_nodes:
> Creating a fake local node
> Jan 13 19:33:01 [4292] orana stonith-ng:     info:
> stonith_device_register: Overwriting an existing entry for
> fence-uc-kamet from the cib
> Jan 13 19:33:01 [4292] orana stonith-ng:   notice:
> stonith_device_register: Added 'fence-uc-kamet' to the device list (2
> active devices)
> Jan 13 19:33:01 [4291] orana        cib:     info: write_cib_contents:
> Archived previous version as /var/lib/pacemaker/cib/cib-87.raw
> Jan 13 19:33:01 [4291] orana        cib:     info: write_cib_contents:
> Wrote version 0.77.0 of the CIB to disk (digest:
> 0fef989ac2484c07ebc724551f42a236)
> Jan 13 19:33:01 [4291] orana        cib:     info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.Nchv43
> (digest: /var/lib/pacemaker/cib/cib.oYETEN)
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.77.0 2
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.78.0 (null)
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @epoch=78
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/configuration/constraints:  <rsc_colocation
> id="colocation-fence-uc-kamet-foo-master-INFINITY"
> rsc="fence-uc-kamet" score="INFINITY" with-rsc="foo-master"/>
> Jan 13 19:33:01 [4291] orana        cib:     info:
> cib_process_request: Completed cib_replace operation for section
> configuration: OK (rc=0, origin=kamet/cibadmin/2, version=0.78.0)
> Jan 13 19:33:01 [4291] orana        cib:     info: write_cib_contents:
> Archived previous version as /var/lib/pacemaker/cib/cib-88.raw
> Jan 13 19:33:01 [4291] orana        cib:     info: write_cib_contents:
> Wrote version 0.78.0 of the CIB to disk (digest:
> 1381a968f3bd56162444466add6290cd)
> Jan 13 19:33:01 [4291] orana        cib:     info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.eWlhn4
> (digest: /var/lib/pacemaker/cib/cib.PRhpgO)
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.78.0 2
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.0 (null)
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @epoch=79
> Jan 13 19:33:01 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/configuration/crm_config/cluster_property_set[@id='cib-bootstrap-options']:
>  <nvpair id="cib-bootstrap-options-stonith-enabled"
> name="stonith-enabled" value="true"/>
> Jan 13 19:33:01 [4291] orana        cib:     info:
> cib_process_request: Completed cib_replace operation for section
> configuration: OK (rc=0, origin=kamet/cibadmin/2, version=0.79.0)
> Jan 13 19:33:01 [4291] orana        cib:     info: write_cib_contents:
> Archived previous version as /var/lib/pacemaker/cib/cib-89.raw
> Jan 13 19:33:01 [4291] orana        cib:     info: write_cib_contents:
> Wrote version 0.79.0 of the CIB to disk (digest:
> fba8e35c75c1ba34ae39845213394cb6)
> Jan 13 19:33:01 [4291] orana        cib:     info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.UQzsJ4
> (digest: /var/lib/pacemaker/cib/cib.GINNYO)
> Jan 13 19:33:01 [4296] orana       crmd:     info:
> do_dc_join_offer_one: join-2: Processing join_announce request from
> kamet in state S_INTEGRATION
> Jan 13 19:33:01 [4296] orana       crmd:     info:
> crm_update_peer_join: do_dc_join_offer_one: Node kamet[2] - join-2
> phase 1 -> 0
> Jan 13 19:33:01 [4296] orana       crmd:     info: join_make_offer:
> join-2: Sending offer to kamet
> Jan 13 19:33:01 [4296] orana       crmd:     info:
> crm_update_peer_join: join_make_offer: Node kamet[2] - join-2 phase 0
> -> 1
> Jan 13 19:33:01 [4296] orana       crmd:     info:
> crm_update_peer_join: join_make_offer: Node orana[1] - join-2 phase 4
> -> 0
> Jan 13 19:33:01 [4296] orana       crmd:     info: join_make_offer:
> join-2: Sending offer to orana
> Jan 13 19:33:01 [4296] orana       crmd:     info:
> crm_update_peer_join: join_make_offer: Node orana[1] - join-2 phase 0
> -> 1
> Jan 13 19:33:01 [4296] orana       crmd:     info:
> abort_transition_graph: Transition aborted: Node join
> (source=do_dc_join_offer_one:236, 1)
> Jan 13 19:33:01 [4296] orana       crmd:     info:
> crm_update_peer_join: do_dc_join_filter_offer: Node orana[1] - join-2
> phase 1 -> 2
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_join: do_dc_join_filter_offer: Node kamet[2] - join-2
> phase 1 -> 2
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_expected: do_dc_join_filter_offer: Node kamet[2] -
> expected state is now member (was down)
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN
> [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crmd_join_phase_log: join-2: kamet=integrated
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crmd_join_phase_log: join-2: orana=integrated
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> do_dc_join_finalize: join-2: Syncing our CIB to the rest of the
> cluster
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> do_state_transition: State transition S_FINALIZE_JOIN -> S_INTEGRATION
> [ input=I_JOIN_REQUEST cause=C_HA_MESSAGE origin=route_message ]
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> do_dc_join_offer_one: join-2: Processing join_request request from
> kamet in state S_INTEGRATION
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_join: do_dc_join_offer_one: Node kamet[2] - join-2
> phase 2 -> 0
> Jan 13 19:33:02 [4296] orana       crmd:     info: join_make_offer:
> join-2: Sending offer to kamet
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_join: join_make_offer: Node kamet[2] - join-2 phase 0
> -> 1
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_join: join_make_offer: Node orana[1] - join-2 phase 2
> -> 0
> Jan 13 19:33:02 [4296] orana       crmd:     info: join_make_offer:
> join-2: Sending offer to orana
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_join: join_make_offer: Node orana[1] - join-2 phase 0
> -> 1
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> abort_transition_graph: Transition aborted: Node join
> (source=do_dc_join_offer_one:236, 1)
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_join: do_dc_join_filter_offer: Node kamet[2] - join-2
> phase 1 -> 2
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_join: do_dc_join_filter_offer: Node orana[1] - join-2
> phase 1 -> 2
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN
> [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crmd_join_phase_log: join-2: kamet=integrated
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crmd_join_phase_log: join-2: orana=integrated
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> do_dc_join_finalize: join-2: Syncing our CIB to the rest of the
> cluster
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_replace: Digest matched on replace from orana:
> 1b71d8dd45f16ee42d3e7ee539ee89e0
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_replace: Replaced 0.79.0 with 0.79.0 from orana
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Completed cib_replace operation for section
> 'all': OK (rc=0, origin=orana/crmd/2495, version=0.79.0)
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_join: finalize_join_for: Node kamet[2] - join-2 phase
> 2 -> 3
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> crm_update_peer_join: finalize_join_for: Node orana[1] - join-2 phase
> 2 -> 3
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_replace: Digest matched on replace from orana:
> 1b71d8dd45f16ee42d3e7ee539ee89e0
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_replace: Replaced 0.79.0 with 0.79.0 from orana
> Jan 13 19:33:02 [4291] orana        cib:     info: write_cib_contents:
> Archived previous version as /var/lib/pacemaker/cib/cib-90.raw
> Jan 13 19:33:02 [4291] orana        cib:     info: write_cib_contents:
> Wrote version 0.79.0 of the CIB to disk (digest:
> 0b3b25d4809db68d92bb5b905e3a4194)
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Completed cib_replace operation for section
> 'all': OK (rc=0, origin=orana/crmd/2497, version=0.79.0)
> Jan 13 19:33:02 [4291] orana        cib:     info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.GELr89
> (digest: /var/lib/pacemaker/cib/cib.EElPMZ)
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='kamet']/transient_attributes: OK (rc=0,
> origin=kamet/crmd/10, version=0.79.0)
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section nodes
> to master (origin=local/crmd/2498)
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section nodes
> to master (origin=local/crmd/2499)
> Jan 13 19:33:02 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.0 2
> Jan 13 19:33:02 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.1 (null)
> Jan 13 19:33:02 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=1
> Jan 13 19:33:02 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/configuration/nodes:  <node id="kamet" uname="kamet"/>
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section nodes:
> OK (rc=0, origin=orana/crmd/2498, version=0.79.1)
> Jan 13 19:33:02 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.1 2
> Jan 13 19:33:02 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.2 (null)
> Jan 13 19:33:02 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=2
> Jan 13 19:33:02 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/configuration/nodes:  <node id="orana" uname="orana"/>
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section nodes:
> OK (rc=0, origin=orana/crmd/2499, version=0.79.2)
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/attrd/113)
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/attrd/113, version=0.79.2)
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/attrd/115)
> Jan 13 19:33:02 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/attrd/115, version=0.79.2)
> Jan 13 19:33:02 [4291] orana        cib:     info: write_cib_contents:
> Archived previous version as /var/lib/pacemaker/cib/cib-91.raw
> Jan 13 19:33:02 [4291] orana        cib:     info: write_cib_contents:
> Wrote version 0.79.0 of the CIB to disk (digest:
> e60d7f0838319a2b10b16a2ed5c7920e)
> Jan 13 19:33:02 [4291] orana        cib:     info: retrieveCib:
> Reading cluster configuration from: /var/lib/pacemaker/cib/cib.0uYLba
> (digest: /var/lib/pacemaker/cib/cib.6lSoTZ)
> Jan 13 19:33:02 [4296] orana       crmd:     info:
> services_os_action_execute: Managed IPaddr_meta-data_0 process 6178
> exited with rc=0
> Jan 13 19:33:03 [4296] orana       crmd:     info:
> crm_update_peer_join: do_dc_join_ack: Node kamet[2] - join-2 phase 3
> -> 4
> Jan 13 19:33:03 [4296] orana       crmd:     info: do_dc_join_ack:
> join-2: Updating node state to member for kamet
> Jan 13 19:33:03 [4296] orana       crmd:     info: erase_status_tag:
> Deleting xpath: //node_state[@uname='kamet']/lrm
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_delete operation for section
> //node_state[@uname='kamet']/lrm to master (origin=local/crmd/2500)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/crmd/2501)
> Jan 13 19:33:03 [4296] orana       crmd:     info:
> crm_update_peer_join: do_dc_join_ack: Node orana[1] - join-2 phase 3
> -> 4
> Jan 13 19:33:03 [4296] orana       crmd:     info: do_dc_join_ack:
> join-2: Updating node state to member for orana
> Jan 13 19:33:03 [4296] orana       crmd:     info: erase_status_tag:
> Deleting xpath: //node_state[@uname='orana']/lrm
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='kamet']/lrm: OK (rc=0, origin=orana/crmd/2500,
> version=0.79.2)
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.2 2
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.3 (null)
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=3
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: +
> /cib/status/node_state[@id='kamet']:
> @crm-debug-origin=do_lrm_query_internal, @join=member,
> @expected=member
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/status/node_state[@id='kamet']:  <lrm id="kamet"/>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                        <lrm_resources/>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                      </lrm>
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/crmd/2501, version=0.79.3)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_delete operation for section
> //node_state[@uname='orana']/lrm to master (origin=local/crmd/2502)
> Jan 13 19:33:03 [4296] orana       crmd:     info:
> do_state_transition: State transition S_FINALIZE_JOIN ->
> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL
> origin=check_join_state ]
> Jan 13 19:33:03 [4296] orana       crmd:     info:
> abort_transition_graph: Transition aborted: Peer Cancelled
> (source=do_te_invoke:151, 1)
> Jan 13 19:33:03 [4294] orana      attrd:   notice:
> attrd_local_callback: Sending full refresh (origin=crmd)
> Jan 13 19:33:03 [4294] orana      attrd:   notice:
> attrd_trigger_update: Sending flush op to all hosts for: master-foo
> (10)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/crmd/2503)
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.3 2
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.4 (null)
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: --
> /cib/status/node_state[@id='orana']/lrm[@id='orana']
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=4
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_delete operation for section
> //node_state[@uname='orana']/lrm: OK (rc=0, origin=orana/crmd/2502,
> version=0.79.4)
> Jan 13 19:33:03 [4296] orana       crmd:     info:
> abort_transition_graph: Transition aborted by deletion of
> lrm[@id='orana']: Resource state removal (cib=0.79.4,
> source=te_update_diff:429,
> path=/cib/status/node_state[@id='orana']/lrm[@id='orana'], 1)
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.4 2
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.5 (null)
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=5
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: +
> /cib/status/node_state[@id='orana']:
> @crm-debug-origin=do_lrm_query_internal
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/status/node_state[@id='orana']:  <lrm id="orana"/>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                        <lrm_resources>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="MGMT-FLT"
> type="IPaddr" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="MGMT-FLT_last_0" operation_key="MGMT-FLT_start_0"
> operation="start" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="44:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;44:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="100" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="114"
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="MGMT-FLT_monitor_200" operation_key="MGMT-FLT_monitor_200"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="45:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;45:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="110" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="94" queue-tim
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="SGW-FLT2"
> type="IPaddr" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="SGW-FLT2_last_0" operation_key="SGW-FLT2_start_0"
> operation="start" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="52:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;52:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="104" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="112"
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="SGW-FLT2_monitor_200" operation_key="SGW-FLT2_monitor_200"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="53:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;53:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="112" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="92" queue-tim
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="EPC-3G"
> type="IPaddr" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="EPC-3G_last_0" operation_key="EPC-3G_start_0" operation="start"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.9"
> transition-key="42:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;42:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="99" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="126" queu
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="EPC-3G_monitor_200" operation_key="EPC-3G_monitor_200"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="43:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;43:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="114" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="75" queue-time="0
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="CWS-3G"
> type="IPaddr" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="CWS-3G_last_0" operation_key="CWS-3G_start_0" operation="start"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.9"
> transition-key="36:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;36:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="96" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="123" queu
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="CWS-3G_monitor_200" operation_key="CWS-3G_monitor_200"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="37:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;37:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="113" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="93" queue-time="0
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource
> id="fence-uc-kamet" type="fence_ilo4" class="stonith">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="fence-uc-kamet_last_0" operation_key="fence-uc-kamet_start_0"
> operation="start" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="13:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;13:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="131" rc-code="0" op-status="0" interval="0"
> last-run="1452731302" last-rc-change="1452731302" exe
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="fence-uc-kamet_monitor_5000"
> operation_key="fence-uc-kamet_monitor_5000" operation="monitor"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.9"
> transition-key="1:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;1:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="132" rc-code="0" op-status="0" interval="5000"
> last-rc-change="1452731303" exec-time="
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="SGW-FLT"
> type="IPaddr" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="SGW-FLT_last_0" operation_key="SGW-FLT_start_0" operation="start"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.9"
> transition-key="50:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;50:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="103" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="123" q
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="SGW-FLT_monitor_200" operation_key="SGW-FLT_monitor_200"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="51:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;51:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="116" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="91" queue-time=
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="foo"
> type="uc" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op id="foo_last_0"
> operation_key="foo_promote_0" operation="promote"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.9"
> transition-key="19:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;19:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="133" rc-code="0" op-status="0" interval="0"
> last-run="1452731303" last-rc-change="1452731303" exec-time="
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="foo_monitor_10000" operation_key="foo_monitor_10000"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="20:2304:8:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:8;20:2304:8:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="135" rc-code="8" op-status="0" interval="10000"
> last-rc-change="1452731303" exec-time="39" que
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="MME-FLT"
> type="IPaddr" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="MME-FLT_last_0" operation_key="MME-FLT_start_0" operation="start"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.9"
> transition-key="46:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;46:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="101" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="122" q
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="MME-FLT_monitor_200" operation_key="MME-FLT_monitor_200"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="47:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;47:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="115" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="80" queue-time=
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource
> id="fence-uc-orana" type="fence_ilo4" class="stonith">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="fence-uc-orana_last_0" operation_key="fence-uc-orana_start_0"
> operation="start" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="14:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;14:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="123" rc-code="0" op-status="0" interval="0"
> last-run="1452731302" last-rc-change="1452731302" exe
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="fence-uc-orana_monitor_5000"
> operation_key="fence-uc-orana_monitor_5000" operation="monitor"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.9"
> transition-key="12:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;12:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="124" rc-code="0" op-status="0" interval="5000"
> last-rc-change="1452731302" exec-time
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="CWS-FLT"
> type="IPaddr" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="CWS-FLT_last_0" operation_key="CWS-FLT_start_0" operation="start"
> crm-debug-origin="build_active_RAs" crm_feature_set="3.0.9"
> transition-key="38:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;38:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="97" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="111" qu
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="CWS-FLT_monitor_200" operation_key="CWS-FLT_monitor_200"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="39:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;39:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="108" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="75" queue-time=
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="CWS-FLT2"
> type="IPaddr" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="CWS-FLT2_last_0" operation_key="CWS-FLT2_start_0"
> operation="start" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="40:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;40:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="98" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="113"
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="CWS-FLT2_monitor_200" operation_key="CWS-FLT2_monitor_200"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="41:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;41:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="109" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="89" queue-tim
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <lrm_resource id="MME-FLT2"
> type="IPaddr" class="ocf" provider="pw">
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="MME-FLT2_last_0" operation_key="MME-FLT2_start_0"
> operation="start" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="48:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;48:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="102" rc-code="0" op-status="0" interval="0"
> last-run="1452731294" last-rc-change="1452731294" exec-time="112"
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                            <lrm_rsc_op
> id="MME-FLT2_monitor_200" operation_key="MME-FLT2_monitor_200"
> operation="monitor" crm-debug-origin="build_active_RAs"
> crm_feature_set="3.0.9"
> transition-key="49:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:0;49:2301:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="111" rc-code="0" op-status="0" interval="200"
> last-rc-change="1452731295" exec-time="89" queue-tim
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                          </lrm_resource>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                        </lrm_resources>
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: ++
>                                      </lrm>
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/crmd/2503, version=0.79.5)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section nodes
> to master (origin=local/crmd/2504)
> Jan 13 19:33:03 [4296] orana       crmd:     info:
> abort_transition_graph: Transition aborted: LRM Refresh
> (source=process_resource_updates:266, 1)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/crmd/2505)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section cib
> to master (origin=local/crmd/2506)
> Jan 13 19:33:03 [4294] orana      attrd:   notice:
> attrd_trigger_update: Sending flush op to all hosts for: shutdown (0)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section nodes:
> OK (rc=0, origin=orana/crmd/2504, version=0.79.5)
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.5 2
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.6 (null)
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=6
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: +
> /cib/status/node_state[@id='kamet']:
> @crm-debug-origin=do_state_transition
> Jan 13 19:33:03 [4291] orana        cib:     info: cib_perform_op: +
> /cib/status/node_state[@id='orana']:
> @crm-debug-origin=do_state_transition
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/crmd/2505, version=0.79.6)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section cib:
> OK (rc=0, origin=orana/crmd/2506, version=0.79.6)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/attrd/117)
> Jan 13 19:33:03 [4294] orana      attrd:   notice:
> attrd_trigger_update: Sending flush op to all hosts for:
> probe_complete (true)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/attrd/117, version=0.79.6)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/attrd/119)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/attrd/119, version=0.79.6)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/attrd/121)
> Jan 13 19:33:03 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/attrd/121, version=0.79.6)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> determine_online_status_fencing: Node kamet is active
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> determine_online_status: Node kamet is online
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> determine_online_status_fencing: Node orana is active
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> determine_online_status: Node orana is online
> Jan 13 19:33:03 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan MGMT-FLT running on orana
> Jan 13 19:33:03 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan SGW-FLT2 running on orana
> Jan 13 19:33:03 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan EPC-3G running on orana
> Jan 13 19:33:03 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan CWS-3G running on orana
> Jan 13 19:33:03 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan SGW-FLT running on orana
> Jan 13 19:33:03 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan MME-FLT running on orana
> Jan 13 19:33:03 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan CWS-FLT running on orana
> Jan 13 19:33:03 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan CWS-FLT2 running on orana
> Jan 13 19:33:03 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan MME-FLT2 running on orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: clone_print:
> Master/Slave Set: foo-master [foo]
> Jan 13 19:33:03 [4295] orana    pengine:     info: short_print:
> Masters: [ orana ]
> Jan 13 19:33:03 [4295] orana    pengine:     info: short_print:
> Stopped: [ kamet ]
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> fence-uc-orana (stonith:fence_ilo4): Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> fence-uc-kamet (stonith:fence_ilo4): Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> MGMT-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> SGW-FLT2 (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> EPC-3G (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> CWS-3G (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> SGW-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> MME-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> CWS-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> CWS-FLT2 (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_print:
> MME-FLT2 (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:03 [4295] orana    pengine:   notice: DeleteRsc: Removing
> MGMT-FLT from orana
> Jan 13 19:33:03 [4295] orana    pengine:   notice: DeleteRsc: Removing
> SGW-FLT2 from orana
> Jan 13 19:33:03 [4295] orana    pengine:   notice: DeleteRsc: Removing
> EPC-3G from orana
> Jan 13 19:33:03 [4295] orana    pengine:   notice: DeleteRsc: Removing
> CWS-3G from orana
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="kamet" lanplus="1" action="off"
> delay="10" ipaddr="10.11.10.21"/>
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-kamet_start_0 on orana
> changed: was 93013a04ac64e00066a4f7b080b31515 vs. now
> a204a24f6a125ccea22f6dc681d020f0 (reload:3.0.9)
> 0:0;13:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="kamet" lanplus="1" action="off"
> delay="10" ipaddr="10.11.10.21" CRM_meta_timeout="20000"/>
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-kamet_monitor_5000 on
> orana changed: was e884a8e46b98e8563278b8117eb4db82 vs. now
> 948f08ea79b4152d6055da386ebacbb3 (reload:3.0.9)
> 0:0;1:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:03 [4295] orana    pengine:   notice: DeleteRsc: Removing
> SGW-FLT from orana
> Jan 13 19:33:03 [4295] orana    pengine:   notice: DeleteRsc: Removing
> MME-FLT from orana
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="orana" lanplus="1" action="off"
> delay="0" ipaddr="10.11.10.30"/>
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-orana_start_0 on orana
> changed: was 363cbd5d9ec840db46e5e25e2f8b6652 vs. now
> 44b3f881263f198e021240ff36365f0f (reload:3.0.9)
> 0:0;14:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="orana" lanplus="1" action="off"
> delay="0" ipaddr="10.11.10.30" CRM_meta_timeout="20000"/>
> Jan 13 19:33:03 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-orana_monitor_5000 on
> orana changed: was 8024392825e4a1e81b7f628e6dee69a6 vs. now
> ec6a353c4e45a74e3497360e0581b40c (reload:3.0.9)
> 0:0;12:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:03 [4295] orana    pengine:   notice: DeleteRsc: Removing
> CWS-FLT from orana
> Jan 13 19:33:03 [4295] orana    pengine:   notice: DeleteRsc: Removing
> CWS-FLT2 from orana
> Jan 13 19:33:03 [4295] orana    pengine:   notice: DeleteRsc: Removing
> MME-FLT2 from orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: master_color:
> Promoting foo:0 (Master orana)
> Jan 13 19:33:03 [4295] orana    pengine:     info: master_color:
> foo-master: Promoted 1 instances of a possible 1 to master
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource MGMT-FLT
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource SGW-FLT2
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource EPC-3G
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource CWS-3G
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource SGW-FLT
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource MME-FLT
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource CWS-FLT
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource CWS-FLT2
> Jan 13 19:33:03 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource MME-FLT2
> Jan 13 19:33:03 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (11s) for foo:1 on kamet
> Jan 13 19:33:03 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (11s) for foo:1 on kamet
> Jan 13 19:33:03 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (5s) for fence-uc-orana on orana
> Jan 13 19:33:03 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (5s) for fence-uc-kamet on kamet
> Jan 13 19:33:03 [4295] orana    pengine:     info: LogActions: Leave
> foo:0 (Master orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Start
> foo:1 (kamet)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Restart
> fence-uc-orana (Started orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Move
> fence-uc-kamet (Started orana -> kamet)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Stop
> MGMT-FLT (orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Stop
> SGW-FLT2 (orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Stop
> EPC-3G (orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Stop
> CWS-3G (orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Stop
> SGW-FLT (orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Stop
> MME-FLT (orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Stop
> CWS-FLT (orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Stop
> CWS-FLT2 (orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: LogActions: Stop
> MME-FLT2 (orana)
> Jan 13 19:33:03 [4295] orana    pengine:   notice: process_pe_message:
> Calculated Transition 2316:
> /var/lib/pacemaker/pengine/pe-input-1451.bz2
> Jan 13 19:33:03 [4296] orana       crmd:     info:
> do_state_transition: State transition S_POLICY_ENGINE ->
> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
> origin=handle_response ]
> Jan 13 19:33:03 [4296] orana       crmd:     info: do_te_invoke:
> Processing graph 2316 (ref=pe_calc-dc-1452731583-2557) derived from
> /var/lib/pacemaker/pengine/pe-input-1451.bz2
> Jan 13 19:33:03 [4296] orana       crmd:   notice: te_rsc_command:
> Initiating action 27: monitor foo:1_monitor_0 on kamet
> Jan 13 19:33:03 [4296] orana       crmd:   notice: te_rsc_command:
> Initiating action 28: monitor fence-uc-orana_monitor_0 on kamet
> Jan 13 19:33:03 [4296] orana       crmd:   notice: te_rsc_command:
> Initiating action 29: monitor fence-uc-kamet_monitor_0 on kamet
> Jan 13 19:33:03 [4296] orana       crmd:   notice: te_rsc_command:
> Initiating action 76: notify foo_pre_notify_start_0 on orana (local)
> Jan 13 19:33:03 [4296] orana       crmd:     info: do_lrm_rsc_op:
> Performing key=76:2316:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> op=foo_notify_0
> Jan 13 19:33:03 [4293] orana       lrmd:     info: log_execute:
> executing - rsc:foo action:notify call_id:136
> uc(foo)[7992]: 2016/01/13_19:33:03 INFO: [UC] Wed Jan 13 19:33:03 EST
> 2016 NOTIFY pre-start master=orana  start=kamet
> Jan 13 19:33:03 [4293] orana       lrmd:     info: log_finished:
> finished - rsc:foo action:notify call_id:136 pid:7992 exit-code:0
> exec-time:49ms queue-time:0ms
> Jan 13 19:33:03 [4296] orana       crmd:     info: match_graph_event:
> Action foo_notify_0 (76) confirmed on orana (rc=0)
> Jan 13 19:33:03 [4296] orana       crmd:   notice: process_lrm_event:
> Operation foo_notify_0: ok (node=orana, call=136, rc=0, cib-update=0,
> confirmed=true)
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.6 2
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.7 (null)
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=7
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/status/node_state[@id='kamet']:  <transient_attributes
> id="kamet"/>
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>                                        <instance_attributes
> id="status-kamet">
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>                                          <nvpair
> id="status-kamet-shutdown" name="shutdown" value="0"/>
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>                                        </instance_attributes>
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>                                      </transient_attributes>
> Jan 13 19:33:04 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=kamet/attrd/3, version=0.79.7)
> Jan 13 19:33:04 [4296] orana       crmd:   notice:
> abort_transition_graph: Transition aborted by
> transient_attributes.kamet 'create': Transient attribute change
> (cib=0.79.7, source=te_update_diff:391,
> path=/cib/status/node_state[@id='kamet'], 0)
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.7 2
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.8 (null)
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=8
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: +
> /cib/status/node_state[@id='kamet']:
> @crm-debug-origin=do_update_resource
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/status/node_state[@id='kamet']/lrm[@id='kamet']/lrm_resources:
> <lrm_resource id="fence-uc-orana" type="fence_ilo4" class="stonith"/>
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>
> <lrm_rsc_op id="fence-uc-orana_last_0"
> operation_key="fence-uc-orana_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="28:2316:7:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:7;28:2316:7:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="10" rc-code="7" op-status="0" interval="0"
> last-run="1452731698"
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>
> </lrm_resource>
> Jan 13 19:33:04 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=kamet/crmd/11, version=0.79.8)
> Jan 13 19:33:04 [4296] orana       crmd:     info: match_graph_event:
> Action fence-uc-orana_monitor_0 (28) confirmed on kamet (rc=0)
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.8 2
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.9 (null)
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=9
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/status/node_state[@id='kamet']/lrm[@id='kamet']/lrm_resources:
> <lrm_resource id="fence-uc-kamet" type="fence_ilo4" class="stonith"/>
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>
> <lrm_rsc_op id="fence-uc-kamet_last_0"
> operation_key="fence-uc-kamet_monitor_0" operation="monitor"
> crm-debug-origin="do_update_resource" crm_feature_set="3.0.9"
> transition-key="29:2316:7:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:7;29:2316:7:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="14" rc-code="7" op-status="0" interval="0"
> last-run="1452731698"
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>
> </lrm_resource>
> Jan 13 19:33:04 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=kamet/crmd/12, version=0.79.9)
> Jan 13 19:33:04 [4296] orana       crmd:     info: match_graph_event:
> Action fence-uc-kamet_monitor_0 (29) confirmed on kamet (rc=0)
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.9 2
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.10 (null)
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=10
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/status/node_state[@id='kamet']/lrm[@id='kamet']/lrm_resources:
> <lrm_resource id="foo" type="uc" class="ocf" provider="pw"/>
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>
> <lrm_rsc_op id="foo_last_0" operation_key="foo_monitor_0"
> operation="monitor" crm-debug-origin="do_update_resource"
> crm_feature_set="3.0.9"
> transition-key="27:2316:7:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> transition-magic="0:7;27:2316:7:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7"
> call-id="6" rc-code="7" op-status="0" interval="0"
> last-run="1452731698" last-rc-chan
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
>
> </lrm_resource>
> Jan 13 19:33:04 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=kamet/crmd/13, version=0.79.10)
> Jan 13 19:33:04 [4296] orana       crmd:     info: match_graph_event:
> Action foo_monitor_0 (27) confirmed on kamet (rc=0)
> Jan 13 19:33:04 [4296] orana       crmd:   notice: te_rsc_command:
> Initiating action 26: probe_complete probe_complete-kamet on kamet -
> no waiting
> Jan 13 19:33:04 [4296] orana       crmd:     info: te_rsc_command:
> Action 26 confirmed - no wait
> Jan 13 19:33:04 [4296] orana       crmd:   notice: run_graph:
> Transition 2316 (Complete=8, Pending=0, Fired=0, Skipped=29,
> Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-input-1451.bz2):
> Stopped
> Jan 13 19:33:04 [4296] orana       crmd:     info:
> do_state_transition: State transition S_TRANSITION_ENGINE ->
> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=notify_crmd ]
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: --- 0.79.10 2
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op:
> Diff: +++ 0.79.11 (null)
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: +
> /cib:  @num_updates=11
> Jan 13 19:33:04 [4291] orana        cib:     info: cib_perform_op: ++
> /cib/status/node_state[@id='kamet']/transient_attributes[@id='kamet']/instance_attributes[@id='status-kamet']:
>  <nvpair id="status-kamet-probe_complete" name="probe_complete"
> value="true"/>
> Jan 13 19:33:04 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=kamet/attrd/8, version=0.79.11)
> Jan 13 19:33:04 [4291] orana        cib:     info:
> cib_process_request: Forwarding cib_modify operation for section
> status to master (origin=local/attrd/123)
> Jan 13 19:33:04 [4291] orana        cib:     info:
> cib_process_request: Completed cib_modify operation for section
> status: OK (rc=0, origin=orana/attrd/123, version=0.79.11)
> Jan 13 19:33:04 [4296] orana       crmd:     info:
> abort_transition_graph: Transition aborted by
> status-kamet-probe_complete, probe_complete=true: Transient attribute
> change (create cib=0.79.11, source=te_update_diff:391,
> path=/cib/status/node_state[@id='kamet']/transient_attributes[@id='kamet']/instance_attributes[@id='status-kamet'],
> 1)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> determine_online_status_fencing: Node kamet is active
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> determine_online_status: Node kamet is online
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> determine_online_status_fencing: Node orana is active
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> determine_online_status: Node orana is online
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan MGMT-FLT running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan SGW-FLT2 running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan EPC-3G running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan CWS-3G running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan SGW-FLT running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan MME-FLT running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan CWS-FLT running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan CWS-FLT2 running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan MME-FLT2 running on orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: clone_print:
> Master/Slave Set: foo-master [foo]
> Jan 13 19:33:04 [4295] orana    pengine:     info: short_print:
> Masters: [ orana ]
> Jan 13 19:33:04 [4295] orana    pengine:     info: short_print:
> Stopped: [ kamet ]
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> fence-uc-orana (stonith:fence_ilo4): Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> fence-uc-kamet (stonith:fence_ilo4): Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> MGMT-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> SGW-FLT2 (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> EPC-3G (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> CWS-3G (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> SGW-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> MME-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> CWS-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> CWS-FLT2 (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> MME-FLT2 (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> MGMT-FLT from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> SGW-FLT2 from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> EPC-3G from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> CWS-3G from orana
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="kamet" lanplus="1" action="off"
> delay="10" ipaddr="10.11.10.21"/>
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-kamet_start_0 on orana
> changed: was 93013a04ac64e00066a4f7b080b31515 vs. now
> a204a24f6a125ccea22f6dc681d020f0 (reload:3.0.9)
> 0:0;13:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="kamet" lanplus="1" action="off"
> delay="10" ipaddr="10.11.10.21" CRM_meta_timeout="20000"/>
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-kamet_monitor_5000 on
> orana changed: was e884a8e46b98e8563278b8117eb4db82 vs. now
> 948f08ea79b4152d6055da386ebacbb3 (reload:3.0.9)
> 0:0;1:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> SGW-FLT from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> MME-FLT from orana
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="orana" lanplus="1" action="off"
> delay="0" ipaddr="10.11.10.30"/>
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-orana_start_0 on orana
> changed: was 363cbd5d9ec840db46e5e25e2f8b6652 vs. now
> 44b3f881263f198e021240ff36365f0f (reload:3.0.9)
> 0:0;14:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="orana" lanplus="1" action="off"
> delay="0" ipaddr="10.11.10.30" CRM_meta_timeout="20000"/>
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-orana_monitor_5000 on
> orana changed: was 8024392825e4a1e81b7f628e6dee69a6 vs. now
> ec6a353c4e45a74e3497360e0581b40c (reload:3.0.9)
> 0:0;12:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> CWS-FLT from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> CWS-FLT2 from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> MME-FLT2 from orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: master_color:
> Promoting foo:0 (Master orana)
> Jan 13 19:33:04 [4295] orana    pengine:     info: master_color:
> foo-master: Promoted 1 instances of a possible 1 to master
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource MGMT-FLT
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource SGW-FLT2
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource EPC-3G
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource CWS-3G
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource SGW-FLT
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource MME-FLT
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource CWS-FLT
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource CWS-FLT2
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource MME-FLT2
> Jan 13 19:33:04 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (11s) for foo:1 on kamet
> Jan 13 19:33:04 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (11s) for foo:1 on kamet
> Jan 13 19:33:04 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (5s) for fence-uc-orana on orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (5s) for fence-uc-kamet on kamet
> Jan 13 19:33:04 [4295] orana    pengine:     info: LogActions: Leave
> foo:0 (Master orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Start
> foo:1 (kamet)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Restart
> fence-uc-orana (Started orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Move
> fence-uc-kamet (Started orana -> kamet)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> MGMT-FLT (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> SGW-FLT2 (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> EPC-3G (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> CWS-3G (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> SGW-FLT (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> MME-FLT (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> CWS-FLT (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> CWS-FLT2 (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> MME-FLT2 (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: process_pe_message:
> Calculated Transition 2317:
> /var/lib/pacemaker/pengine/pe-input-1452.bz2
> Jan 13 19:33:04 [4296] orana       crmd:     info: handle_response:
> pe_calc calculation pe_calc-dc-1452731584-2564 is obsolete
> Jan 13 19:33:04 [4295] orana    pengine:   notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> determine_online_status_fencing: Node kamet is active
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> determine_online_status: Node kamet is online
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> determine_online_status_fencing: Node orana is active
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> determine_online_status: Node orana is online
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan MGMT-FLT running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan SGW-FLT2 running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan EPC-3G running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan CWS-3G running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan SGW-FLT running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan MME-FLT running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan CWS-FLT running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan CWS-FLT2 running on orana
> Jan 13 19:33:04 [4295] orana    pengine:  warning: process_rsc_state:
> Detected active orphan MME-FLT2 running on orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: clone_print:
> Master/Slave Set: foo-master [foo]
> Jan 13 19:33:04 [4295] orana    pengine:     info: short_print:
> Masters: [ orana ]
> Jan 13 19:33:04 [4295] orana    pengine:     info: short_print:
> Stopped: [ kamet ]
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> fence-uc-orana (stonith:fence_ilo4): Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> fence-uc-kamet (stonith:fence_ilo4): Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> MGMT-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> SGW-FLT2 (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> EPC-3G (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> CWS-3G (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> SGW-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> MME-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> CWS-FLT (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> CWS-FLT2 (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_print:
> MME-FLT2 (ocf::pw:IPaddr): ORPHANED Started orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> MGMT-FLT from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> SGW-FLT2 from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> EPC-3G from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> CWS-3G from orana
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="kamet" lanplus="1" action="off"
> delay="10" ipaddr="10.11.10.21"/>
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-kamet_start_0 on orana
> changed: was 93013a04ac64e00066a4f7b080b31515 vs. now
> a204a24f6a125ccea22f6dc681d020f0 (reload:3.0.9)
> 0:0;13:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="kamet" lanplus="1" action="off"
> delay="10" ipaddr="10.11.10.21" CRM_meta_timeout="20000"/>
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-kamet_monitor_5000 on
> orana changed: was e884a8e46b98e8563278b8117eb4db82 vs. now
> 948f08ea79b4152d6055da386ebacbb3 (reload:3.0.9)
> 0:0;1:2304:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> SGW-FLT from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> MME-FLT from orana
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="orana" lanplus="1" action="off"
> delay="0" ipaddr="10.11.10.30"/>
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-orana_start_0 on orana
> changed: was 363cbd5d9ec840db46e5e25e2f8b6652 vs. now
> 44b3f881263f198e021240ff36365f0f (reload:3.0.9)
> 0:0;14:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: params:reload   <parameters login="parallel"
> passwd="wireless" pcmk_host_list="orana" lanplus="1" action="off"
> delay="0" ipaddr="10.11.10.30" CRM_meta_timeout="20000"/>
> Jan 13 19:33:04 [4295] orana    pengine:     info:
> check_action_definition: Parameters to fence-uc-orana_monitor_5000 on
> orana changed: was 8024392825e4a1e81b7f628e6dee69a6 vs. now
> ec6a353c4e45a74e3497360e0581b40c (reload:3.0.9)
> 0:0;12:2303:0:70de24fe-0f17-47ec-a193-a2cdd9f9e5c7
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> CWS-FLT from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> CWS-FLT2 from orana
> Jan 13 19:33:04 [4295] orana    pengine:   notice: DeleteRsc: Removing
> MME-FLT2 from orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: master_color:
> Promoting foo:0 (Master orana)
> Jan 13 19:33:04 [4295] orana    pengine:     info: master_color:
> foo-master: Promoted 1 instances of a possible 1 to master
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource MGMT-FLT
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource SGW-FLT2
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource EPC-3G
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource CWS-3G
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource SGW-FLT
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource MME-FLT
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource CWS-FLT
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource CWS-FLT2
> Jan 13 19:33:04 [4295] orana    pengine:     info: native_color:
> Stopping orphan resource MME-FLT2
> Jan 13 19:33:04 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (11s) for foo:1 on kamet
> Jan 13 19:33:04 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (11s) for foo:1 on kamet
> Jan 13 19:33:04 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (5s) for fence-uc-orana on orana
> Jan 13 19:33:04 [4295] orana    pengine:     info: RecurringOp: Start
> recurring monitor (5s) for fence-uc-kamet on kamet
> Jan 13 19:33:04 [4295] orana    pengine:     info: LogActions: Leave
> foo:0 (Master orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Start
> foo:1 (kamet)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Restart
> fence-uc-orana (Started orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Move
> fence-uc-kamet (Started orana -> kamet)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> MGMT-FLT (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> SGW-FLT2 (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> EPC-3G (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> CWS-3G (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> SGW-FLT (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> MME-FLT (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> CWS-FLT (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> CWS-FLT2 (orana)
> Jan 13 19:33:04 [4295] orana    pengine:   notice: LogActions: Stop
> MME-FLT2 (orana)Thanks in advance.
>
> Regards
> Arjun




More information about the Users mailing list