[Pacemaker] Pacemaker/Corosync members do not join cluster
Andrew Beekhof
andrew at beekhof.net
Tue Sep 16 03:29:10 EDT 2014
On 16 Sep 2014, at 11:24 am, Jeffrey Roberts <jeffrey.l.roberts at gmail.com> wrote:
> Greetings,
>
> I have been Googling for hours now, and I can't seem to find an answer to the challenge I face today, which is no matter what I have tried I can not get the nodes to join or sync with each other... Please let me know if there is anymore output I can provide, and thank you very much for your help, it is greatly appreciated! btw, I am hosting on Amazon EC2, and please note I have tried both internal and public IPs with no avail... also, there is no selinux installed, and iptables is disabled, and the ec2 firewall is wide open for TCP and UDP....
>
> Here is the output from my `pcs status` on both nodes
>
> -------------------------------------------------------------------------------------------
> [~]
> [root at jlr-lb-01] -> pcs status
> Cluster name: jlr-lb
> Last updated: Tue Sep 16 01:12:13 2014
> Last change: Tue Sep 16 00:32:28 2014 via cibadmin on jlr-lb-01.aws
> Stack: cman
You posted corosync.conf but this cluster is using cman... what does cluster.conf look like and is it using udpu too?
> Current DC: jlr-lb-01.aws - partition WITHOUT quorum
> Version: 1.1.10-14.el6_5.3-368c726
> 2 Nodes configured
> 1 Resources configured
>
>
> Online: [ jlr-lb-01.aws ]
> OFFLINE: [ jlr-lb-02.aws ]
>
> Full list of resources:
>
> livefrontendIP0 (ocf::heartbeat:AWSVip): Started jlr-lb-01.aws
> -------------------------------------------------------------------------------------------
> [~]
> [root at jlr-lb-02] -> pcs status
> Cluster name: jlr-lb
> Last updated: Tue Sep 16 01:09:27 2014
> Last change: Tue Sep 16 01:04:35 2014 via cibadmin on jlr-lb-02.aws
> Stack: cman
> Current DC: jlr-lb-02.aws - partition WITHOUT quorum
> Version: 1.1.10-14.el6_5.3-368c726
> 2 Nodes configured
> 1 Resources configured
>
>
> Online: [ jlr-lb-02.aws ]
> OFFLINE: [ jlr-lb-01.aws ]
>
> Full list of resources:
>
> livefrontendIP0 (ocf::heartbeat:AWSVip): Started jlr-lb-02.aws
> -------------------------------------------------------------------------------------------
>
> Here is the output from all the commands I could think of
>
> -------------------------------------------------------------------------------------------
> NODE 1
> -------------------------------------------------------------------------------------------
> [~]
> [root at jlr-lb-01] -> pcs config
> Cluster Name: jlr-lb
> Corosync Nodes:
>
> Pacemaker Nodes:
> jlr-lb-01.aws jlr-lb-02.aws
>
> Resources:
> Resource: livefrontendIP0 (class=ocf provider=heartbeat type=AWSVip)
> Operations: monitor interval=60s (livefrontendIP0-monitor-interval-60s)
>
> Stonith Devices:
> Fencing Levels:
>
> Location Constraints:
> Resource: livefrontendIP0
> Enabled on: jlr-lb-01.aws (score:INFINITY) (id:location-livefrontendIP0-jlr-lb-01.aws-INFINITY)
> Ordering Constraints:
> Colocation Constraints:
>
> Cluster Properties:
> cluster-infrastructure: cman
> dc-version: 1.1.10-14.el6_5.3-368c726
> no-quorum-policy: ignore
> stonith-enabled: false
>
>
>
> [~]
> [root at jlr-lb-01] -> netstat -nlpa | grep corosync
> udp 0 0 10.38.24.5:5404 0.0.0.0:* 889/corosync
> udp 0 0 10.38.24.5:5405 0.0.0.0:* 889/corosync
> udp 0 0 239.192.99.112:5405 0.0.0.0:* 889/corosync
> unix 2 [ ACC ] STREAM LISTENING 10089 889/corosync @corosync.ipc
> unix 2 [ ACC ] STREAM LISTENING 10109 889/corosync /var/run/cman_client
> unix 2 [ ACC ] STREAM LISTENING 10111 889/corosync /var/run/cman_admin
> unix 3 [ ] STREAM CONNECTED 12443 889/corosync /var/run/cman_client
> unix 3 [ ] STREAM CONNECTED 12423 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12276 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12244 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12239 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12142 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12136 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 11023 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10874 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10691 889/corosync /var/run/cman_client
> unix 3 [ ] STREAM CONNECTED 10692 889/corosync /var/run/cman_admin
> unix 3 [ ] STREAM CONNECTED 10655 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10609 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10603 889/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10374 889/corosync /var/run/cman_client
> unix 3 [ ] STREAM CONNECTED 10375 889/corosync /var/run/cman_admin
> unix 3 [ ] STREAM CONNECTED 10290 889/corosync /var/run/cman_client
> unix 3 [ ] STREAM CONNECTED 10288 889/corosync /var/run/cman_admin
> unix 3 [ ] DGRAM 10106 889/corosync
> unix 3 [ ] DGRAM 10105 889/corosync
> unix 2 [ ] DGRAM 10085 889/corosync
>
>
> [~]
> [root at jlr-lb-01] -> cman_tool nodes
> Node Sts Inc Joined Name
> 1 M 52 2014-09-16 01:08:19 jlr-lb-01.aws
> 2 X 0 jlr-lb-02.aws
>
>
> [~]
> [root at jlr-lb-01] -> cibadmin -Ql
> <cib epoch="7" num_updates="6" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Tue Sep 16 00:32:28 2014" update-origin="jlr-lb-01.aws" update-client="cibadmin" crm_feature_set="3.0.7" have-quorum="0" dc-uuid="jlr-lb-01.aws">
> <configuration>
> <crm_config>
> <cluster_property_set id="cib-bootstrap-options">
> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.10-14.el6_5.3-368c726"/>
> <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="cman"/>
> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/>
> <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/>
> </cluster_property_set>
> </crm_config>
> <nodes>
> <node id="jlr-lb-01.aws" uname="jlr-lb-01.aws"/>
> <node id="jlr-lb-02.aws" uname="jlr-lb-02.aws"/>
> </nodes>
> <resources>
> <primitive class="ocf" id="livefrontendIP0" provider="heartbeat" type="AWSVip">
> <instance_attributes id="livefrontendIP0-instance_attributes"/>
> <operations>
> <op id="livefrontendIP0-monitor-interval-60s" interval="60s" name="monitor"/>
> </operations>
> </primitive>
> </resources>
> <constraints>
> <rsc_location id="location-livefrontendIP0-jlr-lb-01.aws-INFINITY" node="jlr-lb-01.aws" rsc="livefrontendIP0" score="INFINITY"/>
> </constraints>
> </configuration>
> <status>
> <node_state id="jlr-lb-01.aws" uname="jlr-lb-01.aws" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member">
> <lrm id="jlr-lb-01.aws">
> <lrm_resources>
> <lrm_resource id="livefrontendIP0" type="AWSVip" class="ocf" provider="heartbeat">
> <lrm_rsc_op id="livefrontendIP0_last_failure_0" operation_key="livefrontendIP0_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="4:0:7:036403dd-11d8-4f25-868d-1bbbf69d05fa" transition-magic="0:0;4:0:7:036403dd-11d8-4f25-868d-1bbbf69d05fa" call-id="5" rc-code="0" op-status="0" interval="0" last-run="1410829733" last-rc-change="1410829733" exec-time="65" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8" op-force-restart=" state " op-restart-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
> <lrm_rsc_op id="livefrontendIP0_monitor_60000" operation_key="livefrontendIP0_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="6:1:0:036403dd-11d8-4f25-868d-1bbbf69d05fa" transition-magic="0:0;6:1:0:036403dd-11d8-4f25-868d-1bbbf69d05fa" call-id="8" rc-code="0" op-status="0" interval="60000" last-rc-change="1410829733" exec-time="28" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
> </lrm_resource>
> </lrm_resources>
> </lrm>
> <transient_attributes id="jlr-lb-01.aws">
> <instance_attributes id="status-jlr-lb-01.aws">
> <nvpair id="status-jlr-lb-01.aws-probe_complete" name="probe_complete" value="true"/>
> </instance_attributes>
> </transient_attributes>
> </node_state>
> <node_state id="jlr-lb-02.aws" uname="jlr-lb-02.aws" crmd="offline" join="down" crm-debug-origin="do_state_transition"/>
> </status>
> </cib>
>
>
> [~]
> [root at jlr-lb-01] -> corosync-objctl | grep member
> runtime.totem.pg.mrp.srp.members.1.ip=r(0) ip(10.38.24.5)
> runtime.totem.pg.mrp.srp.members.1.join_count=1
> runtime.totem.pg.mrp.srp.members.1.status=joined
>
>
> [~]
> [root at jlr-lb-01] -> crm_verify -L -V
>
>
> [~]
> [root at jlr-lb-01] -> cat /etc/corosync/corosync.conf
> # Please read the corosync.conf.5 manual page
> compatibility: whitetank
>
> totem {
> version: 2
> secauth: off
> threads: 0
> join: 1000
> rrp_mode: passive
> interface {
> member {
> memberaddr: 10.38.24.5
> }
> member {
> memberaddr: 54.244.91.9
> # memberaddr: 10.235.48.124
> }
> ringnumber: 0
> bindnetaddr: 10.38.24.0
> mcastport: 5405
> ttl: 1
> }
> transport: udpu
> }
>
> logging {
> fileline: off
> to_stderr: no
> to_logfile: yes
> to_syslog: yes
> logfile: /var/log/cluster/corosync.log
> debug: off
> timestamp: on
> logger_subsys {
> subsys: AMF
> debug: off
> }
> }
>
> amf {
> mode: disabled
> }
>
>
> [~]
> [root at jlr-lb-01] -> tcpdump -i eth0 -p udp -envv "port 5405"
> tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 01:13:29.506832 22:00:0a:26:18:05 > 01:00:5e:40:63:70, ethertype IPv4 (0x0800), length 161: (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 147)
> 10.38.24.5.hpoms-dps-lstn > 239.192.99.112.netsupport: [bad udp cksum 310e!] UDP, length 119
> 01:13:31.410357 22:00:0a:26:18:05 > 01:00:5e:40:63:70, ethertype IPv4 (0x0800), length 161: (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 147)
> 10.38.24.5.hpoms-dps-lstn > 239.192.99.112.netsupport: [bad udp cksum 6f85!] UDP, length 119
> ^C
> 2 packets captured
> 2 packets received by filter
> 0 packets dropped by kernel
>
>
> -------------------------------------------------------------------------------------------
> NODE 2
> -------------------------------------------------------------------------------------------
> [~]
> [root at jlr-lb-02] -> pcs config
> Cluster Name: jlr-lb
> Corosync Nodes:
>
> Pacemaker Nodes:
> jlr-lb-01.aws jlr-lb-02.aws
>
> Resources:
> Resource: livefrontendIP0 (class=ocf provider=heartbeat type=AWSVip)
> Operations: monitor interval=60s (livefrontendIP0-monitor-interval-60s)
>
> Stonith Devices:
> Fencing Levels:
>
> Location Constraints:
> Resource: livefrontendIP0
> Enabled on: jlr-lb-01.aws (score:INFINITY) (id:location-livefrontendIP0-jlr-lb-01.aws-INFINITY)
> Ordering Constraints:
> Colocation Constraints:
>
> Cluster Properties:
> cluster-infrastructure: cman
> dc-version: 1.1.10-14.el6_5.3-368c726
> no-quorum-policy: ignore
> stonith-enabled: false
>
>
> [~]
> [root at jlr-lb-02] -> netstat -nlpa | grep corosync
> udp 0 0 10.235.48.124:5404 0.0.0.0:* 892/corosync
> udp 0 0 10.235.48.124:5405 0.0.0.0:* 892/corosync
> udp 0 0 239.192.99.112:5405 0.0.0.0:* 892/corosync
> unix 2 [ ACC ] STREAM LISTENING 10095 892/corosync @corosync.ipc
> unix 2 [ ACC ] STREAM LISTENING 10125 892/corosync /var/run/cman_client
> unix 2 [ ACC ] STREAM LISTENING 10127 892/corosync /var/run/cman_admin
> unix 3 [ ] STREAM CONNECTED 12563 892/corosync /var/run/cman_client
> unix 3 [ ] STREAM CONNECTED 12551 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12480 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12454 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12435 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12268 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 12262 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 11026 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10877 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10694 892/corosync /var/run/cman_client
> unix 3 [ ] STREAM CONNECTED 10695 892/corosync /var/run/cman_admin
> unix 3 [ ] STREAM CONNECTED 10672 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10613 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10607 892/corosync @corosync.ipc
> unix 3 [ ] STREAM CONNECTED 10380 892/corosync /var/run/cman_client
> unix 3 [ ] STREAM CONNECTED 10381 892/corosync /var/run/cman_admin
> unix 3 [ ] STREAM CONNECTED 10298 892/corosync /var/run/cman_client
> unix 3 [ ] STREAM CONNECTED 10293 892/corosync /var/run/cman_admin
> unix 3 [ ] DGRAM 10112 892/corosync
> unix 3 [ ] DGRAM 10111 892/corosync
> unix 2 [ ] DGRAM 10091 892/corosync
>
>
> [~]
> [root at jlr-lb-02] -> cman_tool nodes
> Node Sts Inc Joined Name
> 1 X 0 jlr-lb-01.aws
> 2 M 40 2014-09-16 01:07:57 jlr-lb-02.aws
>
>
> [~]
> [root at jlr-lb-02] -> cibadmin -Ql
> <cib epoch="7" num_updates="6" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Tue Sep 16 01:04:35 2014" update-origin="jlr-lb-02.aws" update-client="cibadmin" crm_feature_set="3.0.7" have-quorum="0" dc-uuid="jlr-lb-02.aws">
> <configuration>
> <crm_config>
> <cluster_property_set id="cib-bootstrap-options">
> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.10-14.el6_5.3-368c726"/>
> <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="cman"/>
> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/>
> <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/>
> </cluster_property_set>
> </crm_config>
> <nodes>
> <node id="jlr-lb-01.aws" uname="jlr-lb-01.aws"/>
> <node id="jlr-lb-02.aws" uname="jlr-lb-02.aws"/>
> </nodes>
> <resources>
> <primitive class="ocf" id="livefrontendIP0" provider="heartbeat" type="AWSVip">
> <instance_attributes id="livefrontendIP0-instance_attributes"/>
> <operations>
> <op id="livefrontendIP0-monitor-interval-60s" interval="60s" name="monitor"/>
> </operations>
> </primitive>
> </resources>
> <constraints>
> <rsc_location id="location-livefrontendIP0-jlr-lb-01.aws-INFINITY" node="jlr-lb-01.aws" rsc="livefrontendIP0" score="INFINITY"/>
> </constraints>
> </configuration>
> <status>
> <node_state id="jlr-lb-02.aws" uname="jlr-lb-02.aws" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member">
> <lrm id="jlr-lb-02.aws">
> <lrm_resources>
> <lrm_resource id="livefrontendIP0" type="AWSVip" class="ocf" provider="heartbeat">
> <lrm_rsc_op id="livefrontendIP0_last_failure_0" operation_key="livefrontendIP0_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="4:0:7:ae731050-4f86-455d-b180-c8948bca0331" transition-magic="0:0;4:0:7:ae731050-4f86-455d-b180-c8948bca0331" call-id="5" rc-code="0" op-status="0" interval="0" last-run="1410829709" last-rc-change="1410829709" exec-time="113" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8" op-force-restart=" state " op-restart-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
> <lrm_rsc_op id="livefrontendIP0_monitor_60000" operation_key="livefrontendIP0_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.7" transition-key="6:1:0:ae731050-4f86-455d-b180-c8948bca0331" transition-magic="0:0;6:1:0:ae731050-4f86-455d-b180-c8948bca0331" call-id="8" rc-code="0" op-status="0" interval="60000" last-rc-change="1410829710" exec-time="22" queue-time="0" op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
> </lrm_resource>
> </lrm_resources>
> </lrm>
> <transient_attributes id="jlr-lb-02.aws">
> <instance_attributes id="status-jlr-lb-02.aws">
> <nvpair id="status-jlr-lb-02.aws-probe_complete" name="probe_complete" value="true"/>
> </instance_attributes>
> </transient_attributes>
> </node_state>
> <node_state id="jlr-lb-01.aws" uname="jlr-lb-01.aws" crmd="offline" join="down" crm-debug-origin="do_state_transition"/>
> </status>
> </cib>
>
>
> [~]
> [root at jlr-lb-02] -> corosync-objctl | grep member
> runtime.totem.pg.mrp.srp.members.2.ip=r(0) ip(10.235.48.124)
> runtime.totem.pg.mrp.srp.members.2.join_count=1
> runtime.totem.pg.mrp.srp.members.2.status=joined
> You have new mail in /var/spool/mail/root
>
>
> [~]
> [root at jlr-lb-02] -> crm_verify -L -V
>
>
> [~]
> [root at jlr-lb-02] -> cat /etc/corosync/corosync.conf
> # Please read the corosync.conf.5 manual page
> compatibility: whitetank
>
> totem {
> version: 2
> secauth: off
> threads: 0
> join: 1000
> rrp_mode: passive
> interface {
> member {
> memberaddr: 54.244.91.8
> # memberaddr: 10.38.24.5
> }
> member {
> memberaddr: 10.235.48.124
> }
> ringnumber: 0
> bindnetaddr: 10.235.48.0
> mcastport: 5405
> ttl: 1
> }
> transpot: udpu
> }
>
> logging {
> fileline: off
> to_stderr: no
> to_logfile: yes
> to_syslog: yes
> logfile: /var/log/cluster/corosync.log
> debug: off
> timestamp: on
> logger_subsys {
> subsys: AMF
> debug: off
> }
> }
>
> amf {
> mode: disabled
> }
>
>
> [~]
> [root at jlr-lb-02] -> tcpdump -i eth0 -p udp -envv "port 5405"
> tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 01:11:52.100121 22:00:0a:eb:30:7c > 01:00:5e:40:63:70, ethertype IPv4 (0x0800), length 161: (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 147)
> 10.235.48.124.hpoms-dps-lstn > 239.192.99.112.netsupport: [bad udp cksum e1de!] UDP, length 119
> 01:11:54.003694 22:00:0a:eb:30:7c > 01:00:5e:40:63:70, ethertype IPv4 (0x0800), length 161: (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto UDP (17), length 147)
> 10.235.48.124.hpoms-dps-lstn > 239.192.99.112.netsupport: [bad udp cksum 2206!] UDP, length 119
> ^C
> 2 packets captured
> 2 packets received by filter
> 0 packets dropped by kernel
>
>
> -------------------------------------------------------------------------------------------
> NODE 1 LOGS
> -------------------------------------------------------------------------------------------
> Sep 16 01:08:19 corosync [MAIN ] Corosync Cluster Engine ('1.4.1'): started and ready to provide service.
> Sep 16 01:08:19 corosync [MAIN ] Corosync built-in features: nss dbus rdma snmp
> Sep 16 01:08:19 corosync [MAIN ] Successfully read config from /etc/cluster/cluster.conf
> Sep 16 01:08:19 corosync [MAIN ] Successfully parsed cman config
> Sep 16 01:08:19 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
> Sep 16 01:08:19 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Sep 16 01:08:19 corosync [TOTEM ] The network interface [10.38.24.5] is now up.
> Sep 16 01:08:19 corosync [QUORUM] Using quorum provider quorum_cman
> Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
> Sep 16 01:08:19 corosync [CMAN ] CMAN 3.0.12.1 (built Apr 14 2014 09:36:10) started
> Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync CMAN membership service 2.90
> Sep 16 01:08:19 corosync [SERV ] Service engine loaded: openais checkpoint service B.01.01
> Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync extended virtual synchrony service
> Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync configuration service
> Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
> Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync cluster config database access v1.01
> Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync profile loading service
> Sep 16 01:08:19 corosync [QUORUM] Using quorum provider quorum_cman
> Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
> Sep 16 01:08:19 corosync [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
> Sep 16 01:08:19 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
> Sep 16 01:08:19 corosync [QUORUM] Members[1]: 1
> Sep 16 01:08:19 corosync [QUORUM] Members[1]: 1
> Sep 16 01:08:19 corosync [CPG ] chosen downlist: sender r(0) ip(10.38.24.5) ; members(old:0 left:0)
> Sep 16 01:08:19 corosync [MAIN ] Completed service synchronization, ready to provide service.
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: notice: main: Starting Pacemaker 1.1.10-14.el6_5.3 (Build: 368c726): generated-manpages agent-manpages ascii-docs publican-docs ncurses libqb-logging libqb-ipc nagios corosync-plugin cman
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: main: Maximum core file size is: 18446744073709551615
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: qb_ipcs_us_publish: server name: pacemakerd
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: crm_get_peer: Created entry 9cefa239-c37c-4590-bca9-5a48d7f967d9/0x759fc0 for node (null)/1 (1 total)
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: crm_get_peer: Cannot obtain a UUID for node 1/(null)
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[1] - corosync-cpg is now online
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: cman_node_name: Using CMAN node name jlr-lb-01.aws for 0
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: crm_get_peer: Node 1 is now known as jlr-lb-01.aws
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: crm_get_peer: Node 1 has uuid jlr-lb-01.aws
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child: Using uid=189 and group=189 for process cib
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child: Forked child 1333 for process cib
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child: Forked child 1334 for process stonith-ng
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child: Forked child 1335 for process lrmd
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child: Using uid=189 and group=189 for process attrd
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child: Forked child 1336 for process attrd
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child: Using uid=189 and group=189 for process pengine
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child: Forked child 1337 for process pengine
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child: Forked child 1338 for process crmd
> Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: main: Starting mainloop
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/hacluster
> Sep 16 01:08:29 [1337] jlr-lb-01.aws pengine: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/hacluster
> Sep 16 01:08:29 [1337] jlr-lb-01.aws pengine: info: qb_ipcs_us_publish: server name: pengine
> Sep 16 01:08:29 [1337] jlr-lb-01.aws pengine: info: main: Starting pengine
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: get_cluster_type: Verifying cluster type: 'cman'
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: get_cluster_type: Assuming an active 'cman' cluster
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig)
> Sep 16 01:08:29 [1336] jlr-lb-01.aws attrd: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/root
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: get_cluster_type: Verifying cluster type: 'cman'
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: get_cluster_type: Assuming an active 'cman' cluster
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: validate_with_relaxng: Creating RNG parser context
> Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/root
> Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: notice: main: CRM Git Version: 368c726
> Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info: do_log: FSA: Input I_STARTUP from crmd_init() received in state S_STARTING
> Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info: get_cluster_type: Verifying cluster type: 'cman'
> Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info: get_cluster_type: Assuming an active 'cman' cluster
> Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info: crm_ipc_connect: Could not establish cib_shm connection: Connection refused (111)
> Sep 16 01:08:29 [1335] jlr-lb-01.aws lrmd: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/root
> Sep 16 01:08:29 [1335] jlr-lb-01.aws lrmd: info: qb_ipcs_us_publish: server name: lrmd
> Sep 16 01:08:29 [1335] jlr-lb-01.aws lrmd: info: main: Starting
> Sep 16 01:08:29 [1336] jlr-lb-01.aws attrd: notice: main: Starting mainloop...
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_get_peer: Created entry f7bd7d66-0f53-4aea-9ee0-c152a3a79785/0x15e8890 for node (null)/1 (1 total)
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_get_peer: Cannot obtain a UUID for node 1/(null)
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[1] - corosync-cpg is now online
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: cman_node_name: Using CMAN node name jlr-lb-01.aws for 0
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: init_cs_connection_once: Connection to 'cman': established
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: cman_node_name: Using CMAN node name jlr-lb-01.aws for 0
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_get_peer: Node 1 is now known as jlr-lb-01.aws
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_get_peer: Node 1 has uuid jlr-lb-01.aws
> Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_ipc_connect: Could not establish cib_rw connection: Connection refused (111)
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: startCib: CIB Initialization completed successfully
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_get_peer: Created entry e5be9125-d252-4ba7-a4cd-3744cb955962/0x1bbe6b0 for node (null)/1 (1 total)
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_get_peer: Cannot obtain a UUID for node 1/(null)
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[1] - corosync-cpg is now online
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: cman_node_name: Using CMAN node name jlr-lb-01.aws for 0
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: init_cs_connection_once: Connection to 'cman': established
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: cman_node_name: Using CMAN node name jlr-lb-01.aws for 0
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_get_peer: Node 1 is now known as jlr-lb-01.aws
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_get_peer: Node 1 has uuid jlr-lb-01.aws
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: qb_ipcs_us_publish: server name: cib_ro
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: qb_ipcs_us_publish: server name: cib_rw
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: qb_ipcs_us_publish: server name: cib_shm
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: cib_init: Starting cib mainloop
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: pcmk_cpg_membership: Joined[0.0] cib.1
> Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: pcmk_cpg_membership: Member[0.0] cib.1
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-24.raw
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: write_cib_contents: Wrote version 0.7.0 of the CIB to disk (digest: f4e1275c2b642b1a630426162e42de0b)
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.zGel81 (digest: /var/lib/pacemaker/cib/cib.5oDZ5a)
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc1630 for uid=0 gid=0 pid=1338 id=152f038f-a799-462b-b6cc-7ec09b4f4e26
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_cib_control: CIB connection established
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/2, version=0.7.0)
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc8910 for uid=0 gid=0 pid=1334 id=8ee650dd-33db-4f11-b1e2-87ccf58e633d
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer: Created entry 8004e047-2a96-490e-83ba-c8510e3f1b94/0x1042580 for node (null)/1 (1 total)
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer: Cannot obtain a UUID for node 1/(null)
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[1] - corosync-cpg is now online
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: cman_node_name: Using CMAN node name jlr-lb-01.aws for 0
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: init_cs_connection_once: Connection to 'cman': established
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: cman_node_name: Using CMAN node name jlr-lb-01.aws for 0
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer: Node 1 is now known as jlr-lb-01.aws
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: peer_update_callback: jlr-lb-01.aws is now (null)
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer: Node 1 has uuid jlr-lb-01.aws
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: init_cman_connection: Configuring Pacemaker to obtain quorum from cman
> Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: notice: setup_cib: Watching for stonith topology changes
> Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info: qb_ipcs_us_publish: server name: stonith-ng
> Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info: main: Starting stonith-ng mainloop
> Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info: pcmk_cpg_membership: Joined[0.0] stonith-ng.1
> Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info: pcmk_cpg_membership: Member[0.0] stonith-ng.1
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: cman_event_callback: Membership 52: quorum still lost
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/2, version=0.7.0)
> Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info: init_cib_cache_cb: Updating device list from the cib: init
> Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: notice: unpack_config: On loss of CCM Quorum: Ignore
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: notice: crm_update_peer_state: cman_event_callback: Node jlr-lb-01.aws[1] - state is now member (was (null))
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: peer_update_callback: jlr-lb-01.aws is now member (was (null))
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer: Created entry cc5d11b6-0635-46ef-b670-d8541632213e/0x1044d30 for node jlr-lb-02.aws/2 (2 total)
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer: Node 2 is now known as jlr-lb-02.aws
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: peer_update_callback: jlr-lb-02.aws is now (null)
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer: Node 2 has uuid jlr-lb-02.aws
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: cman_event_callback: State of node jlr-lb-02.aws[2] is still unknown
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: cman_node_name: Using CMAN node name jlr-lb-01.aws for 0
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_ha_control: Connected to the cluster
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: lrmd_ipc_connect: Connecting to lrmd
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/3, version=0.7.0)
> Sep 16 01:08:30 [1335] jlr-lb-01.aws lrmd: info: crm_client_new: Connecting 0x1e08aa0 for uid=0 gid=0 pid=1338 id=e1350eaf-18c1-4f5e-a253-c52810ba6766
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_lrm_control: LRM connection established
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_started: Delaying start, Config not read (0000000000000040)
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_started: Delaying start, Config not read (0000000000000040)
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/4, version=0.7.0)
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: pcmk_cpg_membership: Joined[0.0] crmd.1
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: pcmk_cpg_membership: Member[0.0] crmd.1
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_started: Delaying start, Config not read (0000000000000040)
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: qb_ipcs_us_publish: server name: crmd
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: notice: do_started: The local CRM is operational
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_log: FSA: Input I_PENDING from do_started() received in state S_STARTING
> Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: notice: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
> Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_slave operation for section 'all': OK (rc=0, origin=local/crmd/5, version=0.7.0)
> Sep 16 01:08:32 [1334] jlr-lb-01.aws stonith-ng: info: crm_client_new: Connecting 0x16ef3b0 for uid=0 gid=0 pid=1338 id=689da5ef-ce47-4610-a744-8edcd995a048
> Sep 16 01:08:32 [1334] jlr-lb-01.aws stonith-ng: info: stonith_command: Processed register from crmd.1338: OK (0)
> Sep 16 01:08:32 [1334] jlr-lb-01.aws stonith-ng: info: stonith_command: Processed st_notify from crmd.1338: OK (0)
> Sep 16 01:08:32 [1334] jlr-lb-01.aws stonith-ng: info: stonith_command: Processed st_notify from crmd.1338: OK (0)
> Sep 16 01:08:34 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1a0f200 for uid=189 gid=0 pid=1336 id=e12f20e0-9c0e-4c55-aa7f-87f2f4e6488d
> Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)
> Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: warning: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info: do_log: FSA: Input I_ELECTION_DC from do_election_check() received in state S_ELECTION
> Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
> Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info: do_te_control: Registering TE UUID: 036403dd-11d8-4f25-868d-1bbbf69d05fa
> Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info: set_graph_functions: Setting custom graph functions
> Sep 16 01:08:52 [1337] jlr-lb-01.aws pengine: info: crm_client_new: Connecting 0x17dd260 for uid=0 gid=0 pid=1338 id=0d609011-5857-461b-9411-a0c20ebc166e
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: do_dc_takeover: Taking over DC status for this partition
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_readwrite: We are now in R/W mode
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/6, version=0.7.0)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/7, version=0.7.0)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='dc-version']: OK (rc=0, origin=local/crmd/8, version=0.7.0)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/9, version=0.7.0)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='cluster-infrastructure']: OK (rc=0, origin=local/crmd/10, version=0.7.0)
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: join_make_offer: Making join offers based on membership 52
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: join_make_offer: join-1: Sending offer to jlr-lb-01.aws
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: crm_update_peer_join: join_make_offer: Node jlr-lb-01.aws[1] - join-1 phase 0 -> 1
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/11, version=0.7.0)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/12, version=0.7.0)
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: update_dc: Set DC to jlr-lb-01.aws (3.0.7)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/13, version=0.7.0)
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node jlr-lb-01.aws[1] - join-1 phase 1 -> 2
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: crm_update_peer_expected: do_dc_join_filter_offer: Node jlr-lb-01.aws[1] - expected state is now member
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: crmd_join_phase_log: join-1: jlr-lb-01.aws=integrated
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: crmd_join_phase_log: join-1: jlr-lb-02.aws=none
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: do_dc_join_finalize: join-1: Syncing our CIB to the rest of the cluster
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cman_node_name: Using CMAN node name jlr-lb-01.aws for 0
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_sync operation for section 'all': OK (rc=0, origin=local/crmd/14, version=0.7.0)
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: crm_update_peer_join: finalize_join_for: Node jlr-lb-01.aws[1] - join-1 phase 2 -> 3
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='jlr-lb-01.aws']/transient_attributes
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: update_attrd: Connecting to attrd... 5 retries remaining
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: crm_update_peer_join: do_dc_join_ack: Node jlr-lb-01.aws[1] - join-1 phase 3 -> 4
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: do_dc_join_ack: join-1: Updating node state to member for jlr-lb-01.aws
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='jlr-lb-01.aws']/lrm
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/15, version=0.7.0)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='jlr-lb-01.aws']/transient_attributes: OK (rc=0, origin=local/crmd/16, version=0.7.0)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='jlr-lb-01.aws']/lrm: OK (rc=0, origin=local/crmd/17, version=0.7.0)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/18, version=0.7.1)
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
> Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: abort_transition_graph: do_te_invoke:151 - Triggered transition abort (complete=1) : Peer Cancelled
> Sep 16 01:08:52 [1336] jlr-lb-01.aws attrd: notice: attrd_local_callback: Sending full refresh (origin=crmd)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/19, version=0.7.1)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/20, version=0.7.2)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/21, version=0.7.3)
> Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/22, version=0.7.3)
> Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/23, version=0.7.3)
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: determine_online_status: Node jlr-lb-01.aws is online
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: native_print: livefrontendIP0 (ocf::heartbeat:AWSVip): Stopped
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: RecurringOp: Start recurring monitor (60s) for livefrontendIP0 on jlr-lb-01.aws
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice: LogActions: Start livefrontendIP0 (jlr-lb-01.aws)
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1410829733-7) derived from /var/lib/pacemaker/pengine/pe-input-72.bz2
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: te_rsc_command: Initiating action 4: monitor livefrontendIP0_monitor_0 on jlr-lb-01.aws (local)
> Sep 16 01:08:53 [1335] jlr-lb-01.aws lrmd: info: process_lrmd_get_rsc_info: Resource 'livefrontendIP0' not found (0 active resources)
> Sep 16 01:08:53 [1335] jlr-lb-01.aws lrmd: info: process_lrmd_rsc_register: Added 'livefrontendIP0' to the rsc list (1 active resources)
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_lrm_rsc_op: Performing key=4:0:7:036403dd-11d8-4f25-868d-1bbbf69d05fa op=livefrontendIP0_monitor_0
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice: process_pe_message: Calculated Transition 0: /var/lib/pacemaker/pengine/pe-input-72.bz2
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: services_os_action_execute: Managed AWSVip_meta-data_0 process 1457 exited with rc=0
> Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/24, version=0.7.4)
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: process_lrm_event: LRM operation livefrontendIP0_monitor_0 (call=5, rc=0, cib-update=24, confirmed=true) ok
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: warning: status_from_rc: Action 4 (livefrontendIP0_monitor_0) on jlr-lb-01.aws failed (target: 7 vs. rc: 0): Error
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: abort_transition_graph: match_graph_event:313 - Triggered transition abort (complete=0, node=jlr-lb-01.aws, tag=lrm_rsc_op, id=livefrontendIP0_last_failure_0, magic=0:0;4:0:7:036403dd-11d8-4f25-868d-1bbbf69d05fa, cib=0.7.4) : Event failed
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: match_graph_event: Action livefrontendIP0_monitor_0 (4) confirmed on jlr-lb-01.aws (rc=4)
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: process_graph_event: Detected action (0.4) livefrontendIP0_monitor_0.5=ok: failed
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: te_rsc_command: Initiating action 3: probe_complete probe_complete on jlr-lb-01.aws (local) - no waiting
> Sep 16 01:08:53 [1336] jlr-lb-01.aws attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
> Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='jlr-lb-01.aws']//transient_attributes//nvpair[@name='probe_complete']: No such device or address (rc=-6, origin=local/attrd/2, version=0.7.4)
> Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section /cib: OK (rc=0, origin=local/attrd/3, version=0.7.4)
> Sep 16 01:08:53 [1336] jlr-lb-01.aws attrd: notice: attrd_perform_update: Sent update 4: probe_complete=true
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: te_rsc_command: Action 3 confirmed - no wait
> Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/4, version=0.7.5)
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: run_graph: Transition 0 (Complete=2, Pending=0, Fired=0, Skipped=3, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-72.bz2): Stopped
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
> Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/25, version=0.7.5)
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: determine_online_status: Node jlr-lb-01.aws is online
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: unpack_rsc_op: Operation monitor found resource livefrontendIP0 active on jlr-lb-01.aws
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: native_print: livefrontendIP0 (ocf::heartbeat:AWSVip): Started jlr-lb-01.aws
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: RecurringOp: Start recurring monitor (60s) for livefrontendIP0 on jlr-lb-01.aws
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: LogActions: Leave livefrontendIP0 (Started jlr-lb-01.aws)
> Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice: process_pe_message: Calculated Transition 1: /var/lib/pacemaker/pengine/pe-input-73.bz2
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1410829733-10) derived from /var/lib/pacemaker/pengine/pe-input-73.bz2
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: te_rsc_command: Initiating action 6: monitor livefrontendIP0_monitor_60000 on jlr-lb-01.aws (local)
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_lrm_rsc_op: Performing key=6:1:0:036403dd-11d8-4f25-868d-1bbbf69d05fa op=livefrontendIP0_monitor_60000
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: process_lrm_event: LRM operation livefrontendIP0_monitor_60000 (call=8, rc=0, cib-update=26, confirmed=false) ok
> Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/26, version=0.7.6)
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: match_graph_event: Action livefrontendIP0_monitor_60000 (6) confirmed on jlr-lb-01.aws (rc=0)
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: run_graph: Transition 1 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-73.bz2): Complete
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
> Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
> Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1525 id=86b4712a-3b41-4580-900f-ad1578b844f5
> Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.7.6)
> Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1526 id=0874dae6-6185-435c-9747-5910c6465083
> Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1532 id=b073a7c8-59df-4a08-81a4-ef62e37b24a3
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1533 id=d7397d96-50d6-4813-8ffd-4a537ed66d73
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1534 id=3aa7a511-0a1c-4b53-b3c8-bfb1a37fef37
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1535 id=4149a25e-19af-4e71-bff3-344633251426
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1536 id=cbc05321-d219-4960-8972-dcca7a93d851
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1537 id=fba48fcb-6e64-47db-a5d2-2ac01e617bbf
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1538 id=893198c3-02e6-4241-9533-1b7f0b029ca7
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1539 id=8e0d79fa-bbae-4c55-a491-a70c19ade564
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1540 id=9c416d6a-ef1a-4c7f-8204-27176f30d6f0
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section //nodes: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:12:56 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1562 id=a1a77b50-c091-42fb-b5b7-ad2266a03147
> Sep 16 01:12:56 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:12:56 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:13:11 [1333] jlr-lb-01.aws cib: info: crm_client_new: Connecting 0x1bc6080 for uid=0 gid=0 pid=1574 id=9cd5d793-35e3-4c07-9442-70f8c697f278
> Sep 16 01:13:11 [1333] jlr-lb-01.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_verify/2, version=0.7.6)
> Sep 16 01:13:11 [1333] jlr-lb-01.aws cib: info: crm_client_destroy: Destroying 0 events
>
> -------------------------------------------------------------------------------------------
> NODE 2 LOGS
> -------------------------------------------------------------------------------------------
> Sep 16 01:07:56 corosync [MAIN ] Corosync Cluster Engine ('1.4.1'): started and ready to provide service.
> Sep 16 01:07:56 corosync [MAIN ] Corosync built-in features: nss dbus rdma snmp
> Sep 16 01:07:56 corosync [MAIN ] Successfully read config from /etc/cluster/cluster.conf
> Sep 16 01:07:56 corosync [MAIN ] Successfully parsed cman config
> Sep 16 01:07:56 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
> Sep 16 01:07:56 corosync [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> Sep 16 01:07:56 corosync [TOTEM ] The network interface [10.235.48.124] is now up.
> Sep 16 01:07:57 corosync [QUORUM] Using quorum provider quorum_cman
> Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
> Sep 16 01:07:57 corosync [CMAN ] CMAN 3.0.12.1 (built Apr 14 2014 09:36:10) started
> Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync CMAN membership service 2.90
> Sep 16 01:07:57 corosync [SERV ] Service engine loaded: openais checkpoint service B.01.01
> Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync extended virtual synchrony service
> Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync configuration service
> Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync cluster closed process group service v1.01
> Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync cluster config database access v1.01
> Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync profile loading service
> Sep 16 01:07:57 corosync [QUORUM] Using quorum provider quorum_cman
> Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync cluster quorum service v0.1
> Sep 16 01:07:57 corosync [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
> Sep 16 01:07:57 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.
> Sep 16 01:07:57 corosync [QUORUM] Members[1]: 2
> Sep 16 01:07:57 corosync [QUORUM] Members[1]: 2
> Sep 16 01:07:57 corosync [CPG ] chosen downlist: sender r(0) ip(10.235.48.124) ; members(old:0 left:0)
> Sep 16 01:07:57 corosync [MAIN ] Completed service synchronization, ready to provide service.
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: notice: main: Starting Pacemaker 1.1.10-14.el6_5.3 (Build: 368c726): generated-manpages agent-manpages ascii-docs publican-docs ncurses libqb-logging libqb-ipc nagios corosync-plugin cman
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: main: Maximum core file size is: 18446744073709551615
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: qb_ipcs_us_publish: server name: pacemakerd
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: crm_get_peer: Created entry 1384f338-5022-44c5-9da7-71b08ea75d42/0x13bbfc0 for node (null)/2 (1 total)
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: crm_get_peer: Cannot obtain a UUID for node 2/(null)
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[2] - corosync-cpg is now online
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: cman_node_name: Using CMAN node name jlr-lb-02.aws for 0
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: crm_get_peer: Node 2 is now known as jlr-lb-02.aws
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: crm_get_peer: Node 2 has uuid jlr-lb-02.aws
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child: Using uid=189 and group=189 for process cib
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child: Forked child 1352 for process cib
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child: Forked child 1353 for process stonith-ng
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child: Forked child 1354 for process lrmd
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child: Using uid=189 and group=189 for process attrd
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child: Forked child 1355 for process attrd
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child: Using uid=189 and group=189 for process pengine
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child: Forked child 1356 for process pengine
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child: Forked child 1357 for process crmd
> Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: main: Starting mainloop
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/hacluster
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: get_cluster_type: Verifying cluster type: 'cman'
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: get_cluster_type: Assuming an active 'cman' cluster
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.xml (digest: /var/lib/pacemaker/cib/cib.xml.sig)
> Sep 16 01:08:06 [1355] jlr-lb-02.aws attrd: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
> Sep 16 01:08:06 [1356] jlr-lb-02.aws pengine: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/hacluster
> Sep 16 01:08:06 [1356] jlr-lb-02.aws pengine: info: qb_ipcs_us_publish: server name: pengine
> Sep 16 01:08:06 [1356] jlr-lb-02.aws pengine: info: main: Starting pengine
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: validate_with_relaxng: Creating RNG parser context
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/root
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: get_cluster_type: Verifying cluster type: 'cman'
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: get_cluster_type: Assuming an active 'cman' cluster
> Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/root
> Sep 16 01:08:06 [1354] jlr-lb-02.aws lrmd: info: crm_log_init: Changed active directory to /var/lib/pacemaker/cores/root
> Sep 16 01:08:06 [1354] jlr-lb-02.aws lrmd: info: qb_ipcs_us_publish: server name: lrmd
> Sep 16 01:08:06 [1354] jlr-lb-02.aws lrmd: info: main: Starting
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
> Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: notice: main: CRM Git Version: 368c726
> Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info: do_log: FSA: Input I_STARTUP from crmd_init() received in state S_STARTING
> Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info: get_cluster_type: Verifying cluster type: 'cman'
> Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info: get_cluster_type: Assuming an active 'cman' cluster
> Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info: crm_ipc_connect: Could not establish cib_shm connection: Connection refused (111)
> Sep 16 01:08:06 [1355] jlr-lb-02.aws attrd: notice: main: Starting mainloop...
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_get_peer: Created entry f202c0cf-5a4a-4336-98e4-8ee64511697e/0x9fe890 for node (null)/2 (1 total)
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_get_peer: Cannot obtain a UUID for node 2/(null)
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[2] - corosync-cpg is now online
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: cman_node_name: Using CMAN node name jlr-lb-02.aws for 0
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: init_cs_connection_once: Connection to 'cman': established
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: cman_node_name: Using CMAN node name jlr-lb-02.aws for 0
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_get_peer: Node 2 is now known as jlr-lb-02.aws
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_get_peer: Node 2 has uuid jlr-lb-02.aws
> Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_ipc_connect: Could not establish cib_rw connection: Connection refused (111)
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: startCib: CIB Initialization completed successfully
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_get_peer: Created entry 63b7c4e6-ee9c-4692-8c00-d72cc1a72625/0x24fe860 for node (null)/2 (1 total)
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_get_peer: Cannot obtain a UUID for node 2/(null)
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[2] - corosync-cpg is now online
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: cman_node_name: Using CMAN node name jlr-lb-02.aws for 0
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: init_cs_connection_once: Connection to 'cman': established
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: cman_node_name: Using CMAN node name jlr-lb-02.aws for 0
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_get_peer: Node 2 is now known as jlr-lb-02.aws
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_get_peer: Node 2 has uuid jlr-lb-02.aws
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: qb_ipcs_us_publish: server name: cib_ro
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: qb_ipcs_us_publish: server name: cib_rw
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: qb_ipcs_us_publish: server name: cib_shm
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: cib_init: Starting cib mainloop
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: pcmk_cpg_membership: Joined[0.0] cib.2
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: pcmk_cpg_membership: Member[0.0] cib.2
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-20.raw
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: write_cib_contents: Wrote version 0.7.0 of the CIB to disk (digest: 9a6dcf33ae0aeb2e09a2e348ec5ec53a)
> Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.EsVx7U (digest: /var/lib/pacemaker/cib/cib.K2HEcT)
> Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2501730 for uid=0 gid=0 pid=1357 id=1b04dbca-8326-4e35-ab40-2a5a8f55a597
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_cib_control: CIB connection established
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
> Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/2, version=0.7.0)
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer: Created entry 659a46dc-8ff4-44cd-94b3-0d4d3caaf300/0x1070580 for node (null)/2 (1 total)
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer: Cannot obtain a UUID for node 2/(null)
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_update_peer_proc: cluster_connect_cpg: Node (null)[2] - corosync-cpg is now online
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: cman_node_name: Using CMAN node name jlr-lb-02.aws for 0
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: init_cs_connection_once: Connection to 'cman': established
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: cman_node_name: Using CMAN node name jlr-lb-02.aws for 0
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer: Node 2 is now known as jlr-lb-02.aws
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: peer_update_callback: jlr-lb-02.aws is now (null)
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer: Node 2 has uuid jlr-lb-02.aws
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: init_cman_connection: Configuring Pacemaker to obtain quorum from cman
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: cman_event_callback: Membership 40: quorum still lost
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer: Created entry 68a0b84f-e28b-47d8-8c54-4cbcfc3c25bf/0x1073020 for node jlr-lb-01.aws/1 (2 total)
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer: Node 1 is now known as jlr-lb-01.aws
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: peer_update_callback: jlr-lb-01.aws is now (null)
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer: Node 1 has uuid jlr-lb-01.aws
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: cman_event_callback: State of node jlr-lb-01.aws[1] is still unknown
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: notice: crm_update_peer_state: cman_event_callback: Node jlr-lb-02.aws[2] - state is now member (was (null))
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: peer_update_callback: jlr-lb-02.aws is now member (was (null))
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: cman_node_name: Using CMAN node name jlr-lb-02.aws for 0
> Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/3, version=0.7.0)
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_ha_control: Connected to the cluster
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: lrmd_ipc_connect: Connecting to lrmd
> Sep 16 01:08:07 [1354] jlr-lb-02.aws lrmd: info: crm_client_new: Connecting 0x1289aa0 for uid=0 gid=0 pid=1357 id=88fb7ff9-1bf4-450c-a658-184bb2689b5d
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_lrm_control: LRM connection established
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_started: Delaying start, Config not read (0000000000000040)
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_started: Delaying start, Config not read (0000000000000040)
> Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/4, version=0.7.0)
> Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x234f200 for uid=0 gid=0 pid=1353 id=5885c19b-6f02-4f68-953b-39583be06e90
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: pcmk_cpg_membership: Joined[0.0] crmd.2
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: pcmk_cpg_membership: Member[0.0] crmd.2
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_started: Delaying start, Config not read (0000000000000040)
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: qb_ipcs_us_publish: server name: crmd
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: notice: do_started: The local CRM is operational
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_log: FSA: Input I_PENDING from do_started() received in state S_STARTING
> Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: notice: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
> Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_slave operation for section 'all': OK (rc=0, origin=local/crmd/5, version=0.7.0)
> Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: notice: setup_cib: Watching for stonith topology changes
> Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info: qb_ipcs_us_publish: server name: stonith-ng
> Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info: main: Starting stonith-ng mainloop
> Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info: pcmk_cpg_membership: Joined[0.0] stonith-ng.2
> Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info: pcmk_cpg_membership: Member[0.0] stonith-ng.2
> Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/2, version=0.7.0)
> Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info: init_cib_cache_cb: Updating device list from the cib: init
> Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: notice: unpack_config: On loss of CCM Quorum: Ignore
> Sep 16 01:08:09 [1353] jlr-lb-02.aws stonith-ng: info: crm_client_new: Connecting 0xb053b0 for uid=0 gid=0 pid=1357 id=13c4ef82-44d1-42b7-99ce-8985d748b2e8
> Sep 16 01:08:09 [1353] jlr-lb-02.aws stonith-ng: info: stonith_command: Processed register from crmd.1357: OK (0)
> Sep 16 01:08:09 [1353] jlr-lb-02.aws stonith-ng: info: stonith_command: Processed st_notify from crmd.1357: OK (0)
> Sep 16 01:08:09 [1353] jlr-lb-02.aws stonith-ng: info: stonith_command: Processed st_notify from crmd.1357: OK (0)
> Sep 16 01:08:11 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x25020e0 for uid=189 gid=0 pid=1355 id=87268c89-a3d2-4c99-baa6-927b4e614c6a
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: warning: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ]
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_log: FSA: Input I_ELECTION_DC from do_election_check() received in state S_ELECTION
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_te_control: Registering TE UUID: ae731050-4f86-455d-b180-c8948bca0331
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: set_graph_functions: Setting custom graph functions
> Sep 16 01:08:28 [1356] jlr-lb-02.aws pengine: info: crm_client_new: Connecting 0x1cf1260 for uid=0 gid=0 pid=1357 id=a4da1ab1-aecb-45fb-87ad-88c3f0fbc212
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_dc_takeover: Taking over DC status for this partition
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_readwrite: We are now in R/W mode
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/6, version=0.7.0)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/7, version=0.7.0)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='dc-version']: OK (rc=0, origin=local/crmd/8, version=0.7.0)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/9, version=0.7.0)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='cluster-infrastructure']: OK (rc=0, origin=local/crmd/10, version=0.7.0)
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: join_make_offer: Making join offers based on membership 40
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: join_make_offer: join-1: Sending offer to jlr-lb-02.aws
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: crm_update_peer_join: join_make_offer: Node jlr-lb-02.aws[2] - join-1 phase 0 -> 1
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: update_dc: Set DC to jlr-lb-02.aws (3.0.7)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/11, version=0.7.0)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/12, version=0.7.0)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/13, version=0.7.0)
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node jlr-lb-02.aws[2] - join-1 phase 1 -> 2
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: crm_update_peer_expected: do_dc_join_filter_offer: Node jlr-lb-02.aws[2] - expected state is now member
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: crmd_join_phase_log: join-1: jlr-lb-02.aws=integrated
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: crmd_join_phase_log: join-1: jlr-lb-01.aws=none
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_dc_join_finalize: join-1: Syncing our CIB to the rest of the cluster
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cman_node_name: Using CMAN node name jlr-lb-02.aws for 0
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_sync operation for section 'all': OK (rc=0, origin=local/crmd/14, version=0.7.0)
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: crm_update_peer_join: finalize_join_for: Node jlr-lb-02.aws[2] - join-1 phase 2 -> 3
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='jlr-lb-02.aws']/transient_attributes
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: update_attrd: Connecting to attrd... 5 retries remaining
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/15, version=0.7.0)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='jlr-lb-02.aws']/transient_attributes: OK (rc=0, origin=local/crmd/16, version=0.7.0)
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: crm_update_peer_join: do_dc_join_ack: Node jlr-lb-02.aws[2] - join-1 phase 3 -> 4
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_dc_join_ack: join-1: Updating node state to member for jlr-lb-02.aws
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='jlr-lb-02.aws']/lrm
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='jlr-lb-02.aws']/lrm: OK (rc=0, origin=local/crmd/17, version=0.7.0)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/18, version=0.7.1)
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
> Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: abort_transition_graph: do_te_invoke:151 - Triggered transition abort (complete=1) : Peer Cancelled
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/19, version=0.7.1)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/20, version=0.7.2)
> Sep 16 01:08:28 [1355] jlr-lb-02.aws attrd: notice: attrd_local_callback: Sending full refresh (origin=crmd)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/21, version=0.7.3)
> Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/22, version=0.7.3)
> Sep 16 01:08:29 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/23, version=0.7.3)
> Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
> Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: info: determine_online_status: Node jlr-lb-02.aws is online
> Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: info: native_print: livefrontendIP0 (ocf::heartbeat:AWSVip): Stopped
> Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: info: RecurringOp: Start recurring monitor (60s) for livefrontendIP0 on jlr-lb-02.aws
> Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: notice: LogActions: Start livefrontendIP0 (jlr-lb-02.aws)
> Sep 16 01:08:29 [1357] jlr-lb-02.aws crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
> Sep 16 01:08:29 [1357] jlr-lb-02.aws crmd: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1410829709-7) derived from /var/lib/pacemaker/pengine/pe-input-30.bz2
> Sep 16 01:08:29 [1357] jlr-lb-02.aws crmd: notice: te_rsc_command: Initiating action 4: monitor livefrontendIP0_monitor_0 on jlr-lb-02.aws (local)
> Sep 16 01:08:29 [1354] jlr-lb-02.aws lrmd: info: process_lrmd_get_rsc_info: Resource 'livefrontendIP0' not found (0 active resources)
> Sep 16 01:08:29 [1354] jlr-lb-02.aws lrmd: info: process_lrmd_rsc_register: Added 'livefrontendIP0' to the rsc list (1 active resources)
> Sep 16 01:08:29 [1357] jlr-lb-02.aws crmd: info: do_lrm_rsc_op: Performing key=4:0:7:ae731050-4f86-455d-b180-c8948bca0331 op=livefrontendIP0_monitor_0
> Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: notice: process_pe_message: Calculated Transition 0: /var/lib/pacemaker/pengine/pe-input-30.bz2
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: services_os_action_execute: Managed AWSVip_meta-data_0 process 1477 exited with rc=0
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: process_lrm_event: LRM operation livefrontendIP0_monitor_0 (call=5, rc=0, cib-update=24, confirmed=true) ok
> Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/24, version=0.7.4)
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: warning: status_from_rc: Action 4 (livefrontendIP0_monitor_0) on jlr-lb-02.aws failed (target: 7 vs. rc: 0): Error
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: abort_transition_graph: match_graph_event:313 - Triggered transition abort (complete=0, node=jlr-lb-02.aws, tag=lrm_rsc_op, id=livefrontendIP0_last_failure_0, magic=0:0;4:0:7:ae731050-4f86-455d-b180-c8948bca0331, cib=0.7.4) : Event failed
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: match_graph_event: Action livefrontendIP0_monitor_0 (4) confirmed on jlr-lb-02.aws (rc=4)
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: process_graph_event: Detected action (0.4) livefrontendIP0_monitor_0.5=ok: failed
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: te_rsc_command: Initiating action 3: probe_complete probe_complete on jlr-lb-02.aws (local) - no waiting
> Sep 16 01:08:30 [1355] jlr-lb-02.aws attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
> Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='jlr-lb-02.aws']//transient_attributes//nvpair[@name='probe_complete']: No such device or address (rc=-6, origin=local/attrd/2, version=0.7.4)
> Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section /cib: OK (rc=0, origin=local/attrd/3, version=0.7.4)
> Sep 16 01:08:30 [1355] jlr-lb-02.aws attrd: notice: attrd_perform_update: Sent update 4: probe_complete=true
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: te_rsc_command: Action 3 confirmed - no wait
> Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/4, version=0.7.5)
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: run_graph: Transition 0 (Complete=2, Pending=0, Fired=0, Skipped=3, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-30.bz2): Stopped
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
> Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/25, version=0.7.5)
> Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
> Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info: determine_online_status: Node jlr-lb-02.aws is online
> Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info: unpack_rsc_op: Operation monitor found resource livefrontendIP0 active on jlr-lb-02.aws
> Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info: native_print: livefrontendIP0 (ocf::heartbeat:AWSVip): Started jlr-lb-02.aws
> Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info: RecurringOp: Start recurring monitor (60s) for livefrontendIP0 on jlr-lb-02.aws
> Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info: LogActions: Leave livefrontendIP0 (Started jlr-lb-02.aws)
> Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: notice: process_pe_message: Calculated Transition 1: /var/lib/pacemaker/pengine/pe-input-31.bz2
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1410829710-10) derived from /var/lib/pacemaker/pengine/pe-input-31.bz2
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: te_rsc_command: Initiating action 6: monitor livefrontendIP0_monitor_60000 on jlr-lb-02.aws (local)
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: do_lrm_rsc_op: Performing key=6:1:0:ae731050-4f86-455d-b180-c8948bca0331 op=livefrontendIP0_monitor_60000
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: process_lrm_event: LRM operation livefrontendIP0_monitor_60000 (call=8, rc=0, cib-update=26, confirmed=false) ok
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: match_graph_event: Action livefrontendIP0_monitor_60000 (6) confirmed on jlr-lb-02.aws (rc=0)
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: run_graph: Transition 1 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-31.bz2): Complete
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
> Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
> Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/26, version=0.7.6)
> Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1490 id=1f235bdf-d2ee-47d9-af5c-c6ae4130f59a
> Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.7.6)
> Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1491 id=0df0f7d7-f92a-4d91-b242-6e645940b3b8
> Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:10:21 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1530 id=3a77dc76-433c-472c-a553-c633d4472e82
> Sep 16 01:10:21 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:10:21 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:10:57 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1546 id=7479bbf2-a878-4863-ab28-6e6924d156f0
> Sep 16 01:10:57 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_verify/2, version=0.7.6)
> Sep 16 01:10:57 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1615 id=028a6604-3038-46e6-9162-83c21670211f
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1616 id=add9ee90-7077-4970-9334-bc96cf2d4dac
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1617 id=43c28e33-aa58-4dc8-8608-23b43eb9a576
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1618 id=9ccbf6c9-9b02-480b-b778-bc2b38d28229
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1619 id=3da1e9e9-ff2e-4c40-9031-589b3dfc134b
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1620 id=d865033d-6df5-4b33-9bf9-43f7a62ea434
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1621 id=1a10c96d-95cc-47c4-9e30-b1c2e4bac374
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section //constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1622 id=6e22e016-d80e-4937-851c-a0b9692322a5
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new: Connecting 0x2508a70 for uid=0 gid=0 pid=1623 id=f4b3fdbf-9ad1-4351-8b30-95ead3f368fa
> Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: cib_process_request: Completed cib_query operation for section //nodes: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
> Sep 16 01:17:33 [1352] jlr-lb-02.aws cib: info: crm_client_destroy: Destroying 0 events
> --
> - Jeff
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140916/f59dffbf/attachment-0003.sig>
More information about the Pacemaker
mailing list