[Pacemaker] Pacemaker/Corosync members do not join cluster
Jeffrey Roberts
jeffrey.l.roberts at gmail.com
Mon Sep 15 21:24:35 EDT 2014
Greetings,
I have been Googling for hours now, and I can't seem to find an answer to
the challenge I face today, which is no matter what I have tried I can not
get the nodes to join or sync with each other... Please let me know if
there is anymore output I can provide, and thank you very much for your
help, it is greatly appreciated! btw, I am hosting on Amazon EC2, and
please note I have tried both internal and public IPs with no avail...
also, there is no selinux installed, and iptables is disabled, and the ec2
firewall is wide open for TCP and UDP....
Here is the output from my `pcs status` on both nodes
-------------------------------------------------------------------------------------------
[~]
[root at jlr-lb-01] -> pcs status
Cluster name: jlr-lb
Last updated: Tue Sep 16 01:12:13 2014
Last change: Tue Sep 16 00:32:28 2014 via cibadmin on jlr-lb-01.aws
Stack: cman
Current DC: jlr-lb-01.aws - partition WITHOUT quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
1 Resources configured
Online: [ jlr-lb-01.aws ]
OFFLINE: [ jlr-lb-02.aws ]
Full list of resources:
livefrontendIP0 (ocf::heartbeat:AWSVip): Started jlr-lb-01.aws
-------------------------------------------------------------------------------------------
[~]
[root at jlr-lb-02] -> pcs status
Cluster name: jlr-lb
Last updated: Tue Sep 16 01:09:27 2014
Last change: Tue Sep 16 01:04:35 2014 via cibadmin on jlr-lb-02.aws
Stack: cman
Current DC: jlr-lb-02.aws - partition WITHOUT quorum
Version: 1.1.10-14.el6_5.3-368c726
2 Nodes configured
1 Resources configured
Online: [ jlr-lb-02.aws ]
OFFLINE: [ jlr-lb-01.aws ]
Full list of resources:
livefrontendIP0 (ocf::heartbeat:AWSVip): Started jlr-lb-02.aws
-------------------------------------------------------------------------------------------
Here is the output from all the commands I could think of
-------------------------------------------------------------------------------------------
NODE 1
-------------------------------------------------------------------------------------------
[~]
[root at jlr-lb-01] -> pcs config
Cluster Name: jlr-lb
Corosync Nodes:
Pacemaker Nodes:
jlr-lb-01.aws jlr-lb-02.aws
Resources:
Resource: livefrontendIP0 (class=ocf provider=heartbeat type=AWSVip)
Operations: monitor interval=60s (livefrontendIP0-monitor-interval-60s)
Stonith Devices:
Fencing Levels:
Location Constraints:
Resource: livefrontendIP0
Enabled on: jlr-lb-01.aws (score:INFINITY)
(id:location-livefrontendIP0-jlr-lb-01.aws-INFINITY)
Ordering Constraints:
Colocation Constraints:
Cluster Properties:
cluster-infrastructure: cman
dc-version: 1.1.10-14.el6_5.3-368c726
no-quorum-policy: ignore
stonith-enabled: false
[~]
[root at jlr-lb-01] -> netstat -nlpa | grep corosync
udp 0 0 10.38.24.5:5404 0.0.0.0:*
889/corosync
udp 0 0 10.38.24.5:5405 0.0.0.0:*
889/corosync
udp 0 0 239.192.99.112:5405 0.0.0.0:*
889/corosync
unix 2 [ ACC ] STREAM LISTENING 10089 889/corosync
@corosync.ipc
unix 2 [ ACC ] STREAM LISTENING 10109 889/corosync
/var/run/cman_client
unix 2 [ ACC ] STREAM LISTENING 10111 889/corosync
/var/run/cman_admin
unix 3 [ ] STREAM CONNECTED 12443 889/corosync
/var/run/cman_client
unix 3 [ ] STREAM CONNECTED 12423 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12276 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12244 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12239 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12142 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12136 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 11023 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10874 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10691 889/corosync
/var/run/cman_client
unix 3 [ ] STREAM CONNECTED 10692 889/corosync
/var/run/cman_admin
unix 3 [ ] STREAM CONNECTED 10655 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10609 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10603 889/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10374 889/corosync
/var/run/cman_client
unix 3 [ ] STREAM CONNECTED 10375 889/corosync
/var/run/cman_admin
unix 3 [ ] STREAM CONNECTED 10290 889/corosync
/var/run/cman_client
unix 3 [ ] STREAM CONNECTED 10288 889/corosync
/var/run/cman_admin
unix 3 [ ] DGRAM 10106 889/corosync
unix 3 [ ] DGRAM 10105 889/corosync
unix 2 [ ] DGRAM 10085 889/corosync
[~]
[root at jlr-lb-01] -> cman_tool nodes
Node Sts Inc Joined Name
1 M 52 2014-09-16 01:08:19 jlr-lb-01.aws
2 X 0 jlr-lb-02.aws
[~]
[root at jlr-lb-01] -> cibadmin -Ql
<cib epoch="7" num_updates="6" admin_epoch="0"
validate-with="pacemaker-1.2" cib-last-written="Tue Sep 16 00:32:28 2014"
update-origin="jlr-lb-01.aws" update-client="cibadmin"
crm_feature_set="3.0.7" have-quorum="0" dc-uuid="jlr-lb-01.aws">
<configuration>
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
<nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
value="1.1.10-14.el6_5.3-368c726"/>
<nvpair id="cib-bootstrap-options-cluster-infrastructure"
name="cluster-infrastructure" value="cman"/>
<nvpair id="cib-bootstrap-options-stonith-enabled"
name="stonith-enabled" value="false"/>
<nvpair id="cib-bootstrap-options-no-quorum-policy"
name="no-quorum-policy" value="ignore"/>
</cluster_property_set>
</crm_config>
<nodes>
<node id="jlr-lb-01.aws" uname="jlr-lb-01.aws"/>
<node id="jlr-lb-02.aws" uname="jlr-lb-02.aws"/>
</nodes>
<resources>
<primitive class="ocf" id="livefrontendIP0" provider="heartbeat"
type="AWSVip">
<instance_attributes id="livefrontendIP0-instance_attributes"/>
<operations>
<op id="livefrontendIP0-monitor-interval-60s" interval="60s"
name="monitor"/>
</operations>
</primitive>
</resources>
<constraints>
<rsc_location id="location-livefrontendIP0-jlr-lb-01.aws-INFINITY"
node="jlr-lb-01.aws" rsc="livefrontendIP0" score="INFINITY"/>
</constraints>
</configuration>
<status>
<node_state id="jlr-lb-01.aws" uname="jlr-lb-01.aws" in_ccm="true"
crmd="online" crm-debug-origin="do_update_resource" join="member"
expected="member">
<lrm id="jlr-lb-01.aws">
<lrm_resources>
<lrm_resource id="livefrontendIP0" type="AWSVip" class="ocf"
provider="heartbeat">
<lrm_rsc_op id="livefrontendIP0_last_failure_0"
operation_key="livefrontendIP0_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.7"
transition-key="4:0:7:036403dd-11d8-4f25-868d-1bbbf69d05fa"
transition-magic="0:0;4:0:7:036403dd-11d8-4f25-868d-1bbbf69d05fa"
call-id="5" rc-code="0" op-status="0" interval="0" last-run="1410829733"
last-rc-change="1410829733" exec-time="65" queue-time="0"
op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8" op-force-restart=" state "
op-restart-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
<lrm_rsc_op id="livefrontendIP0_monitor_60000"
operation_key="livefrontendIP0_monitor_60000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.7"
transition-key="6:1:0:036403dd-11d8-4f25-868d-1bbbf69d05fa"
transition-magic="0:0;6:1:0:036403dd-11d8-4f25-868d-1bbbf69d05fa"
call-id="8" rc-code="0" op-status="0" interval="60000"
last-rc-change="1410829733" exec-time="28" queue-time="0"
op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
</lrm_resource>
</lrm_resources>
</lrm>
<transient_attributes id="jlr-lb-01.aws">
<instance_attributes id="status-jlr-lb-01.aws">
<nvpair id="status-jlr-lb-01.aws-probe_complete"
name="probe_complete" value="true"/>
</instance_attributes>
</transient_attributes>
</node_state>
<node_state id="jlr-lb-02.aws" uname="jlr-lb-02.aws" crmd="offline"
join="down" crm-debug-origin="do_state_transition"/>
</status>
</cib>
[~]
[root at jlr-lb-01] -> corosync-objctl | grep member
runtime.totem.pg.mrp.srp.members.1.ip=r(0) ip(10.38.24.5)
runtime.totem.pg.mrp.srp.members.1.join_count=1
runtime.totem.pg.mrp.srp.members.1.status=joined
[~]
[root at jlr-lb-01] -> crm_verify -L -V
[~]
[root at jlr-lb-01] -> cat /etc/corosync/corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: off
threads: 0
join: 1000
rrp_mode: passive
interface {
member {
memberaddr: 10.38.24.5
}
member {
memberaddr: 54.244.91.9
# memberaddr: 10.235.48.124
}
ringnumber: 0
bindnetaddr: 10.38.24.0
mcastport: 5405
ttl: 1
}
transport: udpu
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: yes
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
[~]
[root at jlr-lb-01] -> tcpdump -i eth0 -p udp -envv "port 5405"
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535
bytes
01:13:29.506832 22:00:0a:26:18:05 > 01:00:5e:40:63:70, ethertype IPv4
(0x0800), length 161: (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto
UDP (17), length 147)
10.38.24.5.hpoms-dps-lstn > 239.192.99.112.netsupport: [bad udp cksum
310e!] UDP, length 119
01:13:31.410357 22:00:0a:26:18:05 > 01:00:5e:40:63:70, ethertype IPv4
(0x0800), length 161: (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto
UDP (17), length 147)
10.38.24.5.hpoms-dps-lstn > 239.192.99.112.netsupport: [bad udp cksum
6f85!] UDP, length 119
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
-------------------------------------------------------------------------------------------
NODE 2
-------------------------------------------------------------------------------------------
[~]
[root at jlr-lb-02] -> pcs config
Cluster Name: jlr-lb
Corosync Nodes:
Pacemaker Nodes:
jlr-lb-01.aws jlr-lb-02.aws
Resources:
Resource: livefrontendIP0 (class=ocf provider=heartbeat type=AWSVip)
Operations: monitor interval=60s (livefrontendIP0-monitor-interval-60s)
Stonith Devices:
Fencing Levels:
Location Constraints:
Resource: livefrontendIP0
Enabled on: jlr-lb-01.aws (score:INFINITY)
(id:location-livefrontendIP0-jlr-lb-01.aws-INFINITY)
Ordering Constraints:
Colocation Constraints:
Cluster Properties:
cluster-infrastructure: cman
dc-version: 1.1.10-14.el6_5.3-368c726
no-quorum-policy: ignore
stonith-enabled: false
[~]
[root at jlr-lb-02] -> netstat -nlpa | grep corosync
udp 0 0 10.235.48.124:5404 0.0.0.0:*
892/corosync
udp 0 0 10.235.48.124:5405 0.0.0.0:*
892/corosync
udp 0 0 239.192.99.112:5405 0.0.0.0:*
892/corosync
unix 2 [ ACC ] STREAM LISTENING 10095 892/corosync
@corosync.ipc
unix 2 [ ACC ] STREAM LISTENING 10125 892/corosync
/var/run/cman_client
unix 2 [ ACC ] STREAM LISTENING 10127 892/corosync
/var/run/cman_admin
unix 3 [ ] STREAM CONNECTED 12563 892/corosync
/var/run/cman_client
unix 3 [ ] STREAM CONNECTED 12551 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12480 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12454 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12435 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12268 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 12262 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 11026 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10877 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10694 892/corosync
/var/run/cman_client
unix 3 [ ] STREAM CONNECTED 10695 892/corosync
/var/run/cman_admin
unix 3 [ ] STREAM CONNECTED 10672 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10613 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10607 892/corosync
@corosync.ipc
unix 3 [ ] STREAM CONNECTED 10380 892/corosync
/var/run/cman_client
unix 3 [ ] STREAM CONNECTED 10381 892/corosync
/var/run/cman_admin
unix 3 [ ] STREAM CONNECTED 10298 892/corosync
/var/run/cman_client
unix 3 [ ] STREAM CONNECTED 10293 892/corosync
/var/run/cman_admin
unix 3 [ ] DGRAM 10112 892/corosync
unix 3 [ ] DGRAM 10111 892/corosync
unix 2 [ ] DGRAM 10091 892/corosync
[~]
[root at jlr-lb-02] -> cman_tool nodes
Node Sts Inc Joined Name
1 X 0 jlr-lb-01.aws
2 M 40 2014-09-16 01:07:57 jlr-lb-02.aws
[~]
[root at jlr-lb-02] -> cibadmin -Ql
<cib epoch="7" num_updates="6" admin_epoch="0"
validate-with="pacemaker-1.2" cib-last-written="Tue Sep 16 01:04:35 2014"
update-origin="jlr-lb-02.aws" update-client="cibadmin"
crm_feature_set="3.0.7" have-quorum="0" dc-uuid="jlr-lb-02.aws">
<configuration>
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
<nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
value="1.1.10-14.el6_5.3-368c726"/>
<nvpair id="cib-bootstrap-options-cluster-infrastructure"
name="cluster-infrastructure" value="cman"/>
<nvpair id="cib-bootstrap-options-stonith-enabled"
name="stonith-enabled" value="false"/>
<nvpair id="cib-bootstrap-options-no-quorum-policy"
name="no-quorum-policy" value="ignore"/>
</cluster_property_set>
</crm_config>
<nodes>
<node id="jlr-lb-01.aws" uname="jlr-lb-01.aws"/>
<node id="jlr-lb-02.aws" uname="jlr-lb-02.aws"/>
</nodes>
<resources>
<primitive class="ocf" id="livefrontendIP0" provider="heartbeat"
type="AWSVip">
<instance_attributes id="livefrontendIP0-instance_attributes"/>
<operations>
<op id="livefrontendIP0-monitor-interval-60s" interval="60s"
name="monitor"/>
</operations>
</primitive>
</resources>
<constraints>
<rsc_location id="location-livefrontendIP0-jlr-lb-01.aws-INFINITY"
node="jlr-lb-01.aws" rsc="livefrontendIP0" score="INFINITY"/>
</constraints>
</configuration>
<status>
<node_state id="jlr-lb-02.aws" uname="jlr-lb-02.aws" in_ccm="true"
crmd="online" crm-debug-origin="do_update_resource" join="member"
expected="member">
<lrm id="jlr-lb-02.aws">
<lrm_resources>
<lrm_resource id="livefrontendIP0" type="AWSVip" class="ocf"
provider="heartbeat">
<lrm_rsc_op id="livefrontendIP0_last_failure_0"
operation_key="livefrontendIP0_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.7"
transition-key="4:0:7:ae731050-4f86-455d-b180-c8948bca0331"
transition-magic="0:0;4:0:7:ae731050-4f86-455d-b180-c8948bca0331"
call-id="5" rc-code="0" op-status="0" interval="0" last-run="1410829709"
last-rc-change="1410829709" exec-time="113" queue-time="0"
op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8" op-force-restart=" state "
op-restart-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
<lrm_rsc_op id="livefrontendIP0_monitor_60000"
operation_key="livefrontendIP0_monitor_60000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.7"
transition-key="6:1:0:ae731050-4f86-455d-b180-c8948bca0331"
transition-magic="0:0;6:1:0:ae731050-4f86-455d-b180-c8948bca0331"
call-id="8" rc-code="0" op-status="0" interval="60000"
last-rc-change="1410829710" exec-time="22" queue-time="0"
op-digest="4811cef7f7f94e3a35a70be7916cb2fd"/>
</lrm_resource>
</lrm_resources>
</lrm>
<transient_attributes id="jlr-lb-02.aws">
<instance_attributes id="status-jlr-lb-02.aws">
<nvpair id="status-jlr-lb-02.aws-probe_complete"
name="probe_complete" value="true"/>
</instance_attributes>
</transient_attributes>
</node_state>
<node_state id="jlr-lb-01.aws" uname="jlr-lb-01.aws" crmd="offline"
join="down" crm-debug-origin="do_state_transition"/>
</status>
</cib>
[~]
[root at jlr-lb-02] -> corosync-objctl | grep member
runtime.totem.pg.mrp.srp.members.2.ip=r(0) ip(10.235.48.124)
runtime.totem.pg.mrp.srp.members.2.join_count=1
runtime.totem.pg.mrp.srp.members.2.status=joined
You have new mail in /var/spool/mail/root
[~]
[root at jlr-lb-02] -> crm_verify -L -V
[~]
[root at jlr-lb-02] -> cat /etc/corosync/corosync.conf
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version: 2
secauth: off
threads: 0
join: 1000
rrp_mode: passive
interface {
member {
memberaddr: 54.244.91.8
# memberaddr: 10.38.24.5
}
member {
memberaddr: 10.235.48.124
}
ringnumber: 0
bindnetaddr: 10.235.48.0
mcastport: 5405
ttl: 1
}
transpot: udpu
}
logging {
fileline: off
to_stderr: no
to_logfile: yes
to_syslog: yes
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
}
}
amf {
mode: disabled
}
[~]
[root at jlr-lb-02] -> tcpdump -i eth0 -p udp -envv "port 5405"
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535
bytes
01:11:52.100121 22:00:0a:eb:30:7c > 01:00:5e:40:63:70, ethertype IPv4
(0x0800), length 161: (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto
UDP (17), length 147)
10.235.48.124.hpoms-dps-lstn > 239.192.99.112.netsupport: [bad udp
cksum e1de!] UDP, length 119
01:11:54.003694 22:00:0a:eb:30:7c > 01:00:5e:40:63:70, ethertype IPv4
(0x0800), length 161: (tos 0x0, ttl 1, id 0, offset 0, flags [DF], proto
UDP (17), length 147)
10.235.48.124.hpoms-dps-lstn > 239.192.99.112.netsupport: [bad udp
cksum 2206!] UDP, length 119
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
-------------------------------------------------------------------------------------------
NODE 1 LOGS
-------------------------------------------------------------------------------------------
Sep 16 01:08:19 corosync [MAIN ] Corosync Cluster Engine ('1.4.1'):
started and ready to provide service.
Sep 16 01:08:19 corosync [MAIN ] Corosync built-in features: nss dbus rdma
snmp
Sep 16 01:08:19 corosync [MAIN ] Successfully read config from
/etc/cluster/cluster.conf
Sep 16 01:08:19 corosync [MAIN ] Successfully parsed cman config
Sep 16 01:08:19 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Sep 16 01:08:19 corosync [TOTEM ] Initializing transmit/receive security:
libtomcrypt SOBER128/SHA1HMAC (mode 0).
Sep 16 01:08:19 corosync [TOTEM ] The network interface [10.38.24.5] is now
up.
Sep 16 01:08:19 corosync [QUORUM] Using quorum provider quorum_cman
Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync cluster
quorum service v0.1
Sep 16 01:08:19 corosync [CMAN ] CMAN 3.0.12.1 (built Apr 14 2014
09:36:10) started
Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync CMAN
membership service 2.90
Sep 16 01:08:19 corosync [SERV ] Service engine loaded: openais checkpoint
service B.01.01
Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync extended
virtual synchrony service
Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync
configuration service
Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync cluster
closed process group service v1.01
Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync cluster
config database access v1.01
Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync profile
loading service
Sep 16 01:08:19 corosync [QUORUM] Using quorum provider quorum_cman
Sep 16 01:08:19 corosync [SERV ] Service engine loaded: corosync cluster
quorum service v0.1
Sep 16 01:08:19 corosync [MAIN ] Compatibility mode set to whitetank.
Using V1 and V2 of the synchronization engine.
Sep 16 01:08:19 corosync [TOTEM ] A processor joined or left the membership
and a new membership was formed.
Sep 16 01:08:19 corosync [QUORUM] Members[1]: 1
Sep 16 01:08:19 corosync [QUORUM] Members[1]: 1
Sep 16 01:08:19 corosync [CPG ] chosen downlist: sender r(0)
ip(10.38.24.5) ; members(old:0 left:0)
Sep 16 01:08:19 corosync [MAIN ] Completed service synchronization, ready
to provide service.
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: notice: main: Starting
Pacemaker 1.1.10-14.el6_5.3 (Build: 368c726): generated-manpages
agent-manpages ascii-docs publican-docs ncurses libqb-logging libqb-ipc
nagios corosync-plugin cman
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: main: Maximum
core file size is: 18446744073709551615
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info:
qb_ipcs_us_publish: server name: pacemakerd
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: crm_get_peer:
Created entry 9cefa239-c37c-4590-bca9-5a48d7f967d9/0x759fc0 for node
(null)/1 (1 total)
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: crm_get_peer:
Cannot obtain a UUID for node 1/(null)
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info:
crm_update_peer_proc: cluster_connect_cpg: Node (null)[1] - corosync-cpg is
now online
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: cman_node_name:
Using CMAN node name jlr-lb-01.aws for 0
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: crm_get_peer:
Node 1 is now known as jlr-lb-01.aws
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: crm_get_peer:
Node 1 has uuid jlr-lb-01.aws
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child:
Using uid=189 and group=189 for process cib
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child:
Forked child 1333 for process cib
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child:
Forked child 1334 for process stonith-ng
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child:
Forked child 1335 for process lrmd
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child:
Using uid=189 and group=189 for process attrd
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child:
Forked child 1336 for process attrd
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child:
Using uid=189 and group=189 for process pengine
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child:
Forked child 1337 for process pengine
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: start_child:
Forked child 1338 for process crmd
Sep 16 01:08:29 [1327] jlr-lb-01.aws pacemakerd: info: main: Starting
mainloop
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/hacluster
Sep 16 01:08:29 [1337] jlr-lb-01.aws pengine: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/hacluster
Sep 16 01:08:29 [1337] jlr-lb-01.aws pengine: info:
qb_ipcs_us_publish: server name: pengine
Sep 16 01:08:29 [1337] jlr-lb-01.aws pengine: info: main: Starting
pengine
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
get_cluster_type: Verifying cluster type: 'cman'
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
get_cluster_type: Assuming an active 'cman' cluster
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: retrieveCib:
Reading cluster configuration from: /var/lib/pacemaker/cib/cib.xml (digest:
/var/lib/pacemaker/cib/cib.xml.sig)
Sep 16 01:08:29 [1336] jlr-lb-01.aws attrd: notice:
crm_cluster_connect: Connecting to cluster infrastructure: cman
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/root
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info:
get_cluster_type: Verifying cluster type: 'cman'
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info:
get_cluster_type: Assuming an active 'cman' cluster
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: notice:
crm_cluster_connect: Connecting to cluster infrastructure: cman
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
validate_with_relaxng: Creating RNG parser context
Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/root
Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: notice: main: CRM Git
Version: 368c726
Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info: do_log: FSA:
Input I_STARTUP from crmd_init() received in state S_STARTING
Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info:
get_cluster_type: Verifying cluster type: 'cman'
Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info:
get_cluster_type: Assuming an active 'cman' cluster
Sep 16 01:08:29 [1338] jlr-lb-01.aws crmd: info: crm_ipc_connect:
Could not establish cib_shm connection: Connection refused (111)
Sep 16 01:08:29 [1335] jlr-lb-01.aws lrmd: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/root
Sep 16 01:08:29 [1335] jlr-lb-01.aws lrmd: info:
qb_ipcs_us_publish: server name: lrmd
Sep 16 01:08:29 [1335] jlr-lb-01.aws lrmd: info: main: Starting
Sep 16 01:08:29 [1336] jlr-lb-01.aws attrd: notice: main: Starting
mainloop...
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_get_peer:
Created entry f7bd7d66-0f53-4aea-9ee0-c152a3a79785/0x15e8890 for node
(null)/1 (1 total)
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_get_peer:
Cannot obtain a UUID for node 1/(null)
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info:
crm_update_peer_proc: cluster_connect_cpg: Node (null)[1] - corosync-cpg is
now online
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: cman_node_name:
Using CMAN node name jlr-lb-01.aws for 0
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info:
init_cs_connection_once: Connection to 'cman': established
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: cman_node_name:
Using CMAN node name jlr-lb-01.aws for 0
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_get_peer:
Node 1 is now known as jlr-lb-01.aws
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_get_peer:
Node 1 has uuid jlr-lb-01.aws
Sep 16 01:08:29 [1334] jlr-lb-01.aws stonith-ng: info: crm_ipc_connect:
Could not establish cib_rw connection: Connection refused (111)
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: startCib: CIB
Initialization completed successfully
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: notice:
crm_cluster_connect: Connecting to cluster infrastructure: cman
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_get_peer:
Created entry e5be9125-d252-4ba7-a4cd-3744cb955962/0x1bbe6b0 for node
(null)/1 (1 total)
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_get_peer:
Cannot obtain a UUID for node 1/(null)
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
crm_update_peer_proc: cluster_connect_cpg: Node (null)[1] - corosync-cpg is
now online
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: cman_node_name:
Using CMAN node name jlr-lb-01.aws for 0
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
init_cs_connection_once: Connection to 'cman': established
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: cman_node_name:
Using CMAN node name jlr-lb-01.aws for 0
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_get_peer:
Node 1 is now known as jlr-lb-01.aws
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: crm_get_peer:
Node 1 has uuid jlr-lb-01.aws
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
qb_ipcs_us_publish: server name: cib_ro
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
qb_ipcs_us_publish: server name: cib_rw
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
qb_ipcs_us_publish: server name: cib_shm
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info: cib_init:
Starting cib mainloop
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
pcmk_cpg_membership: Joined[0.0] cib.1
Sep 16 01:08:29 [1333] jlr-lb-01.aws cib: info:
pcmk_cpg_membership: Member[0.0] cib.1
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info:
write_cib_contents: Archived previous version as
/var/lib/pacemaker/cib/cib-24.raw
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info:
write_cib_contents: Wrote version 0.7.0 of the CIB to disk (digest:
f4e1275c2b642b1a630426162e42de0b)
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: retrieveCib:
Reading cluster configuration from: /var/lib/pacemaker/cib/cib.zGel81
(digest: /var/lib/pacemaker/cib/cib.5oDZ5a)
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc1630 for uid=0 gid=0 pid=1338
id=152f038f-a799-462b-b6cc-7ec09b4f4e26
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_cib_control:
CIB connection established
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: notice:
crm_cluster_connect: Connecting to cluster infrastructure: cman
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/2, version=0.7.0)
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc8910 for uid=0 gid=0 pid=1334
id=8ee650dd-33db-4f11-b1e2-87ccf58e633d
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer:
Created entry 8004e047-2a96-490e-83ba-c8510e3f1b94/0x1042580 for node
(null)/1 (1 total)
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer:
Cannot obtain a UUID for node 1/(null)
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
crm_update_peer_proc: cluster_connect_cpg: Node (null)[1] - corosync-cpg is
now online
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: cman_node_name:
Using CMAN node name jlr-lb-01.aws for 0
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
init_cs_connection_once: Connection to 'cman': established
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: cman_node_name:
Using CMAN node name jlr-lb-01.aws for 0
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer:
Node 1 is now known as jlr-lb-01.aws
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
peer_update_callback: jlr-lb-01.aws is now (null)
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer:
Node 1 has uuid jlr-lb-01.aws
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
init_cman_connection: Configuring Pacemaker to obtain quorum from cman
Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: notice: setup_cib:
Watching for stonith topology changes
Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info:
qb_ipcs_us_publish: server name: stonith-ng
Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info: main: Starting
stonith-ng mainloop
Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info:
pcmk_cpg_membership: Joined[0.0] stonith-ng.1
Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info:
pcmk_cpg_membership: Member[0.0] stonith-ng.1
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
cman_event_callback: Membership 52: quorum still lost
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/2, version=0.7.0)
Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: info:
init_cib_cache_cb: Updating device list from the cib: init
Sep 16 01:08:30 [1334] jlr-lb-01.aws stonith-ng: notice: unpack_config:
On loss of CCM Quorum: Ignore
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: notice:
crm_update_peer_state: cman_event_callback: Node jlr-lb-01.aws[1] - state
is now member (was (null))
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
peer_update_callback: jlr-lb-01.aws is now member (was (null))
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer:
Created entry cc5d11b6-0635-46ef-b670-d8541632213e/0x1044d30 for node
jlr-lb-02.aws/2 (2 total)
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer:
Node 2 is now known as jlr-lb-02.aws
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
peer_update_callback: jlr-lb-02.aws is now (null)
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: crm_get_peer:
Node 2 has uuid jlr-lb-02.aws
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
cman_event_callback: State of node jlr-lb-02.aws[2] is still unknown
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: cman_node_name:
Using CMAN node name jlr-lb-01.aws for 0
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_ha_control:
Connected to the cluster
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
lrmd_ipc_connect: Connecting to lrmd
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section nodes: OK
(rc=0, origin=local/crmd/3, version=0.7.0)
Sep 16 01:08:30 [1335] jlr-lb-01.aws lrmd: info: crm_client_new:
Connecting 0x1e08aa0 for uid=0 gid=0 pid=1338
id=e1350eaf-18c1-4f5e-a253-c52810ba6766
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_lrm_control:
LRM connection established
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_started:
Delaying start, Config not read (0000000000000040)
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_started:
Delaying start, Config not read (0000000000000040)
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section crm_config:
OK (rc=0, origin=local/crmd/4, version=0.7.0)
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
pcmk_cpg_membership: Joined[0.0] crmd.1
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
pcmk_cpg_membership: Member[0.0] crmd.1
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_started:
Delaying start, Config not read (0000000000000040)
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info:
qb_ipcs_us_publish: server name: crmd
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: notice: do_started: The
local CRM is operational
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: info: do_log: FSA:
Input I_PENDING from do_started() received in state S_STARTING
Sep 16 01:08:30 [1338] jlr-lb-01.aws crmd: notice:
do_state_transition: State transition S_STARTING -> S_PENDING [
input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Sep 16 01:08:30 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_slave operation for section 'all': OK
(rc=0, origin=local/crmd/5, version=0.7.0)
Sep 16 01:08:32 [1334] jlr-lb-01.aws stonith-ng: info: crm_client_new:
Connecting 0x16ef3b0 for uid=0 gid=0 pid=1338
id=689da5ef-ce47-4610-a744-8edcd995a048
Sep 16 01:08:32 [1334] jlr-lb-01.aws stonith-ng: info: stonith_command:
Processed register from crmd.1338: OK (0)
Sep 16 01:08:32 [1334] jlr-lb-01.aws stonith-ng: info: stonith_command:
Processed st_notify from crmd.1338: OK (0)
Sep 16 01:08:32 [1334] jlr-lb-01.aws stonith-ng: info: stonith_command:
Processed st_notify from crmd.1338: OK (0)
Sep 16 01:08:34 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1a0f200 for uid=189 gid=0 pid=1336
id=e12f20e0-9c0e-4c55-aa7f-87f2f4e6488d
Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info:
crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)
Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: warning: do_log: FSA:
Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info:
do_state_transition: State transition S_PENDING -> S_ELECTION [
input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ]
Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info: do_log: FSA:
Input I_ELECTION_DC from do_election_check() received in state S_ELECTION
Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: notice:
do_state_transition: State transition S_ELECTION -> S_INTEGRATION [
input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info: do_te_control:
Registering TE UUID: 036403dd-11d8-4f25-868d-1bbbf69d05fa
Sep 16 01:08:51 [1338] jlr-lb-01.aws crmd: info:
set_graph_functions: Setting custom graph functions
Sep 16 01:08:52 [1337] jlr-lb-01.aws pengine: info: crm_client_new:
Connecting 0x17dd260 for uid=0 gid=0 pid=1338
id=0d609011-5857-461b-9411-a0c20ebc166e
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: do_dc_takeover:
Taking over DC status for this partition
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_readwrite: We are now in R/W mode
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_master operation for section 'all': OK
(rc=0, origin=local/crmd/6, version=0.7.0)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section cib: OK
(rc=0, origin=local/crmd/7, version=0.7.0)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section
//cib/configuration/crm_config//cluster_property_set//nvpair[@name='dc-version']:
OK (rc=0, origin=local/crmd/8, version=0.7.0)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section crm_config:
OK (rc=0, origin=local/crmd/9, version=0.7.0)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section
//cib/configuration/crm_config//cluster_property_set//nvpair[@name='cluster-infrastructure']:
OK (rc=0, origin=local/crmd/10, version=0.7.0)
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: join_make_offer:
Making join offers based on membership 52
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: join_make_offer:
join-1: Sending offer to jlr-lb-01.aws
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
crm_update_peer_join: join_make_offer: Node jlr-lb-01.aws[1] - join-1 phase
0 -> 1
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section crm_config:
OK (rc=0, origin=local/crmd/11, version=0.7.0)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section crm_config:
OK (rc=0, origin=local/crmd/12, version=0.7.0)
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: update_dc: Set
DC to jlr-lb-01.aws (3.0.7)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/13, version=0.7.0)
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
crm_update_peer_join: do_dc_join_filter_offer: Node jlr-lb-01.aws[1] -
join-1 phase 1 -> 2
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
crm_update_peer_expected: do_dc_join_filter_offer: Node jlr-lb-01.aws[1] -
expected state is now member
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [
input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
crmd_join_phase_log: join-1: jlr-lb-01.aws=integrated
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
crmd_join_phase_log: join-1: jlr-lb-02.aws=none
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
do_dc_join_finalize: join-1: Syncing our CIB to the rest of the cluster
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info: cman_node_name:
Using CMAN node name jlr-lb-01.aws for 0
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_sync operation for section 'all': OK
(rc=0, origin=local/crmd/14, version=0.7.0)
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
crm_update_peer_join: finalize_join_for: Node jlr-lb-01.aws[1] - join-1
phase 2 -> 3
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
erase_status_tag: Deleting xpath:
//node_state[@uname='jlr-lb-01.aws']/transient_attributes
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: update_attrd:
Connecting to attrd... 5 retries remaining
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
crm_update_peer_join: do_dc_join_ack: Node jlr-lb-01.aws[1] - join-1 phase
3 -> 4
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info: do_dc_join_ack:
join-1: Updating node state to member for jlr-lb-01.aws
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
erase_status_tag: Deleting xpath: //node_state[@uname='jlr-lb-01.aws']/lrm
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section nodes: OK
(rc=0, origin=local/crmd/15, version=0.7.0)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_delete operation for section
//node_state[@uname='jlr-lb-01.aws']/transient_attributes: OK (rc=0,
origin=local/crmd/16, version=0.7.0)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_delete operation for section
//node_state[@uname='jlr-lb-01.aws']/lrm: OK (rc=0, origin=local/crmd/17,
version=0.7.0)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/crmd/18, version=0.7.1)
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Sep 16 01:08:52 [1338] jlr-lb-01.aws crmd: info:
abort_transition_graph: do_te_invoke:151 - Triggered transition abort
(complete=1) : Peer Cancelled
Sep 16 01:08:52 [1336] jlr-lb-01.aws attrd: notice:
attrd_local_callback: Sending full refresh (origin=crmd)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section nodes: OK
(rc=0, origin=local/crmd/19, version=0.7.1)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/crmd/20, version=0.7.2)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section cib: OK
(rc=0, origin=local/crmd/21, version=0.7.3)
Sep 16 01:08:52 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/22, version=0.7.3)
Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/23, version=0.7.3)
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice: unpack_config:
On loss of CCM Quorum: Ignore
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info:
determine_online_status: Node jlr-lb-01.aws is online
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: native_print:
livefrontendIP0 (ocf::heartbeat:AWSVip): Stopped
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: RecurringOp:
Start recurring monitor (60s) for livefrontendIP0 on jlr-lb-01.aws
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice: LogActions:
Start livefrontendIP0 (jlr-lb-01.aws)
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info:
do_state_transition: State transition S_POLICY_ENGINE ->
S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
origin=handle_response ]
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_te_invoke:
Processing graph 0 (ref=pe_calc-dc-1410829733-7) derived from
/var/lib/pacemaker/pengine/pe-input-72.bz2
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: te_rsc_command:
Initiating action 4: monitor livefrontendIP0_monitor_0 on jlr-lb-01.aws
(local)
Sep 16 01:08:53 [1335] jlr-lb-01.aws lrmd: info:
process_lrmd_get_rsc_info: Resource 'livefrontendIP0' not found (0 active
resources)
Sep 16 01:08:53 [1335] jlr-lb-01.aws lrmd: info:
process_lrmd_rsc_register: Added 'livefrontendIP0' to the rsc list (1
active resources)
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_lrm_rsc_op:
Performing key=4:0:7:036403dd-11d8-4f25-868d-1bbbf69d05fa
op=livefrontendIP0_monitor_0
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice:
process_pe_message: Calculated Transition 0:
/var/lib/pacemaker/pengine/pe-input-72.bz2
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info:
services_os_action_execute: Managed AWSVip_meta-data_0 process 1457 exited
with rc=0
Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/crmd/24, version=0.7.4)
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice:
process_lrm_event: LRM operation livefrontendIP0_monitor_0 (call=5, rc=0,
cib-update=24, confirmed=true) ok
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: warning: status_from_rc:
Action 4 (livefrontendIP0_monitor_0) on jlr-lb-01.aws failed (target: 7 vs.
rc: 0): Error
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info:
abort_transition_graph: match_graph_event:313 - Triggered transition abort
(complete=0, node=jlr-lb-01.aws, tag=lrm_rsc_op,
id=livefrontendIP0_last_failure_0,
magic=0:0;4:0:7:036403dd-11d8-4f25-868d-1bbbf69d05fa, cib=0.7.4) : Event
failed
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info:
match_graph_event: Action livefrontendIP0_monitor_0 (4) confirmed on
jlr-lb-01.aws (rc=4)
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info:
process_graph_event: Detected action (0.4) livefrontendIP0_monitor_0.5=ok:
failed
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: te_rsc_command:
Initiating action 3: probe_complete probe_complete on jlr-lb-01.aws (local)
- no waiting
Sep 16 01:08:53 [1336] jlr-lb-01.aws attrd: notice:
attrd_trigger_update: Sending flush op to all hosts for: probe_complete
(true)
Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section
//cib/status//node_state[@id='jlr-lb-01.aws']//transient_attributes//nvpair[@name='probe_complete']:
No such device or address (rc=-6, origin=local/attrd/2, version=0.7.4)
Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section /cib: OK
(rc=0, origin=local/attrd/3, version=0.7.4)
Sep 16 01:08:53 [1336] jlr-lb-01.aws attrd: notice:
attrd_perform_update: Sent update 4: probe_complete=true
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: te_rsc_command:
Action 3 confirmed - no wait
Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/attrd/4, version=0.7.5)
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: run_graph:
Transition 0 (Complete=2, Pending=0, Fired=0, Skipped=3, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-72.bz2): Stopped
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info:
do_state_transition: State transition S_TRANSITION_ENGINE ->
S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/25, version=0.7.5)
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice: unpack_config:
On loss of CCM Quorum: Ignore
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info:
determine_online_status: Node jlr-lb-01.aws is online
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: unpack_rsc_op:
Operation monitor found resource livefrontendIP0 active on jlr-lb-01.aws
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: native_print:
livefrontendIP0 (ocf::heartbeat:AWSVip): Started jlr-lb-01.aws
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: RecurringOp:
Start recurring monitor (60s) for livefrontendIP0 on jlr-lb-01.aws
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: info: LogActions:
Leave livefrontendIP0 (Started jlr-lb-01.aws)
Sep 16 01:08:53 [1337] jlr-lb-01.aws pengine: notice:
process_pe_message: Calculated Transition 1:
/var/lib/pacemaker/pengine/pe-input-73.bz2
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info:
do_state_transition: State transition S_POLICY_ENGINE ->
S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
origin=handle_response ]
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_te_invoke:
Processing graph 1 (ref=pe_calc-dc-1410829733-10) derived from
/var/lib/pacemaker/pengine/pe-input-73.bz2
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: te_rsc_command:
Initiating action 6: monitor livefrontendIP0_monitor_60000 on jlr-lb-01.aws
(local)
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_lrm_rsc_op:
Performing key=6:1:0:036403dd-11d8-4f25-868d-1bbbf69d05fa
op=livefrontendIP0_monitor_60000
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice:
process_lrm_event: LRM operation livefrontendIP0_monitor_60000 (call=8,
rc=0, cib-update=26, confirmed=false) ok
Sep 16 01:08:53 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/crmd/26, version=0.7.6)
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info:
match_graph_event: Action livefrontendIP0_monitor_60000 (6) confirmed on
jlr-lb-01.aws (rc=0)
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice: run_graph:
Transition 1 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-73.bz2): Complete
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: info: do_log: FSA:
Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Sep 16 01:08:53 [1338] jlr-lb-01.aws crmd: notice:
do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [
input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1525
id=86b4712a-3b41-4580-900f-ad1578b844f5
Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crm_mon/2, version=0.7.6)
Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1526
id=0874dae6-6185-435c-9747-5910c6465083
Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:13 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1532
id=b073a7c8-59df-4a08-81a4-ef62e37b24a3
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1533
id=d7397d96-50d6-4813-8ffd-4a537ed66d73
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1534
id=3aa7a511-0a1c-4b53-b3c8-bfb1a37fef37
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1535
id=4149a25e-19af-4e71-bff3-344633251426
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1536
id=cbc05321-d219-4960-8972-dcca7a93d851
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section
//constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1537
id=fba48fcb-6e64-47db-a5d2-2ac01e617bbf
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section
//constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1538
id=893198c3-02e6-4241-9533-1b7f0b029ca7
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section
//constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1539
id=8e0d79fa-bbae-4c55-a491-a70c19ade564
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section crm_config:
OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1540
id=9c416d6a-ef1a-4c7f-8204-27176f30d6f0
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section //nodes: OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:15 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:12:56 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1562
id=a1a77b50-c091-42fb-b5b7-ad2266a03147
Sep 16 01:12:56 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:12:56 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:13:11 [1333] jlr-lb-01.aws cib: info: crm_client_new:
Connecting 0x1bc6080 for uid=0 gid=0 pid=1574
id=9cd5d793-35e3-4c07-9442-70f8c697f278
Sep 16 01:13:11 [1333] jlr-lb-01.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crm_verify/2, version=0.7.6)
Sep 16 01:13:11 [1333] jlr-lb-01.aws cib: info:
crm_client_destroy: Destroying 0 events
-------------------------------------------------------------------------------------------
NODE 2 LOGS
-------------------------------------------------------------------------------------------
Sep 16 01:07:56 corosync [MAIN ] Corosync Cluster Engine ('1.4.1'):
started and ready to provide service.
Sep 16 01:07:56 corosync [MAIN ] Corosync built-in features: nss dbus rdma
snmp
Sep 16 01:07:56 corosync [MAIN ] Successfully read config from
/etc/cluster/cluster.conf
Sep 16 01:07:56 corosync [MAIN ] Successfully parsed cman config
Sep 16 01:07:56 corosync [TOTEM ] Initializing transport (UDP/IP Multicast).
Sep 16 01:07:56 corosync [TOTEM ] Initializing transmit/receive security:
libtomcrypt SOBER128/SHA1HMAC (mode 0).
Sep 16 01:07:56 corosync [TOTEM ] The network interface [10.235.48.124] is
now up.
Sep 16 01:07:57 corosync [QUORUM] Using quorum provider quorum_cman
Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync cluster
quorum service v0.1
Sep 16 01:07:57 corosync [CMAN ] CMAN 3.0.12.1 (built Apr 14 2014
09:36:10) started
Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync CMAN
membership service 2.90
Sep 16 01:07:57 corosync [SERV ] Service engine loaded: openais checkpoint
service B.01.01
Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync extended
virtual synchrony service
Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync
configuration service
Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync cluster
closed process group service v1.01
Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync cluster
config database access v1.01
Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync profile
loading service
Sep 16 01:07:57 corosync [QUORUM] Using quorum provider quorum_cman
Sep 16 01:07:57 corosync [SERV ] Service engine loaded: corosync cluster
quorum service v0.1
Sep 16 01:07:57 corosync [MAIN ] Compatibility mode set to whitetank.
Using V1 and V2 of the synchronization engine.
Sep 16 01:07:57 corosync [TOTEM ] A processor joined or left the membership
and a new membership was formed.
Sep 16 01:07:57 corosync [QUORUM] Members[1]: 2
Sep 16 01:07:57 corosync [QUORUM] Members[1]: 2
Sep 16 01:07:57 corosync [CPG ] chosen downlist: sender r(0)
ip(10.235.48.124) ; members(old:0 left:0)
Sep 16 01:07:57 corosync [MAIN ] Completed service synchronization, ready
to provide service.
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: notice: main: Starting
Pacemaker 1.1.10-14.el6_5.3 (Build: 368c726): generated-manpages
agent-manpages ascii-docs publican-docs ncurses libqb-logging libqb-ipc
nagios corosync-plugin cman
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: main: Maximum
core file size is: 18446744073709551615
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info:
qb_ipcs_us_publish: server name: pacemakerd
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: crm_get_peer:
Created entry 1384f338-5022-44c5-9da7-71b08ea75d42/0x13bbfc0 for node
(null)/2 (1 total)
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: crm_get_peer:
Cannot obtain a UUID for node 2/(null)
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info:
crm_update_peer_proc: cluster_connect_cpg: Node (null)[2] - corosync-cpg is
now online
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: cman_node_name:
Using CMAN node name jlr-lb-02.aws for 0
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: crm_get_peer:
Node 2 is now known as jlr-lb-02.aws
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: crm_get_peer:
Node 2 has uuid jlr-lb-02.aws
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child:
Using uid=189 and group=189 for process cib
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child:
Forked child 1352 for process cib
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child:
Forked child 1353 for process stonith-ng
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child:
Forked child 1354 for process lrmd
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child:
Using uid=189 and group=189 for process attrd
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child:
Forked child 1355 for process attrd
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child:
Using uid=189 and group=189 for process pengine
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child:
Forked child 1356 for process pengine
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: start_child:
Forked child 1357 for process crmd
Sep 16 01:08:06 [1346] jlr-lb-02.aws pacemakerd: info: main: Starting
mainloop
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/hacluster
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
get_cluster_type: Verifying cluster type: 'cman'
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
get_cluster_type: Assuming an active 'cman' cluster
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: retrieveCib:
Reading cluster configuration from: /var/lib/pacemaker/cib/cib.xml (digest:
/var/lib/pacemaker/cib/cib.xml.sig)
Sep 16 01:08:06 [1355] jlr-lb-02.aws attrd: notice:
crm_cluster_connect: Connecting to cluster infrastructure: cman
Sep 16 01:08:06 [1356] jlr-lb-02.aws pengine: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/hacluster
Sep 16 01:08:06 [1356] jlr-lb-02.aws pengine: info:
qb_ipcs_us_publish: server name: pengine
Sep 16 01:08:06 [1356] jlr-lb-02.aws pengine: info: main: Starting
pengine
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
validate_with_relaxng: Creating RNG parser context
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/root
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info:
get_cluster_type: Verifying cluster type: 'cman'
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info:
get_cluster_type: Assuming an active 'cman' cluster
Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/root
Sep 16 01:08:06 [1354] jlr-lb-02.aws lrmd: info: crm_log_init:
Changed active directory to /var/lib/pacemaker/cores/root
Sep 16 01:08:06 [1354] jlr-lb-02.aws lrmd: info:
qb_ipcs_us_publish: server name: lrmd
Sep 16 01:08:06 [1354] jlr-lb-02.aws lrmd: info: main: Starting
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: notice:
crm_cluster_connect: Connecting to cluster infrastructure: cman
Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: notice: main: CRM Git
Version: 368c726
Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info: do_log: FSA:
Input I_STARTUP from crmd_init() received in state S_STARTING
Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info:
get_cluster_type: Verifying cluster type: 'cman'
Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info:
get_cluster_type: Assuming an active 'cman' cluster
Sep 16 01:08:06 [1357] jlr-lb-02.aws crmd: info: crm_ipc_connect:
Could not establish cib_shm connection: Connection refused (111)
Sep 16 01:08:06 [1355] jlr-lb-02.aws attrd: notice: main: Starting
mainloop...
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_get_peer:
Created entry f202c0cf-5a4a-4336-98e4-8ee64511697e/0x9fe890 for node
(null)/2 (1 total)
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_get_peer:
Cannot obtain a UUID for node 2/(null)
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info:
crm_update_peer_proc: cluster_connect_cpg: Node (null)[2] - corosync-cpg is
now online
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: cman_node_name:
Using CMAN node name jlr-lb-02.aws for 0
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info:
init_cs_connection_once: Connection to 'cman': established
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: cman_node_name:
Using CMAN node name jlr-lb-02.aws for 0
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_get_peer:
Node 2 is now known as jlr-lb-02.aws
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_get_peer:
Node 2 has uuid jlr-lb-02.aws
Sep 16 01:08:06 [1353] jlr-lb-02.aws stonith-ng: info: crm_ipc_connect:
Could not establish cib_rw connection: Connection refused (111)
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: startCib: CIB
Initialization completed successfully
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: notice:
crm_cluster_connect: Connecting to cluster infrastructure: cman
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_get_peer:
Created entry 63b7c4e6-ee9c-4692-8c00-d72cc1a72625/0x24fe860 for node
(null)/2 (1 total)
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_get_peer:
Cannot obtain a UUID for node 2/(null)
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
crm_update_peer_proc: cluster_connect_cpg: Node (null)[2] - corosync-cpg is
now online
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: cman_node_name:
Using CMAN node name jlr-lb-02.aws for 0
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
init_cs_connection_once: Connection to 'cman': established
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: cman_node_name:
Using CMAN node name jlr-lb-02.aws for 0
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_get_peer:
Node 2 is now known as jlr-lb-02.aws
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: crm_get_peer:
Node 2 has uuid jlr-lb-02.aws
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
qb_ipcs_us_publish: server name: cib_ro
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
qb_ipcs_us_publish: server name: cib_rw
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
qb_ipcs_us_publish: server name: cib_shm
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: cib_init:
Starting cib mainloop
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
pcmk_cpg_membership: Joined[0.0] cib.2
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
pcmk_cpg_membership: Member[0.0] cib.2
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
write_cib_contents: Archived previous version as
/var/lib/pacemaker/cib/cib-20.raw
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info:
write_cib_contents: Wrote version 0.7.0 of the CIB to disk (digest:
9a6dcf33ae0aeb2e09a2e348ec5ec53a)
Sep 16 01:08:06 [1352] jlr-lb-02.aws cib: info: retrieveCib:
Reading cluster configuration from: /var/lib/pacemaker/cib/cib.EsVx7U
(digest: /var/lib/pacemaker/cib/cib.K2HEcT)
Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2501730 for uid=0 gid=0 pid=1357
id=1b04dbca-8326-4e35-ab40-2a5a8f55a597
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_cib_control:
CIB connection established
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: notice:
crm_cluster_connect: Connecting to cluster infrastructure: cman
Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/2, version=0.7.0)
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer:
Created entry 659a46dc-8ff4-44cd-94b3-0d4d3caaf300/0x1070580 for node
(null)/2 (1 total)
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer:
Cannot obtain a UUID for node 2/(null)
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
crm_update_peer_proc: cluster_connect_cpg: Node (null)[2] - corosync-cpg is
now online
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: cman_node_name:
Using CMAN node name jlr-lb-02.aws for 0
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
init_cs_connection_once: Connection to 'cman': established
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: cman_node_name:
Using CMAN node name jlr-lb-02.aws for 0
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer:
Node 2 is now known as jlr-lb-02.aws
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
peer_update_callback: jlr-lb-02.aws is now (null)
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer:
Node 2 has uuid jlr-lb-02.aws
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
init_cman_connection: Configuring Pacemaker to obtain quorum from cman
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
cman_event_callback: Membership 40: quorum still lost
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer:
Created entry 68a0b84f-e28b-47d8-8c54-4cbcfc3c25bf/0x1073020 for node
jlr-lb-01.aws/1 (2 total)
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer:
Node 1 is now known as jlr-lb-01.aws
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
peer_update_callback: jlr-lb-01.aws is now (null)
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: crm_get_peer:
Node 1 has uuid jlr-lb-01.aws
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
cman_event_callback: State of node jlr-lb-01.aws[1] is still unknown
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: notice:
crm_update_peer_state: cman_event_callback: Node jlr-lb-02.aws[2] - state
is now member (was (null))
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
peer_update_callback: jlr-lb-02.aws is now member (was (null))
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: cman_node_name:
Using CMAN node name jlr-lb-02.aws for 0
Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section nodes: OK
(rc=0, origin=local/crmd/3, version=0.7.0)
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_ha_control:
Connected to the cluster
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
lrmd_ipc_connect: Connecting to lrmd
Sep 16 01:08:07 [1354] jlr-lb-02.aws lrmd: info: crm_client_new:
Connecting 0x1289aa0 for uid=0 gid=0 pid=1357
id=88fb7ff9-1bf4-450c-a658-184bb2689b5d
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_lrm_control:
LRM connection established
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_started:
Delaying start, Config not read (0000000000000040)
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_started:
Delaying start, Config not read (0000000000000040)
Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section crm_config:
OK (rc=0, origin=local/crmd/4, version=0.7.0)
Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x234f200 for uid=0 gid=0 pid=1353
id=5885c19b-6f02-4f68-953b-39583be06e90
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
pcmk_cpg_membership: Joined[0.0] crmd.2
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
pcmk_cpg_membership: Member[0.0] crmd.2
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_started:
Delaying start, Config not read (0000000000000040)
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info:
qb_ipcs_us_publish: server name: crmd
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: notice: do_started: The
local CRM is operational
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: info: do_log: FSA:
Input I_PENDING from do_started() received in state S_STARTING
Sep 16 01:08:07 [1357] jlr-lb-02.aws crmd: notice:
do_state_transition: State transition S_STARTING -> S_PENDING [
input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_slave operation for section 'all': OK
(rc=0, origin=local/crmd/5, version=0.7.0)
Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: notice: setup_cib:
Watching for stonith topology changes
Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info:
qb_ipcs_us_publish: server name: stonith-ng
Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info: main: Starting
stonith-ng mainloop
Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info:
pcmk_cpg_membership: Joined[0.0] stonith-ng.2
Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info:
pcmk_cpg_membership: Member[0.0] stonith-ng.2
Sep 16 01:08:07 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/2, version=0.7.0)
Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: info:
init_cib_cache_cb: Updating device list from the cib: init
Sep 16 01:08:07 [1353] jlr-lb-02.aws stonith-ng: notice: unpack_config:
On loss of CCM Quorum: Ignore
Sep 16 01:08:09 [1353] jlr-lb-02.aws stonith-ng: info: crm_client_new:
Connecting 0xb053b0 for uid=0 gid=0 pid=1357
id=13c4ef82-44d1-42b7-99ce-8985d748b2e8
Sep 16 01:08:09 [1353] jlr-lb-02.aws stonith-ng: info: stonith_command:
Processed register from crmd.1357: OK (0)
Sep 16 01:08:09 [1353] jlr-lb-02.aws stonith-ng: info: stonith_command:
Processed st_notify from crmd.1357: OK (0)
Sep 16 01:08:09 [1353] jlr-lb-02.aws stonith-ng: info: stonith_command:
Processed st_notify from crmd.1357: OK (0)
Sep 16 01:08:11 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x25020e0 for uid=189 gid=0 pid=1355
id=87268c89-a3d2-4c99-baa6-927b4e614c6a
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: warning: do_log: FSA:
Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
do_state_transition: State transition S_PENDING -> S_ELECTION [
input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ]
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_log: FSA:
Input I_ELECTION_DC from do_election_check() received in state S_ELECTION
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: notice:
do_state_transition: State transition S_ELECTION -> S_INTEGRATION [
input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_te_control:
Registering TE UUID: ae731050-4f86-455d-b180-c8948bca0331
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
set_graph_functions: Setting custom graph functions
Sep 16 01:08:28 [1356] jlr-lb-02.aws pengine: info: crm_client_new:
Connecting 0x1cf1260 for uid=0 gid=0 pid=1357
id=a4da1ab1-aecb-45fb-87ad-88c3f0fbc212
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_dc_takeover:
Taking over DC status for this partition
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_readwrite: We are now in R/W mode
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_master operation for section 'all': OK
(rc=0, origin=local/crmd/6, version=0.7.0)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section cib: OK
(rc=0, origin=local/crmd/7, version=0.7.0)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section
//cib/configuration/crm_config//cluster_property_set//nvpair[@name='dc-version']:
OK (rc=0, origin=local/crmd/8, version=0.7.0)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section crm_config:
OK (rc=0, origin=local/crmd/9, version=0.7.0)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section
//cib/configuration/crm_config//cluster_property_set//nvpair[@name='cluster-infrastructure']:
OK (rc=0, origin=local/crmd/10, version=0.7.0)
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: join_make_offer:
Making join offers based on membership 40
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: join_make_offer:
join-1: Sending offer to jlr-lb-02.aws
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
crm_update_peer_join: join_make_offer: Node jlr-lb-02.aws[2] - join-1 phase
0 -> 1
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: update_dc: Set
DC to jlr-lb-02.aws (3.0.7)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section crm_config:
OK (rc=0, origin=local/crmd/11, version=0.7.0)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section crm_config:
OK (rc=0, origin=local/crmd/12, version=0.7.0)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/13, version=0.7.0)
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
crm_update_peer_join: do_dc_join_filter_offer: Node jlr-lb-02.aws[2] -
join-1 phase 1 -> 2
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
crm_update_peer_expected: do_dc_join_filter_offer: Node jlr-lb-02.aws[2] -
expected state is now member
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [
input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
crmd_join_phase_log: join-1: jlr-lb-02.aws=integrated
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
crmd_join_phase_log: join-1: jlr-lb-01.aws=none
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
do_dc_join_finalize: join-1: Syncing our CIB to the rest of the cluster
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info: cman_node_name:
Using CMAN node name jlr-lb-02.aws for 0
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_sync operation for section 'all': OK
(rc=0, origin=local/crmd/14, version=0.7.0)
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
crm_update_peer_join: finalize_join_for: Node jlr-lb-02.aws[2] - join-1
phase 2 -> 3
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
erase_status_tag: Deleting xpath:
//node_state[@uname='jlr-lb-02.aws']/transient_attributes
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: update_attrd:
Connecting to attrd... 5 retries remaining
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section nodes: OK
(rc=0, origin=local/crmd/15, version=0.7.0)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_delete operation for section
//node_state[@uname='jlr-lb-02.aws']/transient_attributes: OK (rc=0,
origin=local/crmd/16, version=0.7.0)
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
crm_update_peer_join: do_dc_join_ack: Node jlr-lb-02.aws[2] - join-1 phase
3 -> 4
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info: do_dc_join_ack:
join-1: Updating node state to member for jlr-lb-02.aws
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
erase_status_tag: Deleting xpath: //node_state[@uname='jlr-lb-02.aws']/lrm
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_delete operation for section
//node_state[@uname='jlr-lb-02.aws']/lrm: OK (rc=0, origin=local/crmd/17,
version=0.7.0)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/crmd/18, version=0.7.1)
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Sep 16 01:08:28 [1357] jlr-lb-02.aws crmd: info:
abort_transition_graph: do_te_invoke:151 - Triggered transition abort
(complete=1) : Peer Cancelled
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section nodes: OK
(rc=0, origin=local/crmd/19, version=0.7.1)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/crmd/20, version=0.7.2)
Sep 16 01:08:28 [1355] jlr-lb-02.aws attrd: notice:
attrd_local_callback: Sending full refresh (origin=crmd)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section cib: OK
(rc=0, origin=local/crmd/21, version=0.7.3)
Sep 16 01:08:28 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/22, version=0.7.3)
Sep 16 01:08:29 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/23, version=0.7.3)
Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: notice: unpack_config:
On loss of CCM Quorum: Ignore
Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: info:
determine_online_status: Node jlr-lb-02.aws is online
Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: info: native_print:
livefrontendIP0 (ocf::heartbeat:AWSVip): Stopped
Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: info: RecurringOp:
Start recurring monitor (60s) for livefrontendIP0 on jlr-lb-02.aws
Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: notice: LogActions:
Start livefrontendIP0 (jlr-lb-02.aws)
Sep 16 01:08:29 [1357] jlr-lb-02.aws crmd: info:
do_state_transition: State transition S_POLICY_ENGINE ->
S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
origin=handle_response ]
Sep 16 01:08:29 [1357] jlr-lb-02.aws crmd: info: do_te_invoke:
Processing graph 0 (ref=pe_calc-dc-1410829709-7) derived from
/var/lib/pacemaker/pengine/pe-input-30.bz2
Sep 16 01:08:29 [1357] jlr-lb-02.aws crmd: notice: te_rsc_command:
Initiating action 4: monitor livefrontendIP0_monitor_0 on jlr-lb-02.aws
(local)
Sep 16 01:08:29 [1354] jlr-lb-02.aws lrmd: info:
process_lrmd_get_rsc_info: Resource 'livefrontendIP0' not found (0 active
resources)
Sep 16 01:08:29 [1354] jlr-lb-02.aws lrmd: info:
process_lrmd_rsc_register: Added 'livefrontendIP0' to the rsc list (1
active resources)
Sep 16 01:08:29 [1357] jlr-lb-02.aws crmd: info: do_lrm_rsc_op:
Performing key=4:0:7:ae731050-4f86-455d-b180-c8948bca0331
op=livefrontendIP0_monitor_0
Sep 16 01:08:29 [1356] jlr-lb-02.aws pengine: notice:
process_pe_message: Calculated Transition 0:
/var/lib/pacemaker/pengine/pe-input-30.bz2
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info:
services_os_action_execute: Managed AWSVip_meta-data_0 process 1477 exited
with rc=0
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice:
process_lrm_event: LRM operation livefrontendIP0_monitor_0 (call=5, rc=0,
cib-update=24, confirmed=true) ok
Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/crmd/24, version=0.7.4)
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: warning: status_from_rc:
Action 4 (livefrontendIP0_monitor_0) on jlr-lb-02.aws failed (target: 7 vs.
rc: 0): Error
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info:
abort_transition_graph: match_graph_event:313 - Triggered transition abort
(complete=0, node=jlr-lb-02.aws, tag=lrm_rsc_op,
id=livefrontendIP0_last_failure_0,
magic=0:0;4:0:7:ae731050-4f86-455d-b180-c8948bca0331, cib=0.7.4) : Event
failed
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info:
match_graph_event: Action livefrontendIP0_monitor_0 (4) confirmed on
jlr-lb-02.aws (rc=4)
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info:
process_graph_event: Detected action (0.4) livefrontendIP0_monitor_0.5=ok:
failed
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: te_rsc_command:
Initiating action 3: probe_complete probe_complete on jlr-lb-02.aws (local)
- no waiting
Sep 16 01:08:30 [1355] jlr-lb-02.aws attrd: notice:
attrd_trigger_update: Sending flush op to all hosts for: probe_complete
(true)
Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section
//cib/status//node_state[@id='jlr-lb-02.aws']//transient_attributes//nvpair[@name='probe_complete']:
No such device or address (rc=-6, origin=local/attrd/2, version=0.7.4)
Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section /cib: OK
(rc=0, origin=local/attrd/3, version=0.7.4)
Sep 16 01:08:30 [1355] jlr-lb-02.aws attrd: notice:
attrd_perform_update: Sent update 4: probe_complete=true
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: te_rsc_command:
Action 3 confirmed - no wait
Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/attrd/4, version=0.7.5)
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: run_graph:
Transition 0 (Complete=2, Pending=0, Fired=0, Skipped=3, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-30.bz2): Stopped
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info:
do_state_transition: State transition S_TRANSITION_ENGINE ->
S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crmd/25, version=0.7.5)
Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: notice: unpack_config:
On loss of CCM Quorum: Ignore
Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info:
determine_online_status: Node jlr-lb-02.aws is online
Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info: unpack_rsc_op:
Operation monitor found resource livefrontendIP0 active on jlr-lb-02.aws
Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info: native_print:
livefrontendIP0 (ocf::heartbeat:AWSVip): Started jlr-lb-02.aws
Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info: RecurringOp:
Start recurring monitor (60s) for livefrontendIP0 on jlr-lb-02.aws
Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: info: LogActions:
Leave livefrontendIP0 (Started jlr-lb-02.aws)
Sep 16 01:08:30 [1356] jlr-lb-02.aws pengine: notice:
process_pe_message: Calculated Transition 1:
/var/lib/pacemaker/pengine/pe-input-31.bz2
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info:
do_state_transition: State transition S_POLICY_ENGINE ->
S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
origin=handle_response ]
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: do_te_invoke:
Processing graph 1 (ref=pe_calc-dc-1410829710-10) derived from
/var/lib/pacemaker/pengine/pe-input-31.bz2
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: te_rsc_command:
Initiating action 6: monitor livefrontendIP0_monitor_60000 on jlr-lb-02.aws
(local)
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: do_lrm_rsc_op:
Performing key=6:1:0:ae731050-4f86-455d-b180-c8948bca0331
op=livefrontendIP0_monitor_60000
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice:
process_lrm_event: LRM operation livefrontendIP0_monitor_60000 (call=8,
rc=0, cib-update=26, confirmed=false) ok
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info:
match_graph_event: Action livefrontendIP0_monitor_60000 (6) confirmed on
jlr-lb-02.aws (rc=0)
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice: run_graph:
Transition 1 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-31.bz2): Complete
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: info: do_log: FSA:
Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Sep 16 01:08:30 [1357] jlr-lb-02.aws crmd: notice:
do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [
input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Sep 16 01:08:30 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_modify operation for section status: OK
(rc=0, origin=local/crmd/26, version=0.7.6)
Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1490
id=1f235bdf-d2ee-47d9-af5c-c6ae4130f59a
Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crm_mon/2, version=0.7.6)
Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1491
id=0df0f7d7-f92a-4d91-b242-6e645940b3b8
Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:09:27 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:10:21 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1530
id=3a77dc76-433c-472c-a553-c633d4472e82
Sep 16 01:10:21 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:10:21 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:10:57 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1546
id=7479bbf2-a878-4863-ab28-6e6924d156f0
Sep 16 01:10:57 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/crm_verify/2, version=0.7.6)
Sep 16 01:10:57 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1615
id=028a6604-3038-46e6-9162-83c21670211f
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1616
id=add9ee90-7077-4970-9334-bc96cf2d4dac
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1617
id=43c28e33-aa58-4dc8-8608-23b43eb9a576
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1618
id=9ccbf6c9-9b02-480b-b778-bc2b38d28229
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section 'all': OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1619
id=3da1e9e9-ff2e-4c40-9031-589b3dfc134b
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section
//constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1620
id=d865033d-6df5-4b33-9bf9-43f7a62ea434
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section
//constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1621
id=1a10c96d-95cc-47c4-9e30-b1c2e4bac374
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section
//constraints: OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1622
id=6e22e016-d80e-4937-851c-a0b9692322a5
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section crm_config:
OK (rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info: crm_client_new:
Connecting 0x2508a70 for uid=0 gid=0 pid=1623
id=f4b3fdbf-9ad1-4351-8b30-95ead3f368fa
Sep 16 01:17:32 [1352] jlr-lb-02.aws cib: info:
cib_process_request: Completed cib_query operation for section //nodes: OK
(rc=0, origin=local/cibadmin/2, version=0.7.6)
Sep 16 01:17:33 [1352] jlr-lb-02.aws cib: info:
crm_client_destroy: Destroying 0 events
--
- Jeff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/pacemaker/attachments/20140915/906293a0/attachment-0002.html>
More information about the Pacemaker
mailing list