[Pacemaker] pacemaker 1.1.10 + RHEL 7 beta issues
Digimer
lists at alteeve.ca
Wed Jan 1 18:30:36 EST 2014
I have a problem where putting a node into standby causes my "WebSite"
resource to stop and refuse to start. This is the reproducer I show
here, but I have seen this happen with DRBD earlier (before I realized
what was wrong, so I didn't document it), so I don't think this is an
apache RA issue.
[root at an-c03n01 ~]# rpm -q pacemaker httpd pcs corosync
pacemaker-1.1.10-19.el7.x86_64
httpd-2.4.6-7.el7.x86_64
pcs-0.9.99-2.el7.x86_64
corosync-2.3.2-2.el7.x86_64
[root at an-c03n01 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Everything release 7.0 Beta (Maipo)
[root at an-c03n01 ~]# uname -a
Linux an-c03n01.alteeve.ca 3.10.0-54.0.1.el7.x86_64 #1 SMP Tue Nov 26
16:51:22 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
DRBD installed from source:
[root at an-c03n01 ~]# cat /proc/drbd
version: 8.4.4 (api:1/proto:86-101)
GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by
root at an-c03n01.alteeve.ca, 2013-12-23 20:36:27
Starting with everything working:
=====
[root at an-c03n01 ~]# pcs status
Cluster name: an-cluster-03
Last updated: Wed Jan 1 18:13:19 2014
Last change: Wed Jan 1 18:03:08 2014 via cibadmin on an-c03n01.alteeve.ca
Stack: corosync
Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
7 Resources configured
Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
Full list of resources:
fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n01.alteeve.ca
Master/Slave Set: WebDataClone [WebData]
Masters: [ an-c03n01.alteeve.ca ]
Slaves: [ an-c03n02.alteeve.ca ]
WebFS (ocf::heartbeat:Filesystem): Started an-c03n01.alteeve.ca
WebSite (ocf::heartbeat:apache): Started an-c03n01.alteeve.ca
PCSD Status:
an-c03n01.alteeve.ca:
an-c03n01.alteeve.ca: Online
an-c03n02.alteeve.ca:
an-c03n02.alteeve.ca: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
=====
=====
[root at an-c03n01 ~]# pcs config show
Cluster Name: an-cluster-03
Corosync Nodes:
an-c03n01.alteeve.ca an-c03n02.alteeve.ca
Pacemaker Nodes:
an-c03n01.alteeve.ca an-c03n02.alteeve.ca
Resources:
Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.122.10 cidr_netmask=32
Operations: monitor interval=30s (ClusterIP-monitor-interval-30s)
Master: WebDataClone
Meta Attrs: master-max=1 master-node-max=1 clone-max=2
clone-node-max=1 notify=true
Resource: WebData (class=ocf provider=linbit type=drbd)
Attributes: drbd_resource=r0
Operations: monitor interval=60s (WebData-monitor-60s)
Resource: WebFS (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/drbd0 directory=/var/www/html fstype=ext4
Operations: monitor interval=60s (WebFS-monitor-interval-60s)
Resource: WebSite (class=ocf provider=heartbeat type=apache)
Attributes: configfile=/etc/httpd/conf/httpd.conf
statusurl=http://127.0.0.1/server-status
Operations: monitor interval=60s (WebSite-monitor-interval-60s)
Stonith Devices:
Resource: fence_n01_virsh (class=stonith type=fence_virsh)
Attributes: pcmk_host_list=an-c03n01.alteeve.ca ipaddr=lemass
login=root passwd_script=/root/lemass.pw delay=15 port=an-c03n01
Operations: monitor interval=60s (fence_n01_virsh-monitor-interval-60s)
Resource: fence_n02_virsh (class=stonith type=fence_virsh)
Attributes: pcmk_host_list=an-c03n02.alteeve.ca ipaddr=lemass
login=root passwd_script=/root/lemass.pw port=an-c03n02
Operations: monitor interval=60s (fence_n02_virsh-monitor-interval-60s)
Fencing Levels:
Location Constraints:
Resource: ClusterIP
Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
(id:cli-prefer-ClusterIP)
Resource: WebDataClone
Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
(id:cli-prefer-WebDataClone)
Resource: WebSite
Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
(id:cli-prefer-WebSite)
Ordering Constraints:
promote WebDataClone then start WebFS (Mandatory)
(id:order-WebDataClone-WebFS-mandatory)
start WebFS then start WebSite (Mandatory)
(id:order-WebFS-WebSite-mandatory)
Colocation Constraints:
WebFS with WebDataClone (INFINITY) (with-rsc-role:Master)
(id:colocation-WebFS-WebDataClone-INFINITY)
WebSite with ClusterIP (INFINITY)
(id:colocation-WebSite-ClusterIP-INFINITY)
WebSite with WebFS (INFINITY) (id:colocation-WebSite-WebFS-INFINITY)
Cluster Properties:
cluster-infrastructure: corosync
dc-version: 1.1.10-19.el7-368c726
last-lrm-refresh: 1388615115
no-quorum-policy: ignore
stonith-enabled: true
=====
Put an-c03n01 into standby:
=====
pcs cluster standby an-c03n01.alteeve.ca
==[an-c03n01 logs]===
Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.189.16
Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.190.1
87f59f171d277d9cb1b51cbda41fbf24
Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: -- <cib
admin_epoch="0" epoch="189" num_updates="16"/>
Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++
<instance_attributes id="nodes-1">
Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++
<nvpair id="nodes-1-standby" name="standby" value="on"/>
Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++
</instance_attributes>
Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move
ClusterIP (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Demote
WebData:0 (Master -> Stopped an-c03n01.alteeve.ca)
Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Promote
WebData:1 (Slave -> Master an-c03n02.alteeve.ca)
Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move
WebFS (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move
WebSite (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 37: /var/lib/pacemaker/pengine/pe-input-176.bz2
Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 15: stop ClusterIP_stop_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 6: cancel WebData_cancel_60000 on an-c03n02.alteeve.ca
Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 50: stop WebSite_stop_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 64: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca
(local)
Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 66: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
Jan 1 18:16:31 an-c03n01 apache(WebSite)[14743]: INFO: Attempting
graceful stop of apache PID 10641
Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=97, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:31 an-c03n01 IPaddr2(ClusterIP)[14742]: INFO: IP status =
ok, IP_CIP=
Jan 1 18:16:31 an-c03n01 avahi-daemon[298]: Withdrawing address record
for 192.168.122.10 on eth0.
Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation ClusterIP_stop_0 (call=94, rc=0, cib-update=283,
confirmed=true) ok
Jan 1 18:16:33 an-c03n01 apache(WebSite)[14743]: INFO: apache stopped.
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebSite_stop_0 (call=96, rc=0, cib-update=284, confirmed=true) ok
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: run_graph: Transition 37
(Complete=6, Pending=0, Fired=0, Skipped=22, Incomplete=13,
Source=/var/lib/pacemaker/pengine/pe-input-176.bz2): Stopped
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
ClusterIP (an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Demote
WebData:0 (Master -> Stopped an-c03n01.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote
WebData:1 (Slave -> Master an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Move
WebFS (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
WebSite (an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 38: /var/lib/pacemaker/pengine/pe-input-177.bz2
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 12: start ClusterIP_start_0 on an-c03n02.alteeve.ca
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 43: stop WebFS_stop_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 59: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca
(local)
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 61: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=100, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: Running stop
for /dev/drbd0 on /var/www/html
Jan 1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: Trying to
unmount /var/www/html
Jan 1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: unmounted
/var/www/html successfully
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebFS_stop_0 (call=99, rc=0, cib-update=286, confirmed=true) ok
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 14: demote WebData_demote_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 13: monitor ClusterIP_monitor_30000 on an-c03n02.alteeve.ca
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.547981] block drbd0: role(
Primary -> Secondary )
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.548034] block drbd0: bitmap
WRITE of 0 pages took 0 jiffies
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.548037] block drbd0: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_demote_0 (call=101, rc=0, cib-update=287,
confirmed=true) ok
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 60: notify WebData_post_notify_demote_0 on an-c03n01.alteeve.ca
(local)
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 62: notify WebData_post_notify_demote_0 on an-c03n02.alteeve.ca
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=102, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 54: notify WebData_pre_notify_stop_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 55: notify WebData_pre_notify_stop_0 on an-c03n02.alteeve.ca
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=103, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 15: stop WebData_stop_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638480] drbd r0: peer(
Secondary -> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate
-> DUnknown )
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638490] drbd r0: asender terminated
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638492] drbd r0: Terminating
drbd_a_r0
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638732] drbd r0: Connection closed
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638741] drbd r0: conn(
Disconnecting -> StandAlone )
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638742] drbd r0: receiver
terminated
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638743] drbd r0: Terminating
drbd_r_r0
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638776] block drbd0: disk(
UpToDate -> Failed )
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651161] block drbd0: bitmap
WRITE of 0 pages took 0 jiffies
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651164] block drbd0: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651167] block drbd0: disk(
Failed -> Diskless )
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651239] block drbd0:
drbd_bm_resize called with capacity == 0
Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651279] drbd r0: Terminating
drbd_w_r0
Jan 1 18:16:33 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-WebData (<null>)
Jan 1 18:16:33 an-c03n01 attrd[1765]: notice: attrd_perform_update:
Sent delete 154: node=1, attr=master-WebData, id=<n/a>, set=(null),
section=status
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_stop_0 (call=104, rc=0, cib-update=288, confirmed=true) ok
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 56: notify WebData_post_notify_stop_0 on an-c03n02.alteeve.ca
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: run_graph: Transition 38
(Complete=24, Pending=0, Fired=0, Skipped=10, Incomplete=4,
Source=/var/lib/pacemaker/pengine/pe-input-177.bz2): Stopped
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote
WebData:0 (Slave -> Master an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
WebFS (an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
WebSite (an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 39: /var/lib/pacemaker/pengine/pe-input-178.bz2
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote
WebData:0 (Slave -> Master an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
WebFS (an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
WebSite (an-c03n02.alteeve.ca)
Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 40: /var/lib/pacemaker/pengine/pe-input-178.bz2
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 54: notify WebData_pre_notify_promote_0 on an-c03n02.alteeve.ca
Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 16: promote WebData_promote_0 on an-c03n02.alteeve.ca
Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: handle_request: Current
ping state: S_TRANSITION_ENGINE
Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.190.11
Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.191.1
ba2e8658b93b742e3ef335fa93b50c26
Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: -- <cib
admin_epoch="0" epoch="190" num_updates="11"/>
Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++
<rsc_location rsc="WebDataClone" id="drbd-fence-by-handler-r0-WebDataClone">
Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ <rule
role="Master" score="-INFINITY"
id="drbd-fence-by-handler-r0-rule-WebDataClone">
Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++
<expression attribute="#uname" operation="ne"
value="an-c03n02.alteeve.ca"
id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ </rule>
Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++
</rsc_location>
Jan 1 18:16:34 an-c03n01 stonith-ng[1763]: notice: unpack_config: On
loss of CCM Quorum: Ignore
Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 55: notify WebData_post_notify_promote_0 on an-c03n02.alteeve.ca
Jan 1 18:16:34 an-c03n01 stonith-ng[1763]: notice:
stonith_device_register: Added 'fence_n01_virsh' to the device list (2
active devices)
Jan 1 18:16:34 an-c03n01 stonith-ng[1763]: notice:
stonith_device_register: Added 'fence_n02_virsh' to the device list (2
active devices)
Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 40
(Complete=9, Pending=0, Fired=0, Skipped=4, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-178.bz2): Stopped
Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: LogActions: Start
WebFS (an-c03n02.alteeve.ca)
Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: LogActions: Start
WebSite (an-c03n02.alteeve.ca)
Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 41: /var/lib/pacemaker/pengine/pe-input-179.bz2
Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 42: start WebFS_start_0 on an-c03n02.alteeve.ca
Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 43: monitor WebFS_monitor_60000 on an-c03n02.alteeve.ca
Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 44: start WebSite_start_0 on an-c03n02.alteeve.ca
Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 45: monitor WebSite_monitor_60000 on an-c03n02.alteeve.ca
Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 41
(Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-179.bz2): Complete
Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
==[an-c03n02 logs]===
Jan 1 18:16:31 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=90, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=92, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO: Adding inet
address 192.168.122.10/32 to device eth0
Jan 1 18:16:33 an-c03n02 avahi-daemon[295]: Registering new address
record for 192.168.122.10 on eth0.IPv4.
Jan 1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO: Bringing
device eth0 up
Jan 1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/var/run/resource-agents/send_arp-192.168.122.10 eth0 192.168.122.10
auto not_used not_used
Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation ClusterIP_start_0 (call=91, rc=0, cib-update=62,
confirmed=true) ok
Jan 1 18:16:33 an-c03n02 kernel: [19473.555292] block drbd0: peer(
Primary -> Secondary )
Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation ClusterIP_monitor_30000 (call=93, rc=0, cib-update=63,
confirmed=false) ok
Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=94, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=95, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:33 an-c03n02 kernel: [19473.645249] drbd r0: peer(
Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate ->
DUnknown )
Jan 1 18:16:33 an-c03n02 kernel: [19473.645267] drbd r0: asender terminated
Jan 1 18:16:33 an-c03n02 kernel: [19473.645269] drbd r0: Terminating
drbd_a_r0
Jan 1 18:16:33 an-c03n02 kernel: [19473.645531] drbd r0: Connection closed
Jan 1 18:16:33 an-c03n02 kernel: [19473.645538] drbd r0: conn( TearDown
-> Unconnected )
Jan 1 18:16:33 an-c03n02 kernel: [19473.645539] drbd r0: receiver
terminated
Jan 1 18:16:33 an-c03n02 kernel: [19473.645540] drbd r0: Restarting
receiver thread
Jan 1 18:16:33 an-c03n02 kernel: [19473.645541] drbd r0: receiver
(re)started
Jan 1 18:16:33 an-c03n02 kernel: [19473.645546] drbd r0: conn(
Unconnected -> WFConnection )
Jan 1 18:16:33 an-c03n02 attrd[16386]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-WebData (1000)
Jan 1 18:16:33 an-c03n02 attrd[16386]: notice: attrd_perform_update:
Sent update 152: master-WebData=1000
Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=96, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=97, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:34 an-c03n02 kernel: [19473.791812] drbd r0: helper
command: /sbin/drbdadm fence-peer r0
Jan 1 18:16:34 an-c03n02 crm-fence-peer.sh[32287]: invoked for r0
Jan 1 18:16:34 an-c03n02 cibadmin[32319]: notice: crm_log_args:
Invoked: cibadmin -C -o constraints -X <rsc_location rsc="WebDataClone"
id="drbd-fence-by-handler-r0-WebDataClone">
<rule role="Master" score="-INFINITY"
id="drbd-fence-by-handler-r0-rule-WebDataClone">
<expression attribute="#uname" operation="ne"
value="an-c03n02.alteeve.ca"
id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
</rule>
</rsc_location>
Jan 1 18:16:34 an-c03n02 kernel: [19473.836661] drbd r0: helper
command: /sbin/drbdadm fence-peer r0 exit code 4 (0x400)
Jan 1 18:16:34 an-c03n02 kernel: [19473.836664] drbd r0: fence-peer
helper returned 4 (peer was fenced)
Jan 1 18:16:34 an-c03n02 kernel: [19473.836676] drbd r0: pdsk( DUnknown
-> Outdated )
Jan 1 18:16:34 an-c03n02 kernel: [19473.836681] block drbd0: role(
Secondary -> Primary )
Jan 1 18:16:34 an-c03n02 stonith-ng[16384]: notice: unpack_config: On
loss of CCM Quorum: Ignore
Jan 1 18:16:34 an-c03n02 crm-fence-peer.sh[32287]: INFO peer is
reachable, my disk is UpToDate: placed constraint
'drbd-fence-by-handler-r0-WebDataClone'
Jan 1 18:16:34 an-c03n02 kernel: [19473.837205] block drbd0: new
current UUID
3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC
Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_promote_0 (call=98, rc=0, cib-update=64,
confirmed=true) ok
Jan 1 18:16:34 an-c03n02 stonith-ng[16384]: notice:
stonith_device_register: Device 'fence_n01_virsh' already existed in
device list (2 active devices)
Jan 1 18:16:34 an-c03n02 attrd[16386]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-WebData (10000)
Jan 1 18:16:34 an-c03n02 attrd[16386]: notice: attrd_perform_update:
Sent update 154: master-WebData=10000
Jan 1 18:16:34 an-c03n02 stonith-ng[16384]: notice:
stonith_device_register: Device 'fence_n02_virsh' already existed in
device list (2 active devices)
Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=99, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:16:34 an-c03n02 Filesystem(WebFS)[32362]: INFO: Running start
for /dev/drbd0 on /var/www/html
Jan 1 18:16:34 an-c03n02 kernel: [19473.948256] EXT4-fs (drbd0):
mounted filesystem with ordered data mode. Opts: (null)
Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebFS_start_0 (call=100, rc=0, cib-update=65, confirmed=true) ok
Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebFS_monitor_60000 (call=101, rc=0, cib-update=66,
confirmed=false) ok
Jan 1 18:16:34 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
Jan 1 18:16:34 an-c03n02 apache(WebSite)[32427]: INFO: AH00558: httpd:
Could not reliably determine the server's fully qualified domain name,
using an-c03n02.alteeve.ca. Set the 'ServerName' directive globally to
suppress this message
Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebSite_start_0 (call=102, rc=0, cib-update=67, confirmed=true) ok
Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebSite_monitor_60000 (call=103, rc=0, cib-update=68,
confirmed=false) ok
Jan 1 18:16:43 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
Jan 1 18:16:53 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
Jan 1 18:17:03 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
Jan 1 18:17:13 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
Jan 1 18:17:23 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
Jan 1 18:17:33 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
Jan 1 18:17:43 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
Jan 1 18:17:53 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
Jan 1 18:18:03 an-c03n02 systemd: pacemaker.service: Got notification
message from PID 32497, but reception only permitted for PID 16382
=====
Not sure why this happened:
=====
Jan 1 18:16:34 an-c03n02 cibadmin[32319]: notice: crm_log_args:
Invoked: cibadmin -C -o constraints -X <rsc_location rsc="WebDataClone"
id="drbd-fence-by-handler-r0-WebDataClone">
<rule role="Master" score="-INFINITY"
id="drbd-fence-by-handler-r0-rule-WebDataClone">
<expression attribute="#uname" operation="ne"
value="an-c03n02.alteeve.ca"
id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
</rule>
</rsc_location>
=====
That seems to be the problem, but let me continue.
Everything fails over properly:
=====
[root at an-c03n01 ~]# pcs status
Cluster name: an-cluster-03
Last updated: Wed Jan 1 18:19:01 2014
Last change: Wed Jan 1 18:16:34 2014 via cibadmin on an-c03n02.alteeve.ca
Stack: corosync
Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
7 Resources configured
Node an-c03n01.alteeve.ca (1): standby
Online: [ an-c03n02.alteeve.ca ]
Full list of resources:
fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n02.alteeve.ca
Master/Slave Set: WebDataClone [WebData]
Masters: [ an-c03n02.alteeve.ca ]
Stopped: [ an-c03n01.alteeve.ca ]
WebFS (ocf::heartbeat:Filesystem): Started an-c03n02.alteeve.ca
WebSite (ocf::heartbeat:apache): Started an-c03n02.alteeve.ca
PCSD Status:
an-c03n01.alteeve.ca:
an-c03n01.alteeve.ca: Online
an-c03n02.alteeve.ca:
an-c03n02.alteeve.ca: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
=====
Current config:
=====
[root at an-c03n01 ~]# pcs config show
Cluster Name: an-cluster-03
Corosync Nodes:
an-c03n01.alteeve.ca an-c03n02.alteeve.ca
Pacemaker Nodes:
an-c03n01.alteeve.ca an-c03n02.alteeve.ca
Resources:
Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.122.10 cidr_netmask=32
Operations: monitor interval=30s (ClusterIP-monitor-interval-30s)
Master: WebDataClone
Meta Attrs: master-max=1 master-node-max=1 clone-max=2
clone-node-max=1 notify=true
Resource: WebData (class=ocf provider=linbit type=drbd)
Attributes: drbd_resource=r0
Operations: monitor interval=60s (WebData-monitor-60s)
Resource: WebFS (class=ocf provider=heartbeat type=Filesystem)
Attributes: device=/dev/drbd0 directory=/var/www/html fstype=ext4
Operations: monitor interval=60s (WebFS-monitor-interval-60s)
Resource: WebSite (class=ocf provider=heartbeat type=apache)
Attributes: configfile=/etc/httpd/conf/httpd.conf
statusurl=http://127.0.0.1/server-status
Operations: monitor interval=60s (WebSite-monitor-interval-60s)
Stonith Devices:
Resource: fence_n01_virsh (class=stonith type=fence_virsh)
Attributes: pcmk_host_list=an-c03n01.alteeve.ca ipaddr=lemass
login=root passwd_script=/root/lemass.pw delay=15 port=an-c03n01
Operations: monitor interval=60s (fence_n01_virsh-monitor-interval-60s)
Resource: fence_n02_virsh (class=stonith type=fence_virsh)
Attributes: pcmk_host_list=an-c03n02.alteeve.ca ipaddr=lemass
login=root passwd_script=/root/lemass.pw port=an-c03n02
Operations: monitor interval=60s (fence_n02_virsh-monitor-interval-60s)
Fencing Levels:
Location Constraints:
Resource: ClusterIP
Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
(id:cli-prefer-ClusterIP)
Resource: WebDataClone
Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
(id:cli-prefer-WebDataClone)
Constraint: drbd-fence-by-handler-r0-WebDataClone
Rule: score=-INFINITY role=Master
(id:drbd-fence-by-handler-r0-rule-WebDataClone)
Expression: #uname ne an-c03n02.alteeve.ca
(id:drbd-fence-by-handler-r0-expr-WebDataClone)
Resource: WebSite
Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
(id:cli-prefer-WebSite)
Ordering Constraints:
promote WebDataClone then start WebFS (Mandatory)
(id:order-WebDataClone-WebFS-mandatory)
start WebFS then start WebSite (Mandatory)
(id:order-WebFS-WebSite-mandatory)
Colocation Constraints:
WebFS with WebDataClone (INFINITY) (with-rsc-role:Master)
(id:colocation-WebFS-WebDataClone-INFINITY)
WebSite with ClusterIP (INFINITY)
(id:colocation-WebSite-ClusterIP-INFINITY)
WebSite with WebFS (INFINITY) (id:colocation-WebSite-WebFS-INFINITY)
Cluster Properties:
cluster-infrastructure: corosync
dc-version: 1.1.10-19.el7-368c726
last-lrm-refresh: 1388615115
no-quorum-policy: ignore
stonith-enabled: true
Node Attributes:
an-c03n01.alteeve.ca: standby=on
=====
Put an-c03n01 into unstandby:
=====
[root at an-c03n01 ~]# pcs cluster unstandby an-c03n01.alteeve.ca
==[an-c03n01 logs]===
Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.191.7
Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.192.1
9eac78791868c0b3878e59927a934eba
Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: --
<nvpair id="nodes-1-standby" name="standby" value="on"/>
Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: ++ <cib
admin_epoch="0" cib-last-written="Wed Jan 1 18:22:32 2014"
crm_feature_set="3.0.7" epoch="192" have-quorum="1" num_updates="1"
update-client="crm_attribute" update-origin="an-c03n01.alteeve.ca"
validate-with="pacemaker-1.2" dc-uuid="1"/>
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Move
ClusterIP (Started an-c03n02.alteeve.ca -> an-c03n01.alteeve.ca)
Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Start
WebData:1 (an-c03n01.alteeve.ca)
Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Stop
WebSite (an-c03n02.alteeve.ca)
Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 42: /var/lib/pacemaker/pengine/pe-input-180.bz2
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 14: stop ClusterIP_stop_0 on an-c03n02.alteeve.ca
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 49: stop WebSite_stop_0 on an-c03n02.alteeve.ca
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 53: notify WebData_pre_notify_start_0 on an-c03n02.alteeve.ca
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 21: start WebData:1_start_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 15: start ClusterIP_start_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO: Adding inet
address 192.168.122.10/32 to device eth0
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.282453] drbd r0: Starting
worker thread (from drbdsetup [15705])
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283364] block drbd0: disk(
Diskless -> Attaching )
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283429] drbd r0: Method to
ensure write ordering: drain
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283431] block drbd0: max BIO
size = 1048576
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283433] block drbd0: Adjusting
my ra_pages to backing device's (32 -> 1024)
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283435] block drbd0:
drbd_bm_resize called with capacity == 102392704
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283580] block drbd0: resync
bitmap: bits=12799088 words=199986 pages=391
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283582] block drbd0: size = 49
GB (51196352 KB)
Jan 1 18:22:32 an-c03n01 avahi-daemon[298]: Registering new address
record for 192.168.122.10 on eth0.IPv4.
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285244] block drbd0: bitmap
READ of 391 pages took 2 jiffies
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285466] block drbd0: recounting
of set bits took additional 0 jiffies
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285468] block drbd0: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285473] block drbd0: disk(
Attaching -> Consistent )
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285475] block drbd0: attached
to UUIDs 6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
Jan 1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO: Bringing
device eth0 up
Jan 1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/var/run/resource-agents/send_arp-192.168.122.10 eth0 192.168.122.10
auto not_used not_used
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292172] drbd r0: conn(
StandAlone -> Unconnected )
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292185] drbd r0: Starting
receiver thread (from drbd_w_r0 [15707])
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292199] drbd r0: receiver
(re)started
Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292205] drbd r0: conn(
Unconnected -> WFConnection )
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation ClusterIP_start_0 (call=106, rc=0, cib-update=293,
confirmed=true) ok
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 16: monitor ClusterIP_monitor_30000 on an-c03n01.alteeve.ca (local)
Jan 1 18:22:32 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-WebData (5)
Jan 1 18:22:32 an-c03n01 attrd[1765]: notice: attrd_perform_update:
Sent update 160: master-WebData=5
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_start_0 (call=105, rc=0, cib-update=294,
confirmed=true) ok
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 54: notify WebData_post_notify_start_0 on an-c03n02.alteeve.ca
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 55: notify WebData:1_post_notify_start_0 on an-c03n01.alteeve.ca
(local)
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation ClusterIP_monitor_30000 (call=107, rc=0, cib-update=295,
confirmed=false) ok
Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=108, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793761] drbd r0: Handshake
successful: Agreed network protocol version 101
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793765] drbd r0: Agreed to
support TRIM on protocol level
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793809] drbd r0: conn(
WFConnection -> WFReportParams )
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793812] drbd r0: Starting
asender thread (from drbd_r_r0 [15727])
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799253] block drbd0:
drbd_sync_handshake:
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799258] block drbd0: self
6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
bits:0 flags:0
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799261] block drbd0: peer
3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC
bits:1 flags:0
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799263] block drbd0:
uuid_compare()=-1 by rule 50
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799270] block drbd0: peer(
Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk(
Consistent -> Outdated ) pdsk( DUnknown -> UpToDate )
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.803153] block drbd0: receive
bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23;
compression: 100.0%
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.803403] block drbd0: send
bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23;
compression: 100.0%
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.803409] block drbd0: conn(
WFBitMapT -> WFSyncUUID )
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.805842] block drbd0: updated
sync uuid
6A9A4DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.806189] block drbd0: helper
command: /sbin/drbdadm before-resync-target minor-0
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.807379] block drbd0: helper
command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.807392] block drbd0: conn(
WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.807398] block drbd0: Began
resync as SyncTarget (will sync 4 KB [1 bits set]).
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809259] block drbd0: Resync
done (total 1 sec; paused 0 sec; 4 K/sec)
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809264] block drbd0: updated
UUIDs 3E5C4A9426BEF114:0000000000000000:6A9A4DBF33FF45AE:6A994DBF33FF45AE
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809268] block drbd0: conn(
SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809382] block drbd0: helper
command: /sbin/drbdadm after-resync-target minor-0
Jan 1 18:22:33 an-c03n01 kernel: [ 3270.810777] block drbd0: helper
command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)
Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 42
(Complete=14, Pending=0, Fired=0, Skipped=2, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-180.bz2): Stopped
Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 43: /var/lib/pacemaker/pengine/pe-input-181.bz2
Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 17: monitor WebData_monitor_60000 on an-c03n01.alteeve.ca (local)
Jan 1 18:22:34 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-WebData (10000)
Jan 1 18:22:34 an-c03n01 attrd[1765]: notice: attrd_perform_update:
Sent update 162: master-WebData=10000
Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_monitor_60000 (call=109, rc=0, cib-update=297,
confirmed=false) ok
Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 43
(Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-181.bz2): Complete
Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 44: /var/lib/pacemaker/pengine/pe-input-182.bz2
Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 44
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-182.bz2): Complete
Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
==[an-c03n02 logs]===
Jan 1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=108, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:22:32 an-c03n02 apache(WebSite)[2631]: INFO: Attempting
graceful stop of apache PID 32497
Jan 1 18:22:32 an-c03n02 IPaddr2(ClusterIP)[2630]: INFO: IP status =
ok, IP_CIP=
Jan 1 18:22:32 an-c03n02 avahi-daemon[295]: Withdrawing address record
for 192.168.122.10 on eth0.
Jan 1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation ClusterIP_stop_0 (call=105, rc=0, cib-update=69,
confirmed=true) ok
Jan 1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=109, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:22:33 an-c03n02 kernel: [19832.801173] drbd r0: Handshake
successful: Agreed network protocol version 101
Jan 1 18:22:33 an-c03n02 kernel: [19832.801177] drbd r0: Agreed to
support TRIM on protocol level
Jan 1 18:22:33 an-c03n02 kernel: [19832.801209] drbd r0: conn(
WFConnection -> WFReportParams )
Jan 1 18:22:33 an-c03n02 kernel: [19832.801211] drbd r0: Starting
asender thread (from drbd_r_r0 [19965])
Jan 1 18:22:33 an-c03n02 kernel: [19832.810255] block drbd0:
drbd_sync_handshake:
Jan 1 18:22:33 an-c03n02 kernel: [19832.810260] block drbd0: self
3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC
bits:1 flags:0
Jan 1 18:22:33 an-c03n02 kernel: [19832.810263] block drbd0: peer
6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
bits:0 flags:0
Jan 1 18:22:33 an-c03n02 kernel: [19832.810265] block drbd0:
uuid_compare()=1 by rule 70
Jan 1 18:22:33 an-c03n02 kernel: [19832.810270] block drbd0: peer(
Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk(
Outdated -> Consistent )
Jan 1 18:22:33 an-c03n02 kernel: [19832.810573] block drbd0: send
bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23;
compression: 100.0%
Jan 1 18:22:33 an-c03n02 kernel: [19832.811274] block drbd0: receive
bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23;
compression: 100.0%
Jan 1 18:22:33 an-c03n02 kernel: [19832.811279] block drbd0: helper
command: /sbin/drbdadm before-resync-source minor-0
Jan 1 18:22:33 an-c03n02 kernel: [19832.812937] block drbd0: helper
command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
Jan 1 18:22:33 an-c03n02 kernel: [19832.812951] block drbd0: conn(
WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
Jan 1 18:22:33 an-c03n02 kernel: [19832.812959] block drbd0: Began
resync as SyncSource (will sync 4 KB [1 bits set]).
Jan 1 18:22:33 an-c03n02 kernel: [19832.812996] block drbd0: updated
sync UUID
3E5C4A9426BEF115:6A9A4DBF33FF45AE:6A994DBF33FF45AE:2A36647E5537B8FC
Jan 1 18:22:33 an-c03n02 kernel: [19832.816963] block drbd0: Resync
done (total 1 sec; paused 0 sec; 4 K/sec)
Jan 1 18:22:33 an-c03n02 kernel: [19832.816978] block drbd0: updated
UUIDs 3E5C4A9426BEF115:0000000000000000:6A9A4DBF33FF45AE:6A994DBF33FF45AE
Jan 1 18:22:33 an-c03n02 kernel: [19832.816985] block drbd0: conn(
SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
Jan 1 18:22:34 an-c03n02 apache(WebSite)[2631]: INFO: apache stopped.
Jan 1 18:22:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebSite_stop_0 (call=107, rc=0, cib-update=70, confirmed=true) ok
=====
WebSite stopped and won't start now.
=====
[root at an-c03n01 ~]# pcs status
Cluster name: an-cluster-03
Last updated: Wed Jan 1 18:23:44 2014
Last change: Wed Jan 1 18:22:32 2014 via crm_attribute on
an-c03n01.alteeve.ca
Stack: corosync
Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
7 Resources configured
Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
Full list of resources:
fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n01.alteeve.ca
Master/Slave Set: WebDataClone [WebData]
Masters: [ an-c03n02.alteeve.ca ]
Slaves: [ an-c03n01.alteeve.ca ]
WebFS (ocf::heartbeat:Filesystem): Started an-c03n02.alteeve.ca
WebSite (ocf::heartbeat:apache): Stopped
PCSD Status:
an-c03n01.alteeve.ca:
an-c03n01.alteeve.ca: Online
an-c03n02.alteeve.ca:
an-c03n02.alteeve.ca: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
=====
Here is what I see when I try to start it manually:
=====
[root at an-c03n01 ~]# pcs resource debug-start WebSite --full
Operation start for WebSite (ocf:heartbeat:apache) returned 0
> stderr: + 18:24:17: 46: . /usr/lib/ocf/lib/heartbeat/apache-conf.sh
> stderr: + 18:24:17: 47: . /usr/lib/ocf/lib/heartbeat/http-mon.sh
> stderr: ++ 18:24:17: 18: bind_address=127.0.0.1
> stderr: ++ 18:24:17: 19: curl_ipv6_opts=
> stderr: ++ 18:24:17: 20: ocf_is_true ''
> stderr: ++ 18:24:17: ocf_is_true:101: case "$1" in
> stderr: ++ 18:24:17: ocf_is_true:103: false
> stderr: ++ 18:24:17: 20: grep -qs ::
> stderr: ++ 18:24:17: 20: echo ''
> stderr: ++ 18:24:17: 24: WGETOPTS='-O- -q -L --no-proxy
--bind-address=127.0.0.1'
> stderr: ++ 18:24:17: 25: CURLOPTS='-o - -Ss -L --interface lo '
> stderr: + 18:24:17: 48: HA_VARRUNDIR=/var/run/
> stderr: + 18:24:17: 56: IBMHTTPD=/opt/IBMHTTPServer/bin/httpd
> stderr: + 18:24:17: 57: HTTPDLIST='/sbin/httpd2 /usr/sbin/httpd2
/usr/sbin/apache2 /sbin/httpd /usr/sbin/httpd /usr/sbin/apache
/opt/IBMHTTPServer/bin/httpd'
> stderr: + 18:24:17: 58: MPM=/usr/share/apache2/find_mpm
> stderr: + 18:24:17: 59: '[' -x /usr/share/apache2/find_mpm ']'
> stderr: + 18:24:17: 63: LOCALHOST=http://localhost
> stderr: + 18:24:17: 64: HTTPDOPTS=-DSTATUS
> stderr: + 18:24:17: 65:
DEFAULT_IBMCONFIG=/opt/IBMHTTPServer/conf/httpd.conf
> stderr: + 18:24:17: 66: DEFAULT_SUSECONFIG=/etc/apache2/httpd.conf
> stderr: + 18:24:17: 67: DEFAULT_RHELCONFIG=/etc/httpd/conf/httpd.conf
> stderr: ++ 18:24:17: 79: basename
/usr/lib/ocf/resource.d/heartbeat/apache
> stderr: + 18:24:17: 79: CMD=apache
> stderr: + 18:24:17: 654: OCF_REQUIRED_PARAMS=
> stderr: + 18:24:17: 655: OCF_REQUIRED_BINARIES=
> stderr: + 18:24:17: 656: ocf_rarun start
> stderr: + 18:24:17: ocf_rarun:137: mk_action_func
> stderr: ++ 18:24:17: mk_action_func:50: echo apache_start
> stderr: ++ 18:24:17: mk_action_func:50: tr - _
> stderr: + 18:24:17: mk_action_func:50: ACTION_FUNC=apache_start
> stderr: + 18:24:17: ocf_rarun:138: validate_args
> stderr: + 18:24:17: validate_args:53: is_function apache_start
> stderr: ++ 18:24:17: is_function:41: command -v apache_start
> stderr: + 18:24:17: is_function:41: test zapache_start = zapache_start
> stderr: + 18:24:17: ocf_rarun:139: simple_actions
> stderr: + 18:24:17: simple_actions:60: case $__OCF_ACTION in
> stderr: + 18:24:17: ocf_rarun:140: check_required_params
> stderr: + 18:24:17: check_required_params:74: local v
> stderr: + 18:24:17: ocf_rarun:141: run_function apache_getconfig
> stderr: + 18:24:17: run_function:44: is_function apache_getconfig
> stderr: ++ 18:24:17: is_function:41: command -v apache_getconfig
> stderr: + 18:24:17: is_function:41: test zapache_getconfig =
zapache_getconfig
> stderr: + 18:24:17: run_function:44: apache_getconfig
> stderr: + 18:24:17: apache_getconfig:628: HTTPD=
> stderr: + 18:24:17: apache_getconfig:629: PORT=
> stderr: + 18:24:17: apache_getconfig:630:
STATUSURL=http://127.0.0.1/server-status
> stderr: + 18:24:17: apache_getconfig:631:
CONFIGFILE=/etc/httpd/conf/httpd.conf
> stderr: + 18:24:17: apache_getconfig:632: OPTIONS=
> stderr: + 18:24:17: apache_getconfig:633: CLIENT=
> stderr: + 18:24:17: apache_getconfig:634: TESTREGEX='</ *html *>'
> stderr: + 18:24:17: apache_getconfig:635: TESTURL=
> stderr: + 18:24:17: apache_getconfig:636: TESTREGEX10=
> stderr: + 18:24:17: apache_getconfig:637: TESTCONFFILE=
> stderr: + 18:24:17: apache_getconfig:638: TESTNAME=
> stderr: + 18:24:17: apache_getconfig:639: : /etc/apache2/envvars
> stderr: + 18:24:17: apache_getconfig:640: source_envfiles
/etc/apache2/envvars
> stderr: + 18:24:17: source_envfiles:16: for f in '"$@"'
> stderr: + 18:24:17: source_envfiles:17: '[' -f /etc/apache2/envvars
-a -r /etc/apache2/envvars ']'
> stderr: + 18:24:17: apache_getconfig:642: '[' X = X -o '!' -f '' -o
'!' -x '' ']'
> stderr: + 18:24:17: apache_getconfig:643: find_httpd_prog
> stderr: + 18:24:17: find_httpd_prog:603: case $0 in
> stderr: + 18:24:17: find_httpd_prog:608: HTTPD=
> stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> stderr: + 18:24:17: find_httpd_prog:611: '[' -f /sbin/httpd2 -a -x
/sbin/httpd2 ']'
> stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> stderr: + 18:24:17: find_httpd_prog:611: '[' -f /usr/sbin/httpd2 -a
-x /usr/sbin/httpd2 ']'
> stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> stderr: + 18:24:17: find_httpd_prog:611: '[' -f /usr/sbin/apache2
-a -x /usr/sbin/apache2 ']'
> stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> stderr: + 18:24:17: find_httpd_prog:611: '[' -f /sbin/httpd -a -x
/sbin/httpd ']'
> stderr: + 18:24:17: find_httpd_prog:612: HTTPD=/sbin/httpd
> stderr: + 18:24:17: find_httpd_prog:613: break
> stderr: + 18:24:17: find_httpd_prog:618: '[' X '!=' X -a
X/sbin/httpd '!=' X ']'
> stderr: ++ 18:24:17: find_httpd_prog:621: detect_default_config
> stderr: ++ 18:24:17: detect_default_config:387: '[' -f
/etc/apache2/httpd.conf ']'
> stderr: ++ 18:24:17: detect_default_config:390: echo
/etc/httpd/conf/httpd.conf
> stderr: + 18:24:17: find_httpd_prog:621:
DefaultConfig=/etc/httpd/conf/httpd.conf
> stderr: + 18:24:17: apache_getconfig:646:
CONFIGFILE=/etc/httpd/conf/httpd.conf
> stderr: ++ 18:24:17: apache_getconfig:647: basename /sbin/httpd
> stderr: + 18:24:17: apache_getconfig:647: httpd_basename=httpd
> stderr: + 18:24:17: apache_getconfig:648: case $httpd_basename in
> stderr: + 18:24:17: apache_getconfig:651: GetParams
/etc/httpd/conf/httpd.conf
> stderr: + 18:24:17: GetParams:134:
ConfigFile=/etc/httpd/conf/httpd.conf
> stderr: + 18:24:17: GetParams:135: '[' '!' -f
/etc/httpd/conf/httpd.conf ']'
> stderr: + 18:24:17: GetParams:138: get_apache_params
/etc/httpd/conf/httpd.conf ServerRoot PidFile Port Listen
> stderr: + 18:24:17: get_apache_params:79:
configfile=/etc/httpd/conf/httpd.conf
> stderr: + 18:24:17: get_apache_params:80: shift 1
> stderr: ++ 18:24:17: get_apache_params:81: echo ServerRoot PidFile
Port Listen
> stderr: ++ 18:24:17: get_apache_params:81: sed 's/ /,/g'
> stderr: + 18:24:17: get_apache_params:81:
vars=ServerRoot,PidFile,Port,Listen
> stderr: ++ 18:24:17: get_apache_params:99: apachecat
/etc/httpd/conf/httpd.conf
> stderr: ++ 18:24:17: get_apache_params:99: awk -v
vars=ServerRoot,PidFile,Port,Listen '
> stderr: BEGIN{
> stderr: split(vars,v,",");
> stderr: for( i in v )
> stderr: vl[i]=tolower(v[i]);
> stderr: }
> stderr: {
> stderr: for( i in v )
> stderr: if( tolower($1)==vl[i] ) {
> stderr: print v[i]"="$2
> stderr: delete vl[i]
> stderr: break
> stderr: }
> stderr: }
> stderr: '
> stderr: ++ 18:24:17: apachecat:71: sed
's/#.*//;s/[[:blank:]]*$//;s/^[[:blank:]]*//'
> stderr: ++ 18:24:17: apachecat:70: awk '
> stderr: function procline() {
> stderr: split($0,a);
> stderr: if( a[1]~/^[Ii]nclude$/ ) {
> stderr: procinclude(a[2]);
> stderr: } else {
> stderr: if( a[1]=="ServerRoot" ) {
> stderr: rootdir=a[2];
> stderr: gsub("\"","",rootdir);
> stderr: }
> stderr: print;
> stderr: }
> stderr: }
> stderr: function printfile(infile, a) {
> stderr: while( (getline<infile) > 0 ) {
> stderr: procline();
> stderr: }
> stderr: close(infile);
> stderr: }
> stderr: function allfiles(dir, cmd,f) {
> stderr: cmd="find -L "dir" -type f";
> stderr: while( ( cmd | getline f ) > 0 ) {
> stderr: printfile(f);
> stderr: }
> stderr: close(cmd);
> stderr: }
> stderr: function listfiles(pattern, cmd,f) {
> stderr: cmd="ls "pattern" 2>/dev/null";
> stderr: while( ( cmd | getline f ) > 0 ) {
> stderr: printfile(f);
> stderr: }
> stderr: close(cmd);
> stderr: }
> stderr: function procinclude(spec) {
> stderr: if( rootdir!="" && spec!~/^\// ) {
> stderr: spec=rootdir"/"spec;
> stderr: }
> stderr: if( isdir(spec) ) {
> stderr: allfiles(spec); # read all files in a directory (and
subdirs)
> stderr: } else {
> stderr: listfiles(spec); # there could be jokers
> stderr: }
> stderr: }
> stderr: function isdir(s) {
> stderr: return !system("test -d \""s"\"");
> stderr: }
> stderr: { procline(); }
> stderr: ' /etc/httpd/conf/httpd.conf
> stderr: ++ 18:24:17: apachecat:72: grep -v '^$'
> stderr: + 18:24:17: get_apache_params:98: eval
'ServerRoot="/etc/httpd"' Listen=80
> stderr: ++ 18:24:17: get_apache_params:98: ServerRoot=/etc/httpd
> stderr: ++ 18:24:17: get_apache_params:98: Listen=80
> stderr: + 18:24:17: GetParams:139: case $PidFile in
> stderr: + 18:24:17: GetParams:145: PidFile=/var/run//httpd.pid
> stderr: + 18:24:17: GetParams:149: PIDFILE_DIRECTIVE=true
> stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
> stderr: + 18:24:17: GetParams:154: CheckPort ''
> stderr: + 18:24:17: CheckPort:116: ocf_is_decimal ''
> stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> stderr: + 18:24:17: ocf_is_decimal:94: false
> stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
> stderr: + 18:24:17: GetParams:154: CheckPort ''
> stderr: + 18:24:17: CheckPort:116: ocf_is_decimal ''
> stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> stderr: + 18:24:17: ocf_is_decimal:94: false
> stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
> stderr: + 18:24:17: GetParams:154: CheckPort 80
> stderr: + 18:24:17: CheckPort:116: ocf_is_decimal 80
> stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> stderr: + 18:24:17: ocf_is_decimal:96: true
> stderr: + 18:24:17: CheckPort:116: '[' 80 -gt 0 ']'
> stderr: + 18:24:17: GetParams:155: PORT=80
> stderr: + 18:24:17: GetParams:156: break
> stderr: + 18:24:17: GetParams:160: echo 80
> stderr: + 18:24:17: GetParams:160: grep :
> stderr: + 18:24:17: GetParams:161: Listen=localhost:80
> stderr: + 18:24:17: GetParams:183: '['
Xhttp://127.0.0.1/server-status = X ']'
> stderr: + 18:24:17: GetParams:189: test /var/run//httpd.pid
> stderr: + 18:24:17: GetParams:192: return 0
> stderr: + 18:24:17: ocf_rarun:142: validate_env
> stderr: + 18:24:17: validate_env:122: check_required_binaries
> stderr: + 18:24:17: check_required_binaries:113: local v
> stderr: + 18:24:17: validate_env:123: is_function apache_validate_all
> stderr: ++ 18:24:17: is_function:41: command -v apache_validate_all
> stderr: + 18:24:17: is_function:41: test zapache_validate_all =
zapache_validate_all
> stderr: + 18:24:17: validate_env:125: local rc
> stderr: + 18:24:17: validate_env:126: LSB_STATUS_STOPPED=3
> stderr: + 18:24:17: validate_env:127: apache_validate_all
> stderr: + 18:24:17: apache_validate_all:573: CheckPort 80
> stderr: + 18:24:17: CheckPort:116: ocf_is_decimal 80
> stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> stderr: + 18:24:17: ocf_is_decimal:96: true
> stderr: + 18:24:17: CheckPort:116: '[' 80 -gt 0 ']'
> stderr: + 18:24:17: apache_validate_all:575: : OK
> stderr: + 18:24:17: apache_validate_all:581: case $STATUSURL in
> stderr: + 18:24:17: apache_validate_all:588: '[' '!' -x /sbin/httpd ']'
> stderr: + 18:24:17: apache_validate_all:593: '[' '!' -f
/etc/httpd/conf/httpd.conf ']'
> stderr: + 18:24:17: apache_validate_all:599: return 0
> stderr: + 18:24:17: validate_env:128: rc=0
> stderr: + 18:24:17: validate_env:129: '[' 0 -ne 0 ']'
> stderr: + 18:24:17: ocf_rarun:143: ocf_is_probe
> stderr: + 18:24:17: ocf_is_probe:438: '[' start = monitor -a 0 = 0 ']'
> stderr: + 18:24:17: ocf_rarun:144: shift 1
> stderr: + 18:24:17: ocf_rarun:145: apache_start
> stderr: + 18:24:17: apache_start:164: silent_status
> stderr: + 18:24:17: silent_status:125: '[' -f /var/run//httpd.pid ']'
> stderr: + 18:24:17: silent_status:128: : No pid file
> stderr: + 18:24:17: silent_status:129: false
> stderr: + 18:24:17: apache_start:170: validate_default_config
> stderr: + 18:24:17: validate_default_config:135: '[' -e
/etc/SuSE-release ']'
> stderr: + 18:24:17: validate_default_config:138: return 0
> stderr: + 18:24:17: apache_start:172: '[' -d /var/run/apache2 ']'
> stderr: + 18:24:17: apache_start:174: '[' -z true ']'
> stderr: + 18:24:17: apache_start:177: ocf_run /sbin/httpd -DSTATUS
-f /etc/httpd/conf/httpd.conf -c 'PidFile /var/run//httpd.pid'
> stderr: + 18:24:17: ocf_run:361: local rc
> stderr: + 18:24:17: ocf_run:362: local output
> stderr: + 18:24:17: ocf_run:363: local verbose=1
> stderr: + 18:24:17: ocf_run:364: local loglevel=err
> stderr: + 18:24:17: ocf_run:365: local var
> stderr: + 18:24:17: ocf_run:367: for var in 1 2
> stderr: + 18:24:17: ocf_run:369: case "$1" in
> stderr: + 18:24:17: ocf_run:367: for var in 1 2
> stderr: + 18:24:17: ocf_run:369: case "$1" in
> stderr: ++ 18:24:17: ocf_run:381: /sbin/httpd -DSTATUS -f
/etc/httpd/conf/httpd.conf -c 'PidFile /var/run//httpd.pid'
> stderr: + 18:24:17: ocf_run:381: output=
> stderr: + 18:24:17: ocf_run:382: rc=0
> stderr: ++ 18:24:17: ocf_run:383: echo
> stderr: + 18:24:17: ocf_run:383: output=
> stderr: + 18:24:17: ocf_run:384: '[' 0 -eq 0 ']'
> stderr: + 18:24:17: ocf_run:385: '[' 1 -a '!' -z '' ']'
> stderr: + 18:24:17: ocf_run:388: return 0
> stderr: + 18:24:17: apache_start:180: tries=0
> stderr: + 18:24:17: apache_start:181: :
> stderr: + 18:24:17: apache_start:183: apache_monitor
> stderr: + 18:24:17: apache_monitor:367: silent_status
> stderr: + 18:24:17: silent_status:125: '[' -f /var/run//httpd.pid ']'
> stderr: ++ 18:24:17: silent_status:126: cat /var/run//httpd.pid
> stderr: + 18:24:17: silent_status:126: ProcessRunning 16404
> stderr: + 18:24:17: ProcessRunning:114: ApachePID=16404
> stderr: + 18:24:17: ProcessRunning:116: '[' -d /proc -a -d /proc/1 ']'
> stderr: + 18:24:17: ProcessRunning:117: '[' -d /proc/16404 ']'
> stderr: + 18:24:17: apache_monitor:368: '[' 0 -ne 0 ']'
> stderr: ++ 18:24:17: apache_monitor:373: findhttpclient
> stderr: ++ 18:24:17: findhttpclient:66: '[' x '!=' x ']'
> stderr: ++ 18:24:17: findhttpclient:68: which wget
> stderr: ++ 18:24:17: findhttpclient:69: echo wget
> stderr: + 18:24:17: apache_monitor:373: ourhttpclient=wget
> stderr: + 18:24:17: apache_monitor:374: '[' -z wget ']'
> stderr: + 18:24:17: apache_monitor:379: case `ocf_check_level 10` in
> stderr: ++ 18:24:17: apache_monitor:379: ocf_check_level 10
> stderr: ++ 18:24:17: ocf_check_level:643: local lvl prev
> stderr: ++ 18:24:17: ocf_check_level:644: lvl=0
> stderr: ++ 18:24:17: ocf_check_level:645: prev=0
> stderr: ++ 18:24:17: ocf_check_level:646: ocf_is_decimal 0
> stderr: ++ 18:24:17: ocf_is_decimal:92: case "$1" in
> stderr: ++ 18:24:17: ocf_is_decimal:96: true
> stderr: ++ 18:24:17: ocf_check_level:648: for lvl in '"$@"'
> stderr: ++ 18:24:17: ocf_check_level:649: '[' 10 -eq 0 ']'
> stderr: ++ 18:24:17: ocf_check_level:651: '[' 10 -gt 0 ']'
> stderr: ++ 18:24:17: ocf_check_level:652: lvl=0
> stderr: ++ 18:24:17: ocf_check_level:653: break
> stderr: ++ 18:24:17: ocf_check_level:658: echo 0
> stderr: + 18:24:17: apache_monitor:380: apache_monitor_basic
> stderr: + 18:24:17: apache_monitor_basic:351: wget_func
http://127.0.0.1/server-status
> stderr: + 18:24:17: apache_monitor_basic:351: grep -Ei '</ *html *>'
> stderr: + 18:24:17: wget_func:49: auth=
> stderr: + 18:24:17: wget_func:50: cl_opts='-O- -q -L --no-proxy
--bind-address=127.0.0.1 '
> stderr: + 18:24:17: wget_func:51: '[' x '!=' x ']'
> stderr: + 18:24:17: wget_func:53: wget -O- -q -L --no-proxy
--bind-address=127.0.0.1 http://127.0.0.1/server-status
> stderr: + 18:24:17: apache_monitor_basic:353: return 0
> stderr: + 18:24:17: apache_start:184: ec=0
> stderr: + 18:24:17: apache_start:185: '[' 0 -eq 7 ']'
> stderr: + 18:24:17: apache_start:191: break
> stderr: + 18:24:17: apache_start:195: '[' 0 -ne 0 ']'
> stderr: + 18:24:17: apache_start:198: return 0
=====
Only log entry is on an-c03n01 (where I ran the command):
=====
Jan 1 18:24:17 an-c03n01 crm_resource[16170]: notice: crm_log_args:
Invoked: /usr/sbin/crm_resource -r WebSite --force-start -V
=====
When I delete the constraint that was added when the node went into
standby, WebSite starts.
=====
[root at an-c03n01 ~]# pcs config show | grep uname -B 1
Rule: score=-INFINITY role=Master
(id:drbd-fence-by-handler-r0-rule-WebDataClone)
Expression: #uname ne an-c03n02.alteeve.ca
(id:drbd-fence-by-handler-r0-expr-WebDataClone)
[root at an-c03n01 ~]# pcs constraint delete
drbd-fence-by-handler-r0-rule-WebDataClone
=====
==[an-c03n01 logs]===
Jan 1 18:26:44 an-c03n01 cibadmin[17037]: notice: crm_log_args:
Invoked: /usr/sbin/cibadmin -c -R --xml-text <constraints>
<rsc_location id="cli-prefer-WebDataClone"
node="an-c03n01.alteeve.ca" role="Started" rsc="WebDataClone"
score="INFINITY"/>
<rsc_location id="cli-prefer-ClusterIP" node="an-c03n01.alteeve.ca"
role="Started" rsc="ClusterIP" score="INFINITY"/>
<rsc_colocation id="colocation-WebFS-WebDataClone-INFINITY"
rsc="WebFS" score="INFINITY" with-rsc="WebDataClone"
with-rsc-role="Master"/>
<rsc_order first="WebDataClone" first-ac
Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.192.9
Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.193.1
695289e37618e5e9a184d2bc8c90d421
Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: --
<rsc_location rsc="WebDataClone" id="drbd-fence-by-handler-r0-WebDataClone">
Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- <rule
role="Master" score="-INFINITY"
id="drbd-fence-by-handler-r0-rule-WebDataClone">
Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: --
<expression attribute="#uname" operation="ne"
value="an-c03n02.alteeve.ca"
id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- </rule>
Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: --
</rsc_location>
Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: ++ <cib
admin_epoch="0" cib-last-written="Wed Jan 1 18:26:44 2014"
crm_feature_set="3.0.7" epoch="193" have-quorum="1" num_updates="1"
update-client="cibadmin" update-origin="an-c03n01.alteeve.ca"
validate-with="pacemaker-1.2" dc-uuid="1"/>
Jan 1 18:26:44 an-c03n01 crmd[1767]: notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jan 1 18:26:44 an-c03n01 stonith-ng[1763]: notice: unpack_config: On
loss of CCM Quorum: Ignore
Jan 1 18:26:44 an-c03n01 crmd[1767]: notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Jan 1 18:26:44 an-c03n01 stonith-ng[1763]: notice:
stonith_device_register: Added 'fence_n01_virsh' to the device list (2
active devices)
Jan 1 18:26:44 an-c03n01 stonith-ng[1763]: notice:
stonith_device_register: Added 'fence_n02_virsh' to the device list (2
active devices)
Jan 1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Jan 1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-WebData (10000)
Jan 1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Promote
WebData:0 (Slave -> Master an-c03n01.alteeve.ca)
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Demote
WebData:1 (Master -> Slave an-c03n02.alteeve.ca)
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Move
WebFS (Started an-c03n02.alteeve.ca -> an-c03n01.alteeve.ca)
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start
WebSite (an-c03n01.alteeve.ca)
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 45: /var/lib/pacemaker/pengine/pe-input-183.bz2
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 2: cancel WebData_cancel_60000 on an-c03n01.alteeve.ca (local)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 47: stop WebFS_stop_0 on an-c03n02.alteeve.ca
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 67: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca
(local)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 69: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=111, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: run_graph: Transition 45
(Complete=5, Pending=0, Fired=0, Skipped=15, Incomplete=10,
Source=/var/lib/pacemaker/pengine/pe-input-183.bz2): Stopped
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: unpack_config: On loss
of CCM Quorum: Ignore
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Promote
WebData:0 (Slave -> Master an-c03n01.alteeve.ca)
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Demote
WebData:1 (Master -> Slave an-c03n02.alteeve.ca)
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start
WebFS (an-c03n01.alteeve.ca)
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start
WebSite (an-c03n01.alteeve.ca)
Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: process_pe_message:
Calculated Transition 46: /var/lib/pacemaker/pengine/pe-input-184.bz2
Jan 1 18:26:46 an-c03n01 crmd[1767]: warning: destroy_action:
Cancelling timer for action 2 (src=476)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 64: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca
(local)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 66: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=112, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 18: demote WebData_demote_0 on an-c03n02.alteeve.ca
Jan 1 18:26:46 an-c03n01 kernel: [ 3524.517000] block drbd0: peer(
Primary -> Secondary )
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 65: notify WebData_post_notify_demote_0 on an-c03n01.alteeve.ca
(local)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 67: notify WebData_post_notify_demote_0 on an-c03n02.alteeve.ca
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=113, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 60: notify WebData_pre_notify_promote_0 on an-c03n01.alteeve.ca
(local)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 62: notify WebData_pre_notify_promote_0 on an-c03n02.alteeve.ca
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=114, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 16: promote WebData_promote_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:26:46 an-c03n01 kernel: [ 3524.605607] block drbd0: role(
Secondary -> Primary )
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_promote_0 (call=115, rc=0, cib-update=324,
confirmed=true) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 61: notify WebData_post_notify_promote_0 on an-c03n01.alteeve.ca
(local)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 63: notify WebData_post_notify_promote_0 on an-c03n02.alteeve.ca
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=116, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 45: start WebFS_start_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 20: monitor WebData_monitor_60000 on an-c03n02.alteeve.ca
Jan 1 18:26:46 an-c03n01 Filesystem(WebFS)[17211]: INFO: Running start
for /dev/drbd0 on /var/www/html
Jan 1 18:26:46 an-c03n01 kernel: [ 3524.688633] EXT4-fs (drbd0):
mounted filesystem with ordered data mode. Opts: (null)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebFS_start_0 (call=117, rc=0, cib-update=325, confirmed=true) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 46: monitor WebFS_monitor_60000 on an-c03n01.alteeve.ca (local)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 47: start WebSite_start_0 on an-c03n01.alteeve.ca (local)
Jan 1 18:26:46 an-c03n01 apache(WebSite)[17276]: INFO: apache already
running (pid 16404)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebSite_start_0 (call=119, rc=0, cib-update=326,
confirmed=true) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebFS_monitor_60000 (call=118, rc=0, cib-update=327,
confirmed=false) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
action 48: monitor WebSite_monitor_60000 on an-c03n01.alteeve.ca (local)
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
operation WebSite_monitor_60000 (call=120, rc=0, cib-update=328,
confirmed=false) ok
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: run_graph: Transition 46
(Complete=27, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-input-184.bz2): Complete
Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
==[an-c03n02 logs]===
Jan 1 18:26:44 an-c03n02 crmd[16388]: notice: do_state_transition:
State transition S_NOT_DC -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Jan 1 18:26:44 an-c03n02 stonith-ng[16384]: notice: unpack_config: On
loss of CCM Quorum: Ignore
Jan 1 18:26:44 an-c03n02 stonith-ng[16384]: notice:
stonith_device_register: Device 'fence_n01_virsh' already existed in
device list (2 active devices)
Jan 1 18:26:44 an-c03n02 stonith-ng[16384]: notice:
stonith_device_register: Device 'fence_n02_virsh' already existed in
device list (2 active devices)
Jan 1 18:26:45 an-c03n02 crmd[16388]: notice: do_state_transition:
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC
cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jan 1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Jan 1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_trigger_update:
Sending flush op to all hosts for: master-WebData (10000)
Jan 1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=112, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: Running stop
for /dev/drbd0 on /var/www/html
Jan 1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: Trying to
unmount /var/www/html
Jan 1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: unmounted
/var/www/html successfully
Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebFS_stop_0 (call=111, rc=0, cib-update=73, confirmed=true) ok
Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=113, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n02 kernel: [20086.520840] block drbd0: role(
Primary -> Secondary )
Jan 1 18:26:46 an-c03n02 kernel: [20086.520911] block drbd0: bitmap
WRITE of 0 pages took 0 jiffies
Jan 1 18:26:46 an-c03n02 kernel: [20086.520914] block drbd0: 0 KB (0
bits) marked out-of-sync by on disk bit-map.
Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_demote_0 (call=114, rc=0, cib-update=74,
confirmed=true) ok
Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=115, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=116, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n02 kernel: [20086.610152] block drbd0: peer(
Secondary -> Primary )
Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_notify_0 (call=117, rc=0, cib-update=0, confirmed=true) ok
Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
operation WebData_monitor_60000 (call=118, rc=0, cib-update=75,
confirmed=false) ok
=====
Now WebSite is working:
=====
[root at an-c03n01 ~]# pcs status
Cluster name: an-cluster-03
Last updated: Wed Jan 1 18:28:04 2014
Last change: Wed Jan 1 18:26:44 2014 via cibadmin on an-c03n01.alteeve.ca
Stack: corosync
Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
Version: 1.1.10-19.el7-368c726
2 Nodes configured
7 Resources configured
Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
Full list of resources:
fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n01.alteeve.ca
Master/Slave Set: WebDataClone [WebData]
Masters: [ an-c03n01.alteeve.ca ]
Slaves: [ an-c03n02.alteeve.ca ]
WebFS (ocf::heartbeat:Filesystem): Started an-c03n01.alteeve.ca
WebSite (ocf::heartbeat:apache): Started an-c03n01.alteeve.ca
PCSD Status:
an-c03n01.alteeve.ca:
an-c03n01.alteeve.ca: Online
an-c03n02.alteeve.ca:
an-c03n02.alteeve.ca: Online
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
=====
Is this a bug?
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
More information about the Pacemaker
mailing list