[Pacemaker] pacemaker 1.1.10 + RHEL 7 beta issues
Digimer
lists at alteeve.ca
Wed Jan 1 18:38:47 EST 2014
I didn't show here, but 'pidof httpd' confirmed that apache was not
runnong on either node until the errant constraint was cleared.
On 01/01/14 06:30 PM, Digimer wrote:
> I have a problem where putting a node into standby causes my "WebSite"
> resource to stop and refuse to start. This is the reproducer I show
> here, but I have seen this happen with DRBD earlier (before I realized
> what was wrong, so I didn't document it), so I don't think this is an
> apache RA issue.
>
> [root at an-c03n01 ~]# rpm -q pacemaker httpd pcs corosync
> pacemaker-1.1.10-19.el7.x86_64
> httpd-2.4.6-7.el7.x86_64
> pcs-0.9.99-2.el7.x86_64
> corosync-2.3.2-2.el7.x86_64
>
> [root at an-c03n01 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Everything release 7.0 Beta (Maipo)
>
> [root at an-c03n01 ~]# uname -a
> Linux an-c03n01.alteeve.ca 3.10.0-54.0.1.el7.x86_64 #1 SMP Tue Nov 26
> 16:51:22 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
>
> DRBD installed from source:
>
> [root at an-c03n01 ~]# cat /proc/drbd
> version: 8.4.4 (api:1/proto:86-101)
> GIT-hash: 74402fecf24da8e5438171ee8c19e28627e1c98a build by
> root at an-c03n01.alteeve.ca, 2013-12-23 20:36:27
>
>
> Starting with everything working:
>
> =====
> [root at an-c03n01 ~]# pcs status
> Cluster name: an-cluster-03
> Last updated: Wed Jan 1 18:13:19 2014
> Last change: Wed Jan 1 18:03:08 2014 via cibadmin on an-c03n01.alteeve.ca
> Stack: corosync
> Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
> Version: 1.1.10-19.el7-368c726
> 2 Nodes configured
> 7 Resources configured
>
>
> Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
>
> Full list of resources:
>
> fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n01.alteeve.ca
> Master/Slave Set: WebDataClone [WebData]
> Masters: [ an-c03n01.alteeve.ca ]
> Slaves: [ an-c03n02.alteeve.ca ]
> WebFS (ocf::heartbeat:Filesystem): Started an-c03n01.alteeve.ca
> WebSite (ocf::heartbeat:apache): Started an-c03n01.alteeve.ca
>
> PCSD Status:
> an-c03n01.alteeve.ca:
> an-c03n01.alteeve.ca: Online
> an-c03n02.alteeve.ca:
> an-c03n02.alteeve.ca: Online
>
> Daemon Status:
> corosync: active/disabled
> pacemaker: active/disabled
> pcsd: active/enabled
> =====
>
> =====
> [root at an-c03n01 ~]# pcs config show
> Cluster Name: an-cluster-03
> Corosync Nodes:
> an-c03n01.alteeve.ca an-c03n02.alteeve.ca
> Pacemaker Nodes:
> an-c03n01.alteeve.ca an-c03n02.alteeve.ca
>
> Resources:
> Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
> Attributes: ip=192.168.122.10 cidr_netmask=32
> Operations: monitor interval=30s (ClusterIP-monitor-interval-30s)
> Master: WebDataClone
> Meta Attrs: master-max=1 master-node-max=1 clone-max=2
> clone-node-max=1 notify=true
> Resource: WebData (class=ocf provider=linbit type=drbd)
> Attributes: drbd_resource=r0
> Operations: monitor interval=60s (WebData-monitor-60s)
> Resource: WebFS (class=ocf provider=heartbeat type=Filesystem)
> Attributes: device=/dev/drbd0 directory=/var/www/html fstype=ext4
> Operations: monitor interval=60s (WebFS-monitor-interval-60s)
> Resource: WebSite (class=ocf provider=heartbeat type=apache)
> Attributes: configfile=/etc/httpd/conf/httpd.conf
> statusurl=http://127.0.0.1/server-status
> Operations: monitor interval=60s (WebSite-monitor-interval-60s)
>
> Stonith Devices:
> Resource: fence_n01_virsh (class=stonith type=fence_virsh)
> Attributes: pcmk_host_list=an-c03n01.alteeve.ca ipaddr=lemass
> login=root passwd_script=/root/lemass.pw delay=15 port=an-c03n01
> Operations: monitor interval=60s (fence_n01_virsh-monitor-interval-60s)
> Resource: fence_n02_virsh (class=stonith type=fence_virsh)
> Attributes: pcmk_host_list=an-c03n02.alteeve.ca ipaddr=lemass
> login=root passwd_script=/root/lemass.pw port=an-c03n02
> Operations: monitor interval=60s (fence_n02_virsh-monitor-interval-60s)
> Fencing Levels:
>
> Location Constraints:
> Resource: ClusterIP
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-ClusterIP)
> Resource: WebDataClone
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-WebDataClone)
> Resource: WebSite
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-WebSite)
> Ordering Constraints:
> promote WebDataClone then start WebFS (Mandatory)
> (id:order-WebDataClone-WebFS-mandatory)
> start WebFS then start WebSite (Mandatory)
> (id:order-WebFS-WebSite-mandatory)
> Colocation Constraints:
> WebFS with WebDataClone (INFINITY) (with-rsc-role:Master)
> (id:colocation-WebFS-WebDataClone-INFINITY)
> WebSite with ClusterIP (INFINITY)
> (id:colocation-WebSite-ClusterIP-INFINITY)
> WebSite with WebFS (INFINITY) (id:colocation-WebSite-WebFS-INFINITY)
>
> Cluster Properties:
> cluster-infrastructure: corosync
> dc-version: 1.1.10-19.el7-368c726
> last-lrm-refresh: 1388615115
> no-quorum-policy: ignore
> stonith-enabled: true
> =====
>
> Put an-c03n01 into standby:
>
> =====
> pcs cluster standby an-c03n01.alteeve.ca
>
> ==[an-c03n01 logs]===
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.189.16
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.190.1
> 87f59f171d277d9cb1b51cbda41fbf24
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: -- <cib
> admin_epoch="0" epoch="189" num_updates="16"/>
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++
> <instance_attributes id="nodes-1">
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++ <nvpair
> id="nodes-1-standby" name="standby" value="on"/>
> Jan 1 18:16:31 an-c03n01 cib[1762]: notice: cib:diff: ++
> </instance_attributes>
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move
> ClusterIP (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Demote
> WebData:0 (Master -> Stopped an-c03n01.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:1 (Slave -> Master an-c03n02.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move
> WebFS (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: LogActions: Move
> WebSite (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
> Jan 1 18:16:31 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 37: /var/lib/pacemaker/pengine/pe-input-176.bz2
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 15: stop ClusterIP_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 6: cancel WebData_cancel_60000 on an-c03n02.alteeve.ca
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 50: stop WebSite_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 64: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca
> (local)
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 66: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:31 an-c03n01 apache(WebSite)[14743]: INFO: Attempting
> graceful stop of apache PID 10641
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=97, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:31 an-c03n01 IPaddr2(ClusterIP)[14742]: INFO: IP status =
> ok, IP_CIP=
> Jan 1 18:16:31 an-c03n01 avahi-daemon[298]: Withdrawing address record
> for 192.168.122.10 on eth0.
> Jan 1 18:16:31 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation ClusterIP_stop_0 (call=94, rc=0, cib-update=283,
> confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 apache(WebSite)[14743]: INFO: apache stopped.
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebSite_stop_0 (call=96, rc=0, cib-update=284, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: run_graph: Transition 37
> (Complete=6, Pending=0, Fired=0, Skipped=22, Incomplete=13,
> Source=/var/lib/pacemaker/pengine/pe-input-176.bz2): Stopped
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
> ClusterIP (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Demote
> WebData:0 (Master -> Stopped an-c03n01.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:1 (Slave -> Master an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Move
> WebFS (Started an-c03n01.alteeve.ca -> an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebSite (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 38: /var/lib/pacemaker/pengine/pe-input-177.bz2
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 12: start ClusterIP_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 43: stop WebFS_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 59: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca
> (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 61: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=100, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: Running stop
> for /dev/drbd0 on /var/www/html
> Jan 1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: Trying to
> unmount /var/www/html
> Jan 1 18:16:33 an-c03n01 Filesystem(WebFS)[14862]: INFO: unmounted
> /var/www/html successfully
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebFS_stop_0 (call=99, rc=0, cib-update=286, confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 14: demote WebData_demote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 13: monitor ClusterIP_monitor_30000 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.547981] block drbd0: role(
> Primary -> Secondary )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.548034] block drbd0: bitmap
> WRITE of 0 pages took 0 jiffies
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.548037] block drbd0: 0 KB (0
> bits) marked out-of-sync by on disk bit-map.
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_demote_0 (call=101, rc=0, cib-update=287,
> confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 60: notify WebData_post_notify_demote_0 on an-c03n01.alteeve.ca
> (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 62: notify WebData_post_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=102, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 54: notify WebData_pre_notify_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 55: notify WebData_pre_notify_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=103, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 15: stop WebData_stop_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638480] drbd r0: peer(
> Secondary -> Unknown ) conn( Connected -> Disconnecting ) pdsk( UpToDate
> -> DUnknown )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638490] drbd r0: asender
> terminated
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638492] drbd r0: Terminating
> drbd_a_r0
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638732] drbd r0: Connection closed
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638741] drbd r0: conn(
> Disconnecting -> StandAlone )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638742] drbd r0: receiver
> terminated
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638743] drbd r0: Terminating
> drbd_r_r0
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.638776] block drbd0: disk(
> UpToDate -> Failed )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651161] block drbd0: bitmap
> WRITE of 0 pages took 0 jiffies
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651164] block drbd0: 0 KB (0
> bits) marked out-of-sync by on disk bit-map.
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651167] block drbd0: disk(
> Failed -> Diskless )
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651239] block drbd0:
> drbd_bm_resize called with capacity == 0
> Jan 1 18:16:33 an-c03n01 kernel: [ 2911.651279] drbd r0: Terminating
> drbd_w_r0
> Jan 1 18:16:33 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: master-WebData (<null>)
> Jan 1 18:16:33 an-c03n01 attrd[1765]: notice: attrd_perform_update:
> Sent delete 154: node=1, attr=master-WebData, id=<n/a>, set=(null),
> section=status
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_stop_0 (call=104, rc=0, cib-update=288,
> confirmed=true) ok
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 56: notify WebData_post_notify_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: run_graph: Transition 38
> (Complete=24, Pending=0, Fired=0, Skipped=10, Incomplete=4,
> Source=/var/lib/pacemaker/pengine/pe-input-177.bz2): Stopped
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:0 (Slave -> Master an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebFS (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebSite (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 39: /var/lib/pacemaker/pengine/pe-input-178.bz2
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:0 (Slave -> Master an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebFS (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebSite (an-c03n02.alteeve.ca)
> Jan 1 18:16:33 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 40: /var/lib/pacemaker/pengine/pe-input-178.bz2
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 54: notify WebData_pre_notify_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:33 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 16: promote WebData_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: handle_request: Current
> ping state: S_TRANSITION_ENGINE
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.190.11
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.191.1
> ba2e8658b93b742e3ef335fa93b50c26
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: -- <cib
> admin_epoch="0" epoch="190" num_updates="11"/>
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ <rsc_location
> rsc="WebDataClone" id="drbd-fence-by-handler-r0-WebDataClone">
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ <rule
> role="Master" score="-INFINITY"
> id="drbd-fence-by-handler-r0-rule-WebDataClone">
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ <expression
> attribute="#uname" operation="ne" value="an-c03n02.alteeve.ca"
> id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ </rule>
> Jan 1 18:16:34 an-c03n01 cib[1762]: notice: cib:diff: ++ </rsc_location>
> Jan 1 18:16:34 an-c03n01 stonith-ng[1763]: notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 55: notify WebData_post_notify_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 stonith-ng[1763]: notice:
> stonith_device_register: Added 'fence_n01_virsh' to the device list (2
> active devices)
> Jan 1 18:16:34 an-c03n01 stonith-ng[1763]: notice:
> stonith_device_register: Added 'fence_n02_virsh' to the device list (2
> active devices)
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 40
> (Complete=9, Pending=0, Fired=0, Skipped=4, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-178.bz2): Stopped
> Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebFS (an-c03n02.alteeve.ca)
> Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebSite (an-c03n02.alteeve.ca)
> Jan 1 18:16:34 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 41: /var/lib/pacemaker/pengine/pe-input-179.bz2
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 42: start WebFS_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 43: monitor WebFS_monitor_60000 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 44: start WebSite_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 45: monitor WebSite_monitor_60000 on an-c03n02.alteeve.ca
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 41
> (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-179.bz2): Complete
> Jan 1 18:16:34 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
>
> ==[an-c03n02 logs]===
> Jan 1 18:16:31 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=90, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=92, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO: Adding inet
> address 192.168.122.10/32 to device eth0
> Jan 1 18:16:33 an-c03n02 avahi-daemon[295]: Registering new address
> record for 192.168.122.10 on eth0.IPv4.
> Jan 1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO: Bringing
> device eth0 up
> Jan 1 18:16:33 an-c03n02 IPaddr2(ClusterIP)[32025]: INFO:
> /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
> /var/run/resource-agents/send_arp-192.168.122.10 eth0 192.168.122.10
> auto not_used not_used
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation ClusterIP_start_0 (call=91, rc=0, cib-update=62,
> confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 kernel: [19473.555292] block drbd0: peer(
> Primary -> Secondary )
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation ClusterIP_monitor_30000 (call=93, rc=0, cib-update=63,
> confirmed=false) ok
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=94, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=95, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645249] drbd r0: peer(
> Secondary -> Unknown ) conn( Connected -> TearDown ) pdsk( UpToDate ->
> DUnknown )
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645267] drbd r0: asender
> terminated
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645269] drbd r0: Terminating
> drbd_a_r0
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645531] drbd r0: Connection closed
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645538] drbd r0: conn( TearDown
> -> Unconnected )
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645539] drbd r0: receiver
> terminated
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645540] drbd r0: Restarting
> receiver thread
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645541] drbd r0: receiver
> (re)started
> Jan 1 18:16:33 an-c03n02 kernel: [19473.645546] drbd r0: conn(
> Unconnected -> WFConnection )
> Jan 1 18:16:33 an-c03n02 attrd[16386]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: master-WebData (1000)
> Jan 1 18:16:33 an-c03n02 attrd[16386]: notice: attrd_perform_update:
> Sent update 152: master-WebData=1000
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=96, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:33 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=97, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 kernel: [19473.791812] drbd r0: helper
> command: /sbin/drbdadm fence-peer r0
> Jan 1 18:16:34 an-c03n02 crm-fence-peer.sh[32287]: invoked for r0
> Jan 1 18:16:34 an-c03n02 cibadmin[32319]: notice: crm_log_args:
> Invoked: cibadmin -C -o constraints -X <rsc_location rsc="WebDataClone"
> id="drbd-fence-by-handler-r0-WebDataClone">
> <rule role="Master" score="-INFINITY"
> id="drbd-fence-by-handler-r0-rule-WebDataClone">
> <expression attribute="#uname" operation="ne"
> value="an-c03n02.alteeve.ca"
> id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
> </rule>
> </rsc_location>
> Jan 1 18:16:34 an-c03n02 kernel: [19473.836661] drbd r0: helper
> command: /sbin/drbdadm fence-peer r0 exit code 4 (0x400)
> Jan 1 18:16:34 an-c03n02 kernel: [19473.836664] drbd r0: fence-peer
> helper returned 4 (peer was fenced)
> Jan 1 18:16:34 an-c03n02 kernel: [19473.836676] drbd r0: pdsk( DUnknown
> -> Outdated )
> Jan 1 18:16:34 an-c03n02 kernel: [19473.836681] block drbd0: role(
> Secondary -> Primary )
> Jan 1 18:16:34 an-c03n02 stonith-ng[16384]: notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 1 18:16:34 an-c03n02 crm-fence-peer.sh[32287]: INFO peer is
> reachable, my disk is UpToDate: placed constraint
> 'drbd-fence-by-handler-r0-WebDataClone'
> Jan 1 18:16:34 an-c03n02 kernel: [19473.837205] block drbd0: new
> current UUID
> 3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_promote_0 (call=98, rc=0, cib-update=64,
> confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 stonith-ng[16384]: notice:
> stonith_device_register: Device 'fence_n01_virsh' already existed in
> device list (2 active devices)
> Jan 1 18:16:34 an-c03n02 attrd[16386]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: master-WebData (10000)
> Jan 1 18:16:34 an-c03n02 attrd[16386]: notice: attrd_perform_update:
> Sent update 154: master-WebData=10000
> Jan 1 18:16:34 an-c03n02 stonith-ng[16384]: notice:
> stonith_device_register: Device 'fence_n02_virsh' already existed in
> device list (2 active devices)
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=99, rc=0, cib-update=0, confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 Filesystem(WebFS)[32362]: INFO: Running start
> for /dev/drbd0 on /var/www/html
> Jan 1 18:16:34 an-c03n02 kernel: [19473.948256] EXT4-fs (drbd0):
> mounted filesystem with ordered data mode. Opts: (null)
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebFS_start_0 (call=100, rc=0, cib-update=65, confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebFS_monitor_60000 (call=101, rc=0, cib-update=66,
> confirmed=false) ok
> Jan 1 18:16:34 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:16:34 an-c03n02 apache(WebSite)[32427]: INFO: AH00558: httpd:
> Could not reliably determine the server's fully qualified domain name,
> using an-c03n02.alteeve.ca. Set the 'ServerName' directive globally to
> suppress this message
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebSite_start_0 (call=102, rc=0, cib-update=67,
> confirmed=true) ok
> Jan 1 18:16:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebSite_monitor_60000 (call=103, rc=0, cib-update=68,
> confirmed=false) ok
> Jan 1 18:16:43 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:16:53 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:03 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:13 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:23 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:33 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:43 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:17:53 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> Jan 1 18:18:03 an-c03n02 systemd: pacemaker.service: Got notification
> message from PID 32497, but reception only permitted for PID 16382
> =====
>
> Not sure why this happened:
> =====
> Jan 1 18:16:34 an-c03n02 cibadmin[32319]: notice: crm_log_args:
> Invoked: cibadmin -C -o constraints -X <rsc_location rsc="WebDataClone"
> id="drbd-fence-by-handler-r0-WebDataClone">
> <rule role="Master" score="-INFINITY"
> id="drbd-fence-by-handler-r0-rule-WebDataClone">
> <expression attribute="#uname" operation="ne"
> value="an-c03n02.alteeve.ca"
> id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
> </rule>
> </rsc_location>
> =====
>
> That seems to be the problem, but let me continue.
>
> Everything fails over properly:
>
> =====
> [root at an-c03n01 ~]# pcs status
> Cluster name: an-cluster-03
> Last updated: Wed Jan 1 18:19:01 2014
> Last change: Wed Jan 1 18:16:34 2014 via cibadmin on an-c03n02.alteeve.ca
> Stack: corosync
> Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
> Version: 1.1.10-19.el7-368c726
> 2 Nodes configured
> 7 Resources configured
>
>
> Node an-c03n01.alteeve.ca (1): standby
> Online: [ an-c03n02.alteeve.ca ]
>
> Full list of resources:
>
> fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n02.alteeve.ca
> Master/Slave Set: WebDataClone [WebData]
> Masters: [ an-c03n02.alteeve.ca ]
> Stopped: [ an-c03n01.alteeve.ca ]
> WebFS (ocf::heartbeat:Filesystem): Started an-c03n02.alteeve.ca
> WebSite (ocf::heartbeat:apache): Started an-c03n02.alteeve.ca
>
> PCSD Status:
> an-c03n01.alteeve.ca:
> an-c03n01.alteeve.ca: Online
> an-c03n02.alteeve.ca:
> an-c03n02.alteeve.ca: Online
>
> Daemon Status:
> corosync: active/disabled
> pacemaker: active/disabled
> pcsd: active/enabled
> =====
>
> Current config:
> =====
> [root at an-c03n01 ~]# pcs config show
> Cluster Name: an-cluster-03
> Corosync Nodes:
> an-c03n01.alteeve.ca an-c03n02.alteeve.ca
> Pacemaker Nodes:
> an-c03n01.alteeve.ca an-c03n02.alteeve.ca
>
> Resources:
> Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
> Attributes: ip=192.168.122.10 cidr_netmask=32
> Operations: monitor interval=30s (ClusterIP-monitor-interval-30s)
> Master: WebDataClone
> Meta Attrs: master-max=1 master-node-max=1 clone-max=2
> clone-node-max=1 notify=true
> Resource: WebData (class=ocf provider=linbit type=drbd)
> Attributes: drbd_resource=r0
> Operations: monitor interval=60s (WebData-monitor-60s)
> Resource: WebFS (class=ocf provider=heartbeat type=Filesystem)
> Attributes: device=/dev/drbd0 directory=/var/www/html fstype=ext4
> Operations: monitor interval=60s (WebFS-monitor-interval-60s)
> Resource: WebSite (class=ocf provider=heartbeat type=apache)
> Attributes: configfile=/etc/httpd/conf/httpd.conf
> statusurl=http://127.0.0.1/server-status
> Operations: monitor interval=60s (WebSite-monitor-interval-60s)
>
> Stonith Devices:
> Resource: fence_n01_virsh (class=stonith type=fence_virsh)
> Attributes: pcmk_host_list=an-c03n01.alteeve.ca ipaddr=lemass
> login=root passwd_script=/root/lemass.pw delay=15 port=an-c03n01
> Operations: monitor interval=60s (fence_n01_virsh-monitor-interval-60s)
> Resource: fence_n02_virsh (class=stonith type=fence_virsh)
> Attributes: pcmk_host_list=an-c03n02.alteeve.ca ipaddr=lemass
> login=root passwd_script=/root/lemass.pw port=an-c03n02
> Operations: monitor interval=60s (fence_n02_virsh-monitor-interval-60s)
> Fencing Levels:
>
> Location Constraints:
> Resource: ClusterIP
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-ClusterIP)
> Resource: WebDataClone
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-WebDataClone)
> Constraint: drbd-fence-by-handler-r0-WebDataClone
> Rule: score=-INFINITY role=Master
> (id:drbd-fence-by-handler-r0-rule-WebDataClone)
> Expression: #uname ne an-c03n02.alteeve.ca
> (id:drbd-fence-by-handler-r0-expr-WebDataClone)
> Resource: WebSite
> Enabled on: an-c03n01.alteeve.ca (score:INFINITY) (role: Started)
> (id:cli-prefer-WebSite)
> Ordering Constraints:
> promote WebDataClone then start WebFS (Mandatory)
> (id:order-WebDataClone-WebFS-mandatory)
> start WebFS then start WebSite (Mandatory)
> (id:order-WebFS-WebSite-mandatory)
> Colocation Constraints:
> WebFS with WebDataClone (INFINITY) (with-rsc-role:Master)
> (id:colocation-WebFS-WebDataClone-INFINITY)
> WebSite with ClusterIP (INFINITY)
> (id:colocation-WebSite-ClusterIP-INFINITY)
> WebSite with WebFS (INFINITY) (id:colocation-WebSite-WebFS-INFINITY)
>
> Cluster Properties:
> cluster-infrastructure: corosync
> dc-version: 1.1.10-19.el7-368c726
> last-lrm-refresh: 1388615115
> no-quorum-policy: ignore
> stonith-enabled: true
> Node Attributes:
> an-c03n01.alteeve.ca: standby=on
> =====
>
> Put an-c03n01 into unstandby:
>
> =====
> [root at an-c03n01 ~]# pcs cluster unstandby an-c03n01.alteeve.ca
>
> ==[an-c03n01 logs]===
> Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.191.7
> Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.192.1
> 9eac78791868c0b3878e59927a934eba
> Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: -- <nvpair
> id="nodes-1-standby" name="standby" value="on"/>
> Jan 1 18:22:32 an-c03n01 cib[1762]: notice: cib:diff: ++ <cib
> admin_epoch="0" cib-last-written="Wed Jan 1 18:22:32 2014"
> crm_feature_set="3.0.7" epoch="192" have-quorum="1" num_updates="1"
> update-client="crm_attribute" update-origin="an-c03n01.alteeve.ca"
> validate-with="pacemaker-1.2" dc-uuid="1"/>
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Move
> ClusterIP (Started an-c03n02.alteeve.ca -> an-c03n01.alteeve.ca)
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebData:1 (an-c03n01.alteeve.ca)
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: LogActions: Stop
> WebSite (an-c03n02.alteeve.ca)
> Jan 1 18:22:32 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 42: /var/lib/pacemaker/pengine/pe-input-180.bz2
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 14: stop ClusterIP_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 49: stop WebSite_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 53: notify WebData_pre_notify_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 21: start WebData:1_start_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 15: start ClusterIP_start_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO: Adding inet
> address 192.168.122.10/32 to device eth0
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.282453] drbd r0: Starting
> worker thread (from drbdsetup [15705])
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283364] block drbd0: disk(
> Diskless -> Attaching )
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283429] drbd r0: Method to
> ensure write ordering: drain
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283431] block drbd0: max BIO
> size = 1048576
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283433] block drbd0: Adjusting
> my ra_pages to backing device's (32 -> 1024)
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283435] block drbd0:
> drbd_bm_resize called with capacity == 102392704
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283580] block drbd0: resync
> bitmap: bits=12799088 words=199986 pages=391
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.283582] block drbd0: size = 49
> GB (51196352 KB)
> Jan 1 18:22:32 an-c03n01 avahi-daemon[298]: Registering new address
> record for 192.168.122.10 on eth0.IPv4.
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285244] block drbd0: bitmap
> READ of 391 pages took 2 jiffies
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285466] block drbd0: recounting
> of set bits took additional 0 jiffies
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285468] block drbd0: 0 KB (0
> bits) marked out-of-sync by on disk bit-map.
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285473] block drbd0: disk(
> Attaching -> Consistent )
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.285475] block drbd0: attached
> to UUIDs
> 6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
> Jan 1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO: Bringing
> device eth0 up
> Jan 1 18:22:32 an-c03n01 IPaddr2(ClusterIP)[15640]: INFO:
> /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
> /var/run/resource-agents/send_arp-192.168.122.10 eth0 192.168.122.10
> auto not_used not_used
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292172] drbd r0: conn(
> StandAlone -> Unconnected )
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292185] drbd r0: Starting
> receiver thread (from drbd_w_r0 [15707])
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292199] drbd r0: receiver
> (re)started
> Jan 1 18:22:32 an-c03n01 kernel: [ 3270.292205] drbd r0: conn(
> Unconnected -> WFConnection )
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation ClusterIP_start_0 (call=106, rc=0, cib-update=293,
> confirmed=true) ok
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 16: monitor ClusterIP_monitor_30000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:22:32 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: master-WebData (5)
> Jan 1 18:22:32 an-c03n01 attrd[1765]: notice: attrd_perform_update:
> Sent update 160: master-WebData=5
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_start_0 (call=105, rc=0, cib-update=294,
> confirmed=true) ok
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 54: notify WebData_post_notify_start_0 on an-c03n02.alteeve.ca
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 55: notify WebData:1_post_notify_start_0 on an-c03n01.alteeve.ca
> (local)
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation ClusterIP_monitor_30000 (call=107, rc=0, cib-update=295,
> confirmed=false) ok
> Jan 1 18:22:32 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=108, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793761] drbd r0: Handshake
> successful: Agreed network protocol version 101
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793765] drbd r0: Agreed to
> support TRIM on protocol level
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793809] drbd r0: conn(
> WFConnection -> WFReportParams )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.793812] drbd r0: Starting
> asender thread (from drbd_r_r0 [15727])
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799253] block drbd0:
> drbd_sync_handshake:
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799258] block drbd0: self
> 6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
> bits:0 flags:0
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799261] block drbd0: peer
> 3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC
> bits:1 flags:0
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799263] block drbd0:
> uuid_compare()=-1 by rule 50
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.799270] block drbd0: peer(
> Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk(
> Consistent -> Outdated ) pdsk( DUnknown -> UpToDate )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.803153] block drbd0: receive
> bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23;
> compression: 100.0%
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.803403] block drbd0: send
> bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23;
> compression: 100.0%
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.803409] block drbd0: conn(
> WFBitMapT -> WFSyncUUID )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.805842] block drbd0: updated
> sync uuid
> 6A9A4DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.806189] block drbd0: helper
> command: /sbin/drbdadm before-resync-target minor-0
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.807379] block drbd0: helper
> command: /sbin/drbdadm before-resync-target minor-0 exit code 0 (0x0)
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.807392] block drbd0: conn(
> WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.807398] block drbd0: Began
> resync as SyncTarget (will sync 4 KB [1 bits set]).
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809259] block drbd0: Resync
> done (total 1 sec; paused 0 sec; 4 K/sec)
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809264] block drbd0: updated
> UUIDs 3E5C4A9426BEF114:0000000000000000:6A9A4DBF33FF45AE:6A994DBF33FF45AE
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809268] block drbd0: conn(
> SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.809382] block drbd0: helper
> command: /sbin/drbdadm after-resync-target minor-0
> Jan 1 18:22:33 an-c03n01 kernel: [ 3270.810777] block drbd0: helper
> command: /sbin/drbdadm after-resync-target minor-0 exit code 0 (0x0)
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 42
> (Complete=14, Pending=0, Fired=0, Skipped=2, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-180.bz2): Stopped
> Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 43: /var/lib/pacemaker/pengine/pe-input-181.bz2
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 17: monitor WebData_monitor_60000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:22:34 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: master-WebData (10000)
> Jan 1 18:22:34 an-c03n01 attrd[1765]: notice: attrd_perform_update:
> Sent update 162: master-WebData=10000
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_monitor_60000 (call=109, rc=0, cib-update=297,
> confirmed=false) ok
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 43
> (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-181.bz2): Complete
> Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:22:34 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 44: /var/lib/pacemaker/pengine/pe-input-182.bz2
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: run_graph: Transition 44
> (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-182.bz2): Complete
> Jan 1 18:22:34 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
>
> ==[an-c03n02 logs]===
> Jan 1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=108, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:22:32 an-c03n02 apache(WebSite)[2631]: INFO: Attempting
> graceful stop of apache PID 32497
> Jan 1 18:22:32 an-c03n02 IPaddr2(ClusterIP)[2630]: INFO: IP status =
> ok, IP_CIP=
> Jan 1 18:22:32 an-c03n02 avahi-daemon[295]: Withdrawing address record
> for 192.168.122.10 on eth0.
> Jan 1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation ClusterIP_stop_0 (call=105, rc=0, cib-update=69,
> confirmed=true) ok
> Jan 1 18:22:32 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=109, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:22:33 an-c03n02 kernel: [19832.801173] drbd r0: Handshake
> successful: Agreed network protocol version 101
> Jan 1 18:22:33 an-c03n02 kernel: [19832.801177] drbd r0: Agreed to
> support TRIM on protocol level
> Jan 1 18:22:33 an-c03n02 kernel: [19832.801209] drbd r0: conn(
> WFConnection -> WFReportParams )
> Jan 1 18:22:33 an-c03n02 kernel: [19832.801211] drbd r0: Starting
> asender thread (from drbd_r_r0 [19965])
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810255] block drbd0:
> drbd_sync_handshake:
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810260] block drbd0: self
> 3E5C4A9426BEF115:6A994DBF33FF45AE:2A36647E5537B8FC:2A35647E5537B8FC
> bits:1 flags:0
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810263] block drbd0: peer
> 6A994DBF33FF45AE:0000000000000000:2A36647E5537B8FC:2A35647E5537B8FC
> bits:0 flags:0
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810265] block drbd0:
> uuid_compare()=1 by rule 70
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810270] block drbd0: peer(
> Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk(
> Outdated -> Consistent )
> Jan 1 18:22:33 an-c03n02 kernel: [19832.810573] block drbd0: send
> bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23;
> compression: 100.0%
> Jan 1 18:22:33 an-c03n02 kernel: [19832.811274] block drbd0: receive
> bitmap stats [Bytes(packets)]: plain 0(0), RLE 23(1), total 23;
> compression: 100.0%
> Jan 1 18:22:33 an-c03n02 kernel: [19832.811279] block drbd0: helper
> command: /sbin/drbdadm before-resync-source minor-0
> Jan 1 18:22:33 an-c03n02 kernel: [19832.812937] block drbd0: helper
> command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
> Jan 1 18:22:33 an-c03n02 kernel: [19832.812951] block drbd0: conn(
> WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
> Jan 1 18:22:33 an-c03n02 kernel: [19832.812959] block drbd0: Began
> resync as SyncSource (will sync 4 KB [1 bits set]).
> Jan 1 18:22:33 an-c03n02 kernel: [19832.812996] block drbd0: updated
> sync UUID
> 3E5C4A9426BEF115:6A9A4DBF33FF45AE:6A994DBF33FF45AE:2A36647E5537B8FC
> Jan 1 18:22:33 an-c03n02 kernel: [19832.816963] block drbd0: Resync
> done (total 1 sec; paused 0 sec; 4 K/sec)
> Jan 1 18:22:33 an-c03n02 kernel: [19832.816978] block drbd0: updated
> UUIDs 3E5C4A9426BEF115:0000000000000000:6A9A4DBF33FF45AE:6A994DBF33FF45AE
> Jan 1 18:22:33 an-c03n02 kernel: [19832.816985] block drbd0: conn(
> SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )
> Jan 1 18:22:34 an-c03n02 apache(WebSite)[2631]: INFO: apache stopped.
> Jan 1 18:22:34 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebSite_stop_0 (call=107, rc=0, cib-update=70, confirmed=true) ok
> =====
>
> WebSite stopped and won't start now.
> =====
> [root at an-c03n01 ~]# pcs status
> Cluster name: an-cluster-03
> Last updated: Wed Jan 1 18:23:44 2014
> Last change: Wed Jan 1 18:22:32 2014 via crm_attribute on
> an-c03n01.alteeve.ca
> Stack: corosync
> Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
> Version: 1.1.10-19.el7-368c726
> 2 Nodes configured
> 7 Resources configured
>
>
> Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
>
> Full list of resources:
>
> fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n01.alteeve.ca
> Master/Slave Set: WebDataClone [WebData]
> Masters: [ an-c03n02.alteeve.ca ]
> Slaves: [ an-c03n01.alteeve.ca ]
> WebFS (ocf::heartbeat:Filesystem): Started an-c03n02.alteeve.ca
> WebSite (ocf::heartbeat:apache): Stopped
>
> PCSD Status:
> an-c03n01.alteeve.ca:
> an-c03n01.alteeve.ca: Online
> an-c03n02.alteeve.ca:
> an-c03n02.alteeve.ca: Online
>
> Daemon Status:
> corosync: active/disabled
> pacemaker: active/disabled
> pcsd: active/enabled
> =====
>
> Here is what I see when I try to start it manually:
>
> =====
> [root at an-c03n01 ~]# pcs resource debug-start WebSite --full
> Operation start for WebSite (ocf:heartbeat:apache) returned 0
> > stderr: + 18:24:17: 46: . /usr/lib/ocf/lib/heartbeat/apache-conf.sh
> > stderr: + 18:24:17: 47: . /usr/lib/ocf/lib/heartbeat/http-mon.sh
> > stderr: ++ 18:24:17: 18: bind_address=127.0.0.1
> > stderr: ++ 18:24:17: 19: curl_ipv6_opts=
> > stderr: ++ 18:24:17: 20: ocf_is_true ''
> > stderr: ++ 18:24:17: ocf_is_true:101: case "$1" in
> > stderr: ++ 18:24:17: ocf_is_true:103: false
> > stderr: ++ 18:24:17: 20: grep -qs ::
> > stderr: ++ 18:24:17: 20: echo ''
> > stderr: ++ 18:24:17: 24: WGETOPTS='-O- -q -L --no-proxy
> --bind-address=127.0.0.1'
> > stderr: ++ 18:24:17: 25: CURLOPTS='-o - -Ss -L --interface lo '
> > stderr: + 18:24:17: 48: HA_VARRUNDIR=/var/run/
> > stderr: + 18:24:17: 56: IBMHTTPD=/opt/IBMHTTPServer/bin/httpd
> > stderr: + 18:24:17: 57: HTTPDLIST='/sbin/httpd2 /usr/sbin/httpd2
> /usr/sbin/apache2 /sbin/httpd /usr/sbin/httpd /usr/sbin/apache
> /opt/IBMHTTPServer/bin/httpd'
> > stderr: + 18:24:17: 58: MPM=/usr/share/apache2/find_mpm
> > stderr: + 18:24:17: 59: '[' -x /usr/share/apache2/find_mpm ']'
> > stderr: + 18:24:17: 63: LOCALHOST=http://localhost
> > stderr: + 18:24:17: 64: HTTPDOPTS=-DSTATUS
> > stderr: + 18:24:17: 65:
> DEFAULT_IBMCONFIG=/opt/IBMHTTPServer/conf/httpd.conf
> > stderr: + 18:24:17: 66: DEFAULT_SUSECONFIG=/etc/apache2/httpd.conf
> > stderr: + 18:24:17: 67: DEFAULT_RHELCONFIG=/etc/httpd/conf/httpd.conf
> > stderr: ++ 18:24:17: 79: basename
> /usr/lib/ocf/resource.d/heartbeat/apache
> > stderr: + 18:24:17: 79: CMD=apache
> > stderr: + 18:24:17: 654: OCF_REQUIRED_PARAMS=
> > stderr: + 18:24:17: 655: OCF_REQUIRED_BINARIES=
> > stderr: + 18:24:17: 656: ocf_rarun start
> > stderr: + 18:24:17: ocf_rarun:137: mk_action_func
> > stderr: ++ 18:24:17: mk_action_func:50: echo apache_start
> > stderr: ++ 18:24:17: mk_action_func:50: tr - _
> > stderr: + 18:24:17: mk_action_func:50: ACTION_FUNC=apache_start
> > stderr: + 18:24:17: ocf_rarun:138: validate_args
> > stderr: + 18:24:17: validate_args:53: is_function apache_start
> > stderr: ++ 18:24:17: is_function:41: command -v apache_start
> > stderr: + 18:24:17: is_function:41: test zapache_start = zapache_start
> > stderr: + 18:24:17: ocf_rarun:139: simple_actions
> > stderr: + 18:24:17: simple_actions:60: case $__OCF_ACTION in
> > stderr: + 18:24:17: ocf_rarun:140: check_required_params
> > stderr: + 18:24:17: check_required_params:74: local v
> > stderr: + 18:24:17: ocf_rarun:141: run_function apache_getconfig
> > stderr: + 18:24:17: run_function:44: is_function apache_getconfig
> > stderr: ++ 18:24:17: is_function:41: command -v apache_getconfig
> > stderr: + 18:24:17: is_function:41: test zapache_getconfig =
> zapache_getconfig
> > stderr: + 18:24:17: run_function:44: apache_getconfig
> > stderr: + 18:24:17: apache_getconfig:628: HTTPD=
> > stderr: + 18:24:17: apache_getconfig:629: PORT=
> > stderr: + 18:24:17: apache_getconfig:630:
> STATUSURL=http://127.0.0.1/server-status
> > stderr: + 18:24:17: apache_getconfig:631:
> CONFIGFILE=/etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: apache_getconfig:632: OPTIONS=
> > stderr: + 18:24:17: apache_getconfig:633: CLIENT=
> > stderr: + 18:24:17: apache_getconfig:634: TESTREGEX='</ *html *>'
> > stderr: + 18:24:17: apache_getconfig:635: TESTURL=
> > stderr: + 18:24:17: apache_getconfig:636: TESTREGEX10=
> > stderr: + 18:24:17: apache_getconfig:637: TESTCONFFILE=
> > stderr: + 18:24:17: apache_getconfig:638: TESTNAME=
> > stderr: + 18:24:17: apache_getconfig:639: : /etc/apache2/envvars
> > stderr: + 18:24:17: apache_getconfig:640: source_envfiles
> /etc/apache2/envvars
> > stderr: + 18:24:17: source_envfiles:16: for f in '"$@"'
> > stderr: + 18:24:17: source_envfiles:17: '[' -f /etc/apache2/envvars
> -a -r /etc/apache2/envvars ']'
> > stderr: + 18:24:17: apache_getconfig:642: '[' X = X -o '!' -f '' -o
> '!' -x '' ']'
> > stderr: + 18:24:17: apache_getconfig:643: find_httpd_prog
> > stderr: + 18:24:17: find_httpd_prog:603: case $0 in
> > stderr: + 18:24:17: find_httpd_prog:608: HTTPD=
> > stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> > stderr: + 18:24:17: find_httpd_prog:611: '[' -f /sbin/httpd2 -a -x
> /sbin/httpd2 ']'
> > stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> > stderr: + 18:24:17: find_httpd_prog:611: '[' -f /usr/sbin/httpd2 -a
> -x /usr/sbin/httpd2 ']'
> > stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> > stderr: + 18:24:17: find_httpd_prog:611: '[' -f /usr/sbin/apache2
> -a -x /usr/sbin/apache2 ']'
> > stderr: + 18:24:17: find_httpd_prog:609: for h in '$HTTPDLIST'
> > stderr: + 18:24:17: find_httpd_prog:611: '[' -f /sbin/httpd -a -x
> /sbin/httpd ']'
> > stderr: + 18:24:17: find_httpd_prog:612: HTTPD=/sbin/httpd
> > stderr: + 18:24:17: find_httpd_prog:613: break
> > stderr: + 18:24:17: find_httpd_prog:618: '[' X '!=' X -a
> X/sbin/httpd '!=' X ']'
> > stderr: ++ 18:24:17: find_httpd_prog:621: detect_default_config
> > stderr: ++ 18:24:17: detect_default_config:387: '[' -f
> /etc/apache2/httpd.conf ']'
> > stderr: ++ 18:24:17: detect_default_config:390: echo
> /etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: find_httpd_prog:621:
> DefaultConfig=/etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: apache_getconfig:646:
> CONFIGFILE=/etc/httpd/conf/httpd.conf
> > stderr: ++ 18:24:17: apache_getconfig:647: basename /sbin/httpd
> > stderr: + 18:24:17: apache_getconfig:647: httpd_basename=httpd
> > stderr: + 18:24:17: apache_getconfig:648: case $httpd_basename in
> > stderr: + 18:24:17: apache_getconfig:651: GetParams
> /etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: GetParams:134:
> ConfigFile=/etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: GetParams:135: '[' '!' -f
> /etc/httpd/conf/httpd.conf ']'
> > stderr: + 18:24:17: GetParams:138: get_apache_params
> /etc/httpd/conf/httpd.conf ServerRoot PidFile Port Listen
> > stderr: + 18:24:17: get_apache_params:79:
> configfile=/etc/httpd/conf/httpd.conf
> > stderr: + 18:24:17: get_apache_params:80: shift 1
> > stderr: ++ 18:24:17: get_apache_params:81: echo ServerRoot PidFile
> Port Listen
> > stderr: ++ 18:24:17: get_apache_params:81: sed 's/ /,/g'
> > stderr: + 18:24:17: get_apache_params:81:
> vars=ServerRoot,PidFile,Port,Listen
> > stderr: ++ 18:24:17: get_apache_params:99: apachecat
> /etc/httpd/conf/httpd.conf
> > stderr: ++ 18:24:17: get_apache_params:99: awk -v
> vars=ServerRoot,PidFile,Port,Listen '
> > stderr: BEGIN{
> > stderr: split(vars,v,",");
> > stderr: for( i in v )
> > stderr: vl[i]=tolower(v[i]);
> > stderr: }
> > stderr: {
> > stderr: for( i in v )
> > stderr: if( tolower($1)==vl[i] ) {
> > stderr: print v[i]"="$2
> > stderr: delete vl[i]
> > stderr: break
> > stderr: }
> > stderr: }
> > stderr: '
> > stderr: ++ 18:24:17: apachecat:71: sed
> 's/#.*//;s/[[:blank:]]*$//;s/^[[:blank:]]*//'
> > stderr: ++ 18:24:17: apachecat:70: awk '
> > stderr: function procline() {
> > stderr: split($0,a);
> > stderr: if( a[1]~/^[Ii]nclude$/ ) {
> > stderr: procinclude(a[2]);
> > stderr: } else {
> > stderr: if( a[1]=="ServerRoot" ) {
> > stderr: rootdir=a[2];
> > stderr: gsub("\"","",rootdir);
> > stderr: }
> > stderr: print;
> > stderr: }
> > stderr: }
> > stderr: function printfile(infile, a) {
> > stderr: while( (getline<infile) > 0 ) {
> > stderr: procline();
> > stderr: }
> > stderr: close(infile);
> > stderr: }
> > stderr: function allfiles(dir, cmd,f) {
> > stderr: cmd="find -L "dir" -type f";
> > stderr: while( ( cmd | getline f ) > 0 ) {
> > stderr: printfile(f);
> > stderr: }
> > stderr: close(cmd);
> > stderr: }
> > stderr: function listfiles(pattern, cmd,f) {
> > stderr: cmd="ls "pattern" 2>/dev/null";
> > stderr: while( ( cmd | getline f ) > 0 ) {
> > stderr: printfile(f);
> > stderr: }
> > stderr: close(cmd);
> > stderr: }
> > stderr: function procinclude(spec) {
> > stderr: if( rootdir!="" && spec!~/^\// ) {
> > stderr: spec=rootdir"/"spec;
> > stderr: }
> > stderr: if( isdir(spec) ) {
> > stderr: allfiles(spec); # read all files in a directory
> (and subdirs)
> > stderr: } else {
> > stderr: listfiles(spec); # there could be jokers
> > stderr: }
> > stderr: }
> > stderr: function isdir(s) {
> > stderr: return !system("test -d \""s"\"");
> > stderr: }
> > stderr: { procline(); }
> > stderr: ' /etc/httpd/conf/httpd.conf
> > stderr: ++ 18:24:17: apachecat:72: grep -v '^$'
> > stderr: + 18:24:17: get_apache_params:98: eval
> 'ServerRoot="/etc/httpd"' Listen=80
> > stderr: ++ 18:24:17: get_apache_params:98: ServerRoot=/etc/httpd
> > stderr: ++ 18:24:17: get_apache_params:98: Listen=80
> > stderr: + 18:24:17: GetParams:139: case $PidFile in
> > stderr: + 18:24:17: GetParams:145: PidFile=/var/run//httpd.pid
> > stderr: + 18:24:17: GetParams:149: PIDFILE_DIRECTIVE=true
> > stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
> > stderr: + 18:24:17: GetParams:154: CheckPort ''
> > stderr: + 18:24:17: CheckPort:116: ocf_is_decimal ''
> > stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: + 18:24:17: ocf_is_decimal:94: false
> > stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
> > stderr: + 18:24:17: GetParams:154: CheckPort ''
> > stderr: + 18:24:17: CheckPort:116: ocf_is_decimal ''
> > stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: + 18:24:17: ocf_is_decimal:94: false
> > stderr: + 18:24:17: GetParams:153: for p in '"$PORT"' '"$Port"' 80
> > stderr: + 18:24:17: GetParams:154: CheckPort 80
> > stderr: + 18:24:17: CheckPort:116: ocf_is_decimal 80
> > stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: + 18:24:17: ocf_is_decimal:96: true
> > stderr: + 18:24:17: CheckPort:116: '[' 80 -gt 0 ']'
> > stderr: + 18:24:17: GetParams:155: PORT=80
> > stderr: + 18:24:17: GetParams:156: break
> > stderr: + 18:24:17: GetParams:160: echo 80
> > stderr: + 18:24:17: GetParams:160: grep :
> > stderr: + 18:24:17: GetParams:161: Listen=localhost:80
> > stderr: + 18:24:17: GetParams:183: '['
> Xhttp://127.0.0.1/server-status = X ']'
> > stderr: + 18:24:17: GetParams:189: test /var/run//httpd.pid
> > stderr: + 18:24:17: GetParams:192: return 0
> > stderr: + 18:24:17: ocf_rarun:142: validate_env
> > stderr: + 18:24:17: validate_env:122: check_required_binaries
> > stderr: + 18:24:17: check_required_binaries:113: local v
> > stderr: + 18:24:17: validate_env:123: is_function apache_validate_all
> > stderr: ++ 18:24:17: is_function:41: command -v apache_validate_all
> > stderr: + 18:24:17: is_function:41: test zapache_validate_all =
> zapache_validate_all
> > stderr: + 18:24:17: validate_env:125: local rc
> > stderr: + 18:24:17: validate_env:126: LSB_STATUS_STOPPED=3
> > stderr: + 18:24:17: validate_env:127: apache_validate_all
> > stderr: + 18:24:17: apache_validate_all:573: CheckPort 80
> > stderr: + 18:24:17: CheckPort:116: ocf_is_decimal 80
> > stderr: + 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: + 18:24:17: ocf_is_decimal:96: true
> > stderr: + 18:24:17: CheckPort:116: '[' 80 -gt 0 ']'
> > stderr: + 18:24:17: apache_validate_all:575: : OK
> > stderr: + 18:24:17: apache_validate_all:581: case $STATUSURL in
> > stderr: + 18:24:17: apache_validate_all:588: '[' '!' -x /sbin/httpd
> ']'
> > stderr: + 18:24:17: apache_validate_all:593: '[' '!' -f
> /etc/httpd/conf/httpd.conf ']'
> > stderr: + 18:24:17: apache_validate_all:599: return 0
> > stderr: + 18:24:17: validate_env:128: rc=0
> > stderr: + 18:24:17: validate_env:129: '[' 0 -ne 0 ']'
> > stderr: + 18:24:17: ocf_rarun:143: ocf_is_probe
> > stderr: + 18:24:17: ocf_is_probe:438: '[' start = monitor -a 0 = 0 ']'
> > stderr: + 18:24:17: ocf_rarun:144: shift 1
> > stderr: + 18:24:17: ocf_rarun:145: apache_start
> > stderr: + 18:24:17: apache_start:164: silent_status
> > stderr: + 18:24:17: silent_status:125: '[' -f /var/run//httpd.pid ']'
> > stderr: + 18:24:17: silent_status:128: : No pid file
> > stderr: + 18:24:17: silent_status:129: false
> > stderr: + 18:24:17: apache_start:170: validate_default_config
> > stderr: + 18:24:17: validate_default_config:135: '[' -e
> /etc/SuSE-release ']'
> > stderr: + 18:24:17: validate_default_config:138: return 0
> > stderr: + 18:24:17: apache_start:172: '[' -d /var/run/apache2 ']'
> > stderr: + 18:24:17: apache_start:174: '[' -z true ']'
> > stderr: + 18:24:17: apache_start:177: ocf_run /sbin/httpd -DSTATUS
> -f /etc/httpd/conf/httpd.conf -c 'PidFile /var/run//httpd.pid'
> > stderr: + 18:24:17: ocf_run:361: local rc
> > stderr: + 18:24:17: ocf_run:362: local output
> > stderr: + 18:24:17: ocf_run:363: local verbose=1
> > stderr: + 18:24:17: ocf_run:364: local loglevel=err
> > stderr: + 18:24:17: ocf_run:365: local var
> > stderr: + 18:24:17: ocf_run:367: for var in 1 2
> > stderr: + 18:24:17: ocf_run:369: case "$1" in
> > stderr: + 18:24:17: ocf_run:367: for var in 1 2
> > stderr: + 18:24:17: ocf_run:369: case "$1" in
> > stderr: ++ 18:24:17: ocf_run:381: /sbin/httpd -DSTATUS -f
> /etc/httpd/conf/httpd.conf -c 'PidFile /var/run//httpd.pid'
> > stderr: + 18:24:17: ocf_run:381: output=
> > stderr: + 18:24:17: ocf_run:382: rc=0
> > stderr: ++ 18:24:17: ocf_run:383: echo
> > stderr: + 18:24:17: ocf_run:383: output=
> > stderr: + 18:24:17: ocf_run:384: '[' 0 -eq 0 ']'
> > stderr: + 18:24:17: ocf_run:385: '[' 1 -a '!' -z '' ']'
> > stderr: + 18:24:17: ocf_run:388: return 0
> > stderr: + 18:24:17: apache_start:180: tries=0
> > stderr: + 18:24:17: apache_start:181: :
> > stderr: + 18:24:17: apache_start:183: apache_monitor
> > stderr: + 18:24:17: apache_monitor:367: silent_status
> > stderr: + 18:24:17: silent_status:125: '[' -f /var/run//httpd.pid ']'
> > stderr: ++ 18:24:17: silent_status:126: cat /var/run//httpd.pid
> > stderr: + 18:24:17: silent_status:126: ProcessRunning 16404
> > stderr: + 18:24:17: ProcessRunning:114: ApachePID=16404
> > stderr: + 18:24:17: ProcessRunning:116: '[' -d /proc -a -d /proc/1 ']'
> > stderr: + 18:24:17: ProcessRunning:117: '[' -d /proc/16404 ']'
> > stderr: + 18:24:17: apache_monitor:368: '[' 0 -ne 0 ']'
> > stderr: ++ 18:24:17: apache_monitor:373: findhttpclient
> > stderr: ++ 18:24:17: findhttpclient:66: '[' x '!=' x ']'
> > stderr: ++ 18:24:17: findhttpclient:68: which wget
> > stderr: ++ 18:24:17: findhttpclient:69: echo wget
> > stderr: + 18:24:17: apache_monitor:373: ourhttpclient=wget
> > stderr: + 18:24:17: apache_monitor:374: '[' -z wget ']'
> > stderr: + 18:24:17: apache_monitor:379: case `ocf_check_level 10` in
> > stderr: ++ 18:24:17: apache_monitor:379: ocf_check_level 10
> > stderr: ++ 18:24:17: ocf_check_level:643: local lvl prev
> > stderr: ++ 18:24:17: ocf_check_level:644: lvl=0
> > stderr: ++ 18:24:17: ocf_check_level:645: prev=0
> > stderr: ++ 18:24:17: ocf_check_level:646: ocf_is_decimal 0
> > stderr: ++ 18:24:17: ocf_is_decimal:92: case "$1" in
> > stderr: ++ 18:24:17: ocf_is_decimal:96: true
> > stderr: ++ 18:24:17: ocf_check_level:648: for lvl in '"$@"'
> > stderr: ++ 18:24:17: ocf_check_level:649: '[' 10 -eq 0 ']'
> > stderr: ++ 18:24:17: ocf_check_level:651: '[' 10 -gt 0 ']'
> > stderr: ++ 18:24:17: ocf_check_level:652: lvl=0
> > stderr: ++ 18:24:17: ocf_check_level:653: break
> > stderr: ++ 18:24:17: ocf_check_level:658: echo 0
> > stderr: + 18:24:17: apache_monitor:380: apache_monitor_basic
> > stderr: + 18:24:17: apache_monitor_basic:351: wget_func
> http://127.0.0.1/server-status
> > stderr: + 18:24:17: apache_monitor_basic:351: grep -Ei '</ *html *>'
> > stderr: + 18:24:17: wget_func:49: auth=
> > stderr: + 18:24:17: wget_func:50: cl_opts='-O- -q -L --no-proxy
> --bind-address=127.0.0.1 '
> > stderr: + 18:24:17: wget_func:51: '[' x '!=' x ']'
> > stderr: + 18:24:17: wget_func:53: wget -O- -q -L --no-proxy
> --bind-address=127.0.0.1 http://127.0.0.1/server-status
> > stderr: + 18:24:17: apache_monitor_basic:353: return 0
> > stderr: + 18:24:17: apache_start:184: ec=0
> > stderr: + 18:24:17: apache_start:185: '[' 0 -eq 7 ']'
> > stderr: + 18:24:17: apache_start:191: break
> > stderr: + 18:24:17: apache_start:195: '[' 0 -ne 0 ']'
> > stderr: + 18:24:17: apache_start:198: return 0
> =====
>
> Only log entry is on an-c03n01 (where I ran the command):
>
> =====
> Jan 1 18:24:17 an-c03n01 crm_resource[16170]: notice: crm_log_args:
> Invoked: /usr/sbin/crm_resource -r WebSite --force-start -V
> =====
>
> When I delete the constraint that was added when the node went into
> standby, WebSite starts.
>
> =====
> [root at an-c03n01 ~]# pcs config show | grep uname -B 1
> Rule: score=-INFINITY role=Master
> (id:drbd-fence-by-handler-r0-rule-WebDataClone)
> Expression: #uname ne an-c03n02.alteeve.ca
> (id:drbd-fence-by-handler-r0-expr-WebDataClone)
> [root at an-c03n01 ~]# pcs constraint delete
> drbd-fence-by-handler-r0-rule-WebDataClone
> =====
>
> ==[an-c03n01 logs]===
> Jan 1 18:26:44 an-c03n01 cibadmin[17037]: notice: crm_log_args:
> Invoked: /usr/sbin/cibadmin -c -R --xml-text <constraints>
> <rsc_location id="cli-prefer-WebDataClone"
> node="an-c03n01.alteeve.ca" role="Started" rsc="WebDataClone"
> score="INFINITY"/>
> <rsc_location id="cli-prefer-ClusterIP" node="an-c03n01.alteeve.ca"
> role="Started" rsc="ClusterIP" score="INFINITY"/>
> <rsc_colocation id="colocation-WebFS-WebDataClone-INFINITY"
> rsc="WebFS" score="INFINITY" with-rsc="WebDataClone"
> with-rsc-role="Master"/>
> <rsc_order first="WebDataClone" first-ac
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: Diff: --- 0.192.9
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: Diff: +++ 0.193.1
> 695289e37618e5e9a184d2bc8c90d421
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- <rsc_location
> rsc="WebDataClone" id="drbd-fence-by-handler-r0-WebDataClone">
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- <rule
> role="Master" score="-INFINITY"
> id="drbd-fence-by-handler-r0-rule-WebDataClone">
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- <expression
> attribute="#uname" operation="ne" value="an-c03n02.alteeve.ca"
> id="drbd-fence-by-handler-r0-expr-WebDataClone"/>
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- </rule>
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: -- </rsc_location>
> Jan 1 18:26:44 an-c03n01 cib[1762]: notice: cib:diff: ++ <cib
> admin_epoch="0" cib-last-written="Wed Jan 1 18:26:44 2014"
> crm_feature_set="3.0.7" epoch="193" have-quorum="1" num_updates="1"
> update-client="cibadmin" update-origin="an-c03n01.alteeve.ca"
> validate-with="pacemaker-1.2" dc-uuid="1"/>
> Jan 1 18:26:44 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
> cause=C_FSA_INTERNAL origin=abort_transition_graph ]
> Jan 1 18:26:44 an-c03n01 stonith-ng[1763]: notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 1 18:26:44 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
> cause=C_FSA_INTERNAL origin=do_election_check ]
> Jan 1 18:26:44 an-c03n01 stonith-ng[1763]: notice:
> stonith_device_register: Added 'fence_n01_virsh' to the device list (2
> active devices)
> Jan 1 18:26:44 an-c03n01 stonith-ng[1763]: notice:
> stonith_device_register: Added 'fence_n02_virsh' to the device list (2
> active devices)
> Jan 1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_local_callback:
> Sending full refresh (origin=crmd)
> Jan 1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: master-WebData (10000)
> Jan 1 18:26:45 an-c03n01 attrd[1765]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: probe_complete (true)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:0 (Slave -> Master an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Demote
> WebData:1 (Master -> Slave an-c03n02.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Move
> WebFS (Started an-c03n02.alteeve.ca -> an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebSite (an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 45: /var/lib/pacemaker/pengine/pe-input-183.bz2
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 2: cancel WebData_cancel_60000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 47: stop WebFS_stop_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 67: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca
> (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 69: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=111, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: run_graph: Transition 45
> (Complete=5, Pending=0, Fired=0, Skipped=15, Incomplete=10,
> Source=/var/lib/pacemaker/pengine/pe-input-183.bz2): Stopped
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: unpack_config: On loss
> of CCM Quorum: Ignore
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Promote
> WebData:0 (Slave -> Master an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Demote
> WebData:1 (Master -> Slave an-c03n02.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebFS (an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: LogActions: Start
> WebSite (an-c03n01.alteeve.ca)
> Jan 1 18:26:46 an-c03n01 pengine[1766]: notice: process_pe_message:
> Calculated Transition 46: /var/lib/pacemaker/pengine/pe-input-184.bz2
> Jan 1 18:26:46 an-c03n01 crmd[1767]: warning: destroy_action:
> Cancelling timer for action 2 (src=476)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 64: notify WebData_pre_notify_demote_0 on an-c03n01.alteeve.ca
> (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 66: notify WebData_pre_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=112, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 18: demote WebData_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 kernel: [ 3524.517000] block drbd0: peer(
> Primary -> Secondary )
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 65: notify WebData_post_notify_demote_0 on an-c03n01.alteeve.ca
> (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 67: notify WebData_post_notify_demote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=113, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 60: notify WebData_pre_notify_promote_0 on an-c03n01.alteeve.ca
> (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 62: notify WebData_pre_notify_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=114, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 16: promote WebData_promote_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 kernel: [ 3524.605607] block drbd0: role(
> Secondary -> Primary )
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_promote_0 (call=115, rc=0, cib-update=324,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 61: notify WebData_post_notify_promote_0 on an-c03n01.alteeve.ca
> (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 63: notify WebData_post_notify_promote_0 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=116, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 45: start WebFS_start_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 20: monitor WebData_monitor_60000 on an-c03n02.alteeve.ca
> Jan 1 18:26:46 an-c03n01 Filesystem(WebFS)[17211]: INFO: Running start
> for /dev/drbd0 on /var/www/html
> Jan 1 18:26:46 an-c03n01 kernel: [ 3524.688633] EXT4-fs (drbd0):
> mounted filesystem with ordered data mode. Opts: (null)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebFS_start_0 (call=117, rc=0, cib-update=325, confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 46: monitor WebFS_monitor_60000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 47: start WebSite_start_0 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 apache(WebSite)[17276]: INFO: apache already
> running (pid 16404)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebSite_start_0 (call=119, rc=0, cib-update=326,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebFS_monitor_60000 (call=118, rc=0, cib-update=327,
> confirmed=false) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: te_rsc_command: Initiating
> action 48: monitor WebSite_monitor_60000 on an-c03n01.alteeve.ca (local)
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: process_lrm_event: LRM
> operation WebSite_monitor_60000 (call=120, rc=0, cib-update=328,
> confirmed=false) ok
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: run_graph: Transition 46
> (Complete=27, Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-184.bz2): Complete
> Jan 1 18:26:46 an-c03n01 crmd[1767]: notice: do_state_transition: State
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
> cause=C_FSA_INTERNAL origin=notify_crmd ]
>
> ==[an-c03n02 logs]===
> Jan 1 18:26:44 an-c03n02 crmd[16388]: notice: do_state_transition:
> State transition S_NOT_DC -> S_PENDING [ input=I_PENDING
> cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> Jan 1 18:26:44 an-c03n02 stonith-ng[16384]: notice: unpack_config: On
> loss of CCM Quorum: Ignore
> Jan 1 18:26:44 an-c03n02 stonith-ng[16384]: notice:
> stonith_device_register: Device 'fence_n01_virsh' already existed in
> device list (2 active devices)
> Jan 1 18:26:44 an-c03n02 stonith-ng[16384]: notice:
> stonith_device_register: Device 'fence_n02_virsh' already existed in
> device list (2 active devices)
> Jan 1 18:26:45 an-c03n02 crmd[16388]: notice: do_state_transition:
> State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC
> cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
> Jan 1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_local_callback:
> Sending full refresh (origin=crmd)
> Jan 1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: master-WebData (10000)
> Jan 1 18:26:45 an-c03n02 attrd[16386]: notice: attrd_trigger_update:
> Sending flush op to all hosts for: probe_complete (true)
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=112, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: Running stop
> for /dev/drbd0 on /var/www/html
> Jan 1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: Trying to
> unmount /var/www/html
> Jan 1 18:26:46 an-c03n02 Filesystem(WebFS)[4152]: INFO: unmounted
> /var/www/html successfully
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebFS_stop_0 (call=111, rc=0, cib-update=73, confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=113, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 kernel: [20086.520840] block drbd0: role(
> Primary -> Secondary )
> Jan 1 18:26:46 an-c03n02 kernel: [20086.520911] block drbd0: bitmap
> WRITE of 0 pages took 0 jiffies
> Jan 1 18:26:46 an-c03n02 kernel: [20086.520914] block drbd0: 0 KB (0
> bits) marked out-of-sync by on disk bit-map.
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_demote_0 (call=114, rc=0, cib-update=74,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=115, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=116, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 kernel: [20086.610152] block drbd0: peer(
> Secondary -> Primary )
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_notify_0 (call=117, rc=0, cib-update=0,
> confirmed=true) ok
> Jan 1 18:26:46 an-c03n02 crmd[16388]: notice: process_lrm_event: LRM
> operation WebData_monitor_60000 (call=118, rc=0, cib-update=75,
> confirmed=false) ok
> =====
>
> Now WebSite is working:
> =====
> [root at an-c03n01 ~]# pcs status
> Cluster name: an-cluster-03
> Last updated: Wed Jan 1 18:28:04 2014
> Last change: Wed Jan 1 18:26:44 2014 via cibadmin on an-c03n01.alteeve.ca
> Stack: corosync
> Current DC: an-c03n01.alteeve.ca (1) - partition with quorum
> Version: 1.1.10-19.el7-368c726
> 2 Nodes configured
> 7 Resources configured
>
>
> Online: [ an-c03n01.alteeve.ca an-c03n02.alteeve.ca ]
>
> Full list of resources:
>
> fence_n01_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> fence_n02_virsh (stonith:fence_virsh): Started an-c03n02.alteeve.ca
> ClusterIP (ocf::heartbeat:IPaddr2): Started an-c03n01.alteeve.ca
> Master/Slave Set: WebDataClone [WebData]
> Masters: [ an-c03n01.alteeve.ca ]
> Slaves: [ an-c03n02.alteeve.ca ]
> WebFS (ocf::heartbeat:Filesystem): Started an-c03n01.alteeve.ca
> WebSite (ocf::heartbeat:apache): Started an-c03n01.alteeve.ca
>
> PCSD Status:
> an-c03n01.alteeve.ca:
> an-c03n01.alteeve.ca: Online
> an-c03n02.alteeve.ca:
> an-c03n02.alteeve.ca: Online
>
> Daemon Status:
> corosync: active/disabled
> pacemaker: active/disabled
> pcsd: active/enabled
> =====
>
> Is this a bug?
>
--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?
More information about the Pacemaker
mailing list