[ClusterLabs] GlusterFS all apache instances stopped after failover
lukas
lukas.kostyan at gmail.com
Thu May 7 07:23:50 EDT 2015
Hi,
have a config with 2 nodes where glusterfs is being mounted to root dir
of apache. After producing a kernel panic by echo c >
/proc/sysrq-trigger on node2, node 2 is turned off by stonith. ping
should work further on node1 but it stops since apache is stopped on
node1 as well. So why is apache stopped on node1, has someone any ideas.
Below the logs
following setup: pacemaker 1.1.7, corosync 1.4
Online: [ vm-1 vm-2 ]
Clone Set: cl_gluster_mnt [p_gluster_mnt]
Started: [ vm-1 vm-2 ]
Clone Set: cl_apache [p_apache]
Started: [ vm-1 vm-2 ]
Clone Set: cl_IP [IP] (unique)
IP:0 (ocf::heartbeat:IPaddr2): Started vm-1
IP:1 (ocf::heartbeat:IPaddr2): Started vm-2
p_fence_N1 (stonith:external/libvirt): Started vm-2
p_fence_N2 (stonith:external/libvirt): Started vm-1
root at vm-1:~# crm configure show
node vm-1 \
attributes standby="off"
node vm-2 \
attributes standby="off"
primitive IP ocf:heartbeat:IPaddr2 \
params ip="192.168.122.200" nic="eth0"
clusterip_hash="sourceip-sourceport" \
op monitor interval="10s"
primitive p_apache ocf:heartbeat:apache \
params configfile="/etc/apache2/apache2.conf"
statusurl="http://localhost/server-status" \
op monitor interval="60" timeout="20" \
op start interval="0" timeout="40s" start-delay="0" \
meta is-managed="true"
primitive p_fence_N1 stonith:external/libvirt \
params hostlist="vm-1:N1"
hypervisor_uri="qemu+tcp://192.168.122.1/system" pcmk_reboot_action="off" \
op monitor interval="60" \
meta target-role="Started"
primitive p_fence_N2 stonith:external/libvirt \
params hostlist="vm-2:N2"
hypervisor_uri="qemu+tcp://192.168.122.1/system" pcmk_reboot_action="off" \
op monitor interval="60"
primitive p_gluster_mnt ocf:heartbeat:Filesystem \
params device="localhost:/gvolrep" directory="/var/www/html"
fstype="glusterfs" \
op monitor interval="10"
clone cl_IP IP \
meta globally-unique="true" clone-max="2" clone-node-max="2" \
params resource-stickiness="0"
clone cl_apache p_apache \
meta target-role="Started"
clone cl_gluster_mnt p_gluster_mnt \
meta target-role="Started"
location l_fence_N1 p_fence_N1 -inf: vm-1
location l_fence_N2 p_fence_N2 -inf: vm-2
colocation c_apache_gluster inf: cl_IP cl_gluster_mnt
colocation c_ip_apache inf: cl_apache cl_IP
order o_apache inf: cl_gluster_mnt cl_IP cl_apache
property $id="cib-bootstrap-options" \
dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \
cluster-infrastructure="openais" \
expected-quorum-votes="2" \
stonith-enabled="true" \
no-quorum-policy="ignore" \
last-lrm-refresh="1430996556"
rsc_defaults $id="rsc-options" \
resource-stickiness="100"
op_defaults $id="op-options" \
timeout="240s"
############################
root at vm-1:~# tail -f /var/log/syslog
May 7 13:19:58 vm-1 crmd: [3039]: info: te_rsc_command: Initiating
action 27: monitor p_apache:1_monitor_60000 on vm-2
May 7 13:19:58 vm-1 lrmd: [3036]: info: operation monitor[54] on
p_apache:0 for client 3039: pid 27310 exited with return code 0
May 7 13:19:58 vm-1 crmd: [3039]: info: process_lrm_event: LRM
operation p_apache:0_monitor_60000 (call=54, rc=0, cib-update=322,
confirmed=false) ok
May 7 13:19:58 vm-1 crmd: [3039]: notice: run_graph: ==== Transition 30
(Complete=34, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-1906.bz2): Complete
May 7 13:19:58 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
May 7 13:20:12 vm-1 stonith-ng: [3035]: info: stonith_command:
Processed st_execute from lrmd: rc=-1
May 7 13:20:12 vm-1 external/libvirt[27429]: [27444]: notice:
qemu+tcp://192.168.122.1/system: Running hypervisor: QEMU 2.0.0
May 7 13:20:13 vm-1 stonith: [27422]: info: external/libvirt device OK.
May 7 13:20:13 vm-1 stonith-ng: [3035]: info: log_operation:
p_fence_N2: Performing: stonith -t external/libvirt -S
May 7 13:20:13 vm-1 stonith-ng: [3035]: info: log_operation:
p_fence_N2: success: 0
May 7 13:21:08 vm-1 corosync[2971]: [TOTEM ] A processor failed,
forming new configuration.
May 7 13:21:12 vm-1 corosync[2971]: [pcmk ] notice:
pcmk_peer_update: Transitional membership event on ring 7024: memb=1,
new=0, lost=1
May 7 13:21:12 vm-1 corosync[2971]: [pcmk ] info: pcmk_peer_update:
memb: vm-1 2138745024
May 7 13:21:12 vm-1 corosync[2971]: [pcmk ] info: pcmk_peer_update:
lost: vm-2 930785472
May 7 13:21:12 vm-1 corosync[2971]: [pcmk ] notice:
pcmk_peer_update: Stable membership event on ring 7024: memb=1, new=0,
lost=0
May 7 13:21:12 vm-1 corosync[2971]: [pcmk ] info: pcmk_peer_update:
MEMB: vm-1 2138745024
May 7 13:21:12 vm-1 corosync[2971]: [pcmk ] info:
ais_mark_unseen_peer_dead: Node vm-2 was not seen in the previous transition
May 7 13:21:12 vm-1 corosync[2971]: [pcmk ] info: update_member:
Node 930785472/vm-2 is now: lost
May 7 13:21:12 vm-1 corosync[2971]: [pcmk ] info:
send_member_notification: Sending membership update 7024 to 2 children
May 7 13:21:12 vm-1 cib: [3034]: notice: ais_dispatch_message:
Membership 7024: quorum lost
May 7 13:21:12 vm-1 cib: [3034]: info: crm_update_peer: Node vm-2:
id=930785472 state=lost (new) addr=r(0) ip(192.168.122.183) votes=1
born=7020 seen=7020 proc=00000000000000000000000000111312
May 7 13:21:12 vm-1 crmd: [3039]: notice: ais_dispatch_message:
Membership 7024: quorum lost
May 7 13:21:12 vm-1 crmd: [3039]: info: ais_status_callback: status:
vm-2 is now lost (was member)
May 7 13:21:12 vm-1 corosync[2971]: [TOTEM ] A processor joined or
left the membership and a new membership was formed.
May 7 13:21:12 vm-1 crmd: [3039]: info: crm_update_peer: Node vm-2:
id=930785472 state=lost (new) addr=r(0) ip(192.168.122.183) votes=1
born=7020 seen=7020 proc=00000000000000000000000000111312
May 7 13:21:12 vm-1 corosync[2971]: [CPG ] chosen downlist: sender
r(0) ip(192.168.122.127) ; members(old:2 left:1)
May 7 13:21:12 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/323,
version=0.502.55): ok (rc=0)
May 7 13:21:12 vm-1 corosync[2971]: [MAIN ] Completed service
synchronization, ready to provide service.
May 7 13:21:12 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/325,
version=0.502.57): ok (rc=0)
May 7 13:21:12 vm-1 crmd: [3039]: info: crmd_ais_dispatch: Setting
expected votes to 2
May 7 13:21:12 vm-1 crmd: [3039]: WARN: match_down_event: No match for
shutdown action on vm-2
May 7 13:21:12 vm-1 crmd: [3039]: info: te_update_diff:
Stonith/shutdown of vm-2 not matched
May 7 13:21:12 vm-1 crmd: [3039]: info: abort_transition_graph:
te_update_diff:234 - Triggered transition abort (complete=1,
tag=node_state, id=vm-2, magic=NA, cib=0.502.56) : Node failure
May 7 13:21:12 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
May 7 13:21:12 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/327,
version=0.502.58): ok (rc=0)
May 7 13:21:12 vm-1 pengine: [3038]: notice: unpack_config: On loss of
CCM Quorum: Ignore
May 7 13:21:12 vm-1 pengine: [3038]: WARN: pe_fence_node: Node vm-2
will be fenced because it is un-expectedly down
May 7 13:21:12 vm-1 pengine: [3038]: WARN: determine_online_status:
Node vm-2 is unclean
May 7 13:21:12 vm-1 pengine: [3038]: WARN: custom_action: Action
p_gluster_mnt:1_stop_0 on vm-2 is unrunnable (offline)
May 7 13:21:12 vm-1 pengine: [3038]: WARN: custom_action: Marking node
vm-2 unclean
May 7 13:21:12 vm-1 pengine: [3038]: WARN: custom_action: Action
p_apache:1_stop_0 on vm-2 is unrunnable (offline)
May 7 13:21:12 vm-1 pengine: [3038]: WARN: custom_action: Marking node
vm-2 unclean
May 7 13:21:12 vm-1 pengine: [3038]: WARN: custom_action: Action
IP:1_stop_0 on vm-2 is unrunnable (offline)
May 7 13:21:12 vm-1 pengine: [3038]: WARN: custom_action: Marking node
vm-2 unclean
May 7 13:21:12 vm-1 pengine: [3038]: WARN: custom_action: Action
p_fence_N1_stop_0 on vm-2 is unrunnable (offline)
May 7 13:21:12 vm-1 pengine: [3038]: WARN: custom_action: Marking node
vm-2 unclean
May 7 13:21:12 vm-1 pengine: [3038]: WARN: stage6: Scheduling Node vm-2
for STONITH
May 7 13:21:12 vm-1 pengine: [3038]: notice: LogActions: Stop
p_gluster_mnt:1#011(vm-2)
May 7 13:21:12 vm-1 pengine: [3038]: notice: LogActions: Restart
p_apache:0#011(Started vm-1)
May 7 13:21:12 vm-1 pengine: [3038]: notice: LogActions: Stop
p_apache:1#011(vm-2)
May 7 13:21:12 vm-1 pengine: [3038]: notice: LogActions: Move
IP:1#011(Started vm-2 -> vm-1)
May 7 13:21:12 vm-1 pengine: [3038]: notice: LogActions: Stop
p_fence_N1#011(vm-2)
May 7 13:21:12 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
May 7 13:21:12 vm-1 crmd: [3039]: info: do_te_invoke: Processing graph
31 (ref=pe_calc-dc-1430997672-228) derived from
/var/lib/pengine/pe-warn-661.bz2
May 7 13:21:12 vm-1 crmd: [3039]: notice: te_fence_node: Executing
reboot fencing operation (36) on vm-2 (timeout=60000)
May 7 13:21:12 vm-1 stonith-ng: [3035]: info:
initiate_remote_stonith_op: Initiating remote operation reboot for vm-2:
9a25d1de-b42d-44b1-9d49-368f36fa57fd
May 7 13:21:12 vm-1 pengine: [3038]: WARN: process_pe_message:
Transition 31: WARNINGs found during PE processing. PEngine Input stored
in: /var/lib/pengine/pe-warn-661.bz2
May 7 13:21:12 vm-1 pengine: [3038]: notice: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
May 7 13:21:12 vm-1 stonith-ng: [3035]: info:
can_fence_host_with_device: Refreshing port list for p_fence_N2
May 7 13:21:12 vm-1 stonith-ng: [3035]: info:
can_fence_host_with_device: p_fence_N2 can fence vm-2: dynamic-list
May 7 13:21:12 vm-1 stonith-ng: [3035]: info: call_remote_stonith:
Requesting that vm-1 perform op reboot vm-2
May 7 13:21:12 vm-1 stonith-ng: [3035]: info:
can_fence_host_with_device: p_fence_N2 can fence vm-2: dynamic-list
May 7 13:21:12 vm-1 stonith-ng: [3035]: info: stonith_fence: Found 1
matching devices for 'vm-2'
May 7 13:21:12 vm-1 stonith-ng: [3035]: info: stonith_command:
Processed st_fence from vm-1: rc=-1
May 7 13:21:12 vm-1 stonith-ng: [3035]: info: make_args: Substituting
action 'off' for requested operation 'reboot'
May 7 13:21:12 vm-1 external/libvirt[27944]: [27957]: notice: Domain N2
was stopped
May 7 13:21:13 vm-1 stonith-ng: [3035]: notice: log_operation:
Operation 'reboot' [27936] (call 0 from
c5982dd8-7639-49be-a9f9-c8488bea6091) for host 'vm-2' with device
'p_fence_N2' returned: 0
May 7 13:21:13 vm-1 stonith-ng: [3035]: info: log_operation:
p_fence_N2: Performing: stonith -t external/libvirt -T off vm-2
May 7 13:21:13 vm-1 stonith-ng: [3035]: info: log_operation:
p_fence_N2: success: vm-2 0
May 7 13:21:13 vm-1 stonith-ng: [3035]: notice: remote_op_done:
Operation reboot of vm-2 by vm-1 for
vm-1[c5982dd8-7639-49be-a9f9-c8488bea6091]: OK
May 7 13:21:13 vm-1 crmd: [3039]: info: tengine_stonith_callback:
StonithOp <st-reply st_origin="stonith_construct_async_reply"
t="stonith-ng" st_op="reboot"
st_remote_op="9a25d1de-b42d-44b1-9d49-368f36fa57fd"
st_clientid="c5982dd8-7639-49be-a9f9-c8488bea6091" st_target="vm-2"
st_device_action="st_fence" st_callid="0" st_callopt="0" st_rc="0"
st_output="Performing: stonith -t external/libvirt -T off
vm-2#012success: vm-2 0#012" src="vm-1" seq="16" state="2" />
May 7 13:21:13 vm-1 crmd: [3039]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='vm-2']/lrm
May 7 13:21:13 vm-1 crmd: [3039]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='vm-2']/transient_attributes
May 7 13:21:13 vm-1 crmd: [3039]: notice: crmd_peer_update: Status
update: Client vm-2/crmd now has status [offline] (DC=true)
May 7 13:21:13 vm-1 crmd: [3039]: notice: tengine_stonith_notify: Peer
vm-2 was terminated (reboot) by vm-1 for vm-1: OK
(ref=9a25d1de-b42d-44b1-9d49-368f36fa57fd)
May 7 13:21:13 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_FSA_INTERNAL origin=check_join_state ]
May 7 13:21:13 vm-1 crmd: [3039]: info: abort_transition_graph:
do_te_invoke:169 - Triggered transition abort (complete=0) : Peer Halt
May 7 13:21:13 vm-1 crmd: [3039]: notice: run_graph: ==== Transition 31
(Complete=3, Pending=0, Fired=0, Skipped=15, Incomplete=5,
Source=/var/lib/pengine/pe-warn-661.bz2): Stopped
May 7 13:21:13 vm-1 crmd: [3039]: info: abort_transition_graph:
do_te_invoke:169 - Triggered transition abort (complete=1) : Peer Halt
May 7 13:21:13 vm-1 crmd: [3039]: info: join_make_offer: Making join
offers based on membership 7024
May 7 13:21:13 vm-1 crmd: [3039]: info: do_dc_join_offer_all: join-11:
Waiting on 1 outstanding join acks
May 7 13:21:13 vm-1 crmd: [3039]: info: update_dc: Set DC to vm-1 (3.0.6)
May 7 13:21:13 vm-1 crmd: [3039]: info: cib_fencing_updated: Fencing
update 329 for vm-2: complete
May 7 13:21:13 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='vm-2']/lrm
(origin=local/crmd/330, version=0.502.60): ok (rc=0)
May 7 13:21:13 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_delete for section
//node_state[@uname='vm-2']/transient_attributes (origin=local/crmd/331,
version=0.502.61): ok (rc=0)
May 7 13:21:13 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
May 7 13:21:13 vm-1 crmd: [3039]: info: do_dc_join_finalize: join-11:
Syncing the CIB from vm-1 to the rest of the cluster
May 7 13:21:13 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/334,
version=0.502.62): ok (rc=0)
May 7 13:21:13 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/335,
version=0.502.63): ok (rc=0)
May 7 13:21:13 vm-1 crmd: [3039]: info: do_dc_join_ack: join-11:
Updating node state to member for vm-1
May 7 13:21:13 vm-1 crmd: [3039]: info: erase_status_tag: Deleting
xpath: //node_state[@uname='vm-1']/lrm
May 7 13:21:13 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='vm-1']/lrm
(origin=local/crmd/336, version=0.502.64): ok (rc=0)
May 7 13:21:13 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
May 7 13:21:13 vm-1 crmd: [3039]: info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
May 7 13:21:13 vm-1 attrd: [3037]: notice: attrd_local_callback:
Sending full refresh (origin=crmd)
May 7 13:21:13 vm-1 attrd: [3037]: notice: attrd_trigger_update:
Sending flush op to all hosts for: last-failure-p_apache:0 (1430996422)
May 7 13:21:13 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/338,
version=0.502.66): ok (rc=0)
May 7 13:21:13 vm-1 cib: [3034]: info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/340,
version=0.502.68): ok (rc=0)
May 7 13:21:13 vm-1 attrd: [3037]: notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
May 7 13:21:13 vm-1 pengine: [3038]: notice: unpack_config: On loss of
CCM Quorum: Ignore
May 7 13:21:13 vm-1 pengine: [3038]: notice: LogActions: Restart
p_apache:0#011(Started vm-1)
May 7 13:21:13 vm-1 pengine: [3038]: notice: LogActions: Start
IP:1#011(vm-1)
May 7 13:21:13 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
May 7 13:21:13 vm-1 crmd: [3039]: info: do_te_invoke: Processing graph
32 (ref=pe_calc-dc-1430997673-233) derived from
/var/lib/pengine/pe-input-1907.bz2
May 7 13:21:13 vm-1 crmd: [3039]: info: te_rsc_command: Initiating
action 14: stop p_apache:0_stop_0 on vm-1 (local)
May 7 13:21:13 vm-1 lrmd: [3036]: info: cancel_op: operation
monitor[54] on p_apache:0 for client 3039, its parameters:
CRM_meta_start_delay=[0] CRM_meta_timeout=[20000]
CRM_meta_name=[monitor] crm_feature_set=[3.0.6]
CRM_meta_clone_node_max=[1] configfile=[/etc/apache2/apache2.conf]
CRM_meta_clone=[0] CRM_meta_interval=[60000] CRM_meta_clone_max=[2]
CRM_meta_notify=[false] statusurl=[http://localhost/server-status]
CRM_meta_globally_unique=[false] cancelled
May 7 13:21:13 vm-1 lrmd: [3036]: info: rsc:p_apache:0 stop[55] (pid 27992)
May 7 13:21:13 vm-1 crmd: [3039]: info: te_rsc_command: Initiating
action 22: start IP:1_start_0 on vm-1 (local)
May 7 13:21:13 vm-1 lrmd: [3036]: info: rsc:IP:1 start[56] (pid 27993)
May 7 13:21:13 vm-1 crmd: [3039]: info: process_lrm_event: LRM
operation p_apache:0_monitor_60000 (call=54, status=1, cib-update=0,
confirmed=true) Cancelled
May 7 13:21:13 vm-1 pengine: [3038]: notice: process_pe_message:
Transition 32: PEngine Input stored in: /var/lib/pengine/pe-input-1907.bz2
May 7 13:21:13 vm-1 stonith-ng: [3035]: info: stonith_command:
Processed st_execute from lrmd: rc=-1
May 7 13:21:13 vm-1 external/libvirt[28069]: [28084]: notice:
qemu+tcp://192.168.122.1/system: Running hypervisor: QEMU 2.0.0
May 7 13:21:14 vm-1 IPaddr2[27993]: INFO: /usr/lib/heartbeat/send_arp
-i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.122.200 eth0
192.168.122.200 8bbd5f0b558f not_used not_used
May 7 13:21:14 vm-1 lrmd: [3036]: info: operation start[56] on IP:1 for
client 3039: pid 27993 exited with return code 0
May 7 13:21:14 vm-1 crmd: [3039]: info: process_lrm_event: LRM
operation IP:1_start_0 (call=56, rc=0, cib-update=342, confirmed=true) ok
May 7 13:21:14 vm-1 crmd: [3039]: info: te_rsc_command: Initiating
action 23: monitor IP:1_monitor_10000 on vm-1 (local)
May 7 13:21:14 vm-1 lrmd: [3036]: info: rsc:IP:1 monitor[57] (pid 28104)
May 7 13:21:14 vm-1 lrmd: [3036]: info: operation monitor[57] on IP:1
for client 3039: pid 28104 exited with return code 0
May 7 13:21:14 vm-1 crmd: [3039]: info: process_lrm_event: LRM
operation IP:1_monitor_10000 (call=57, rc=0, cib-update=343,
confirmed=false) ok
May 7 13:21:14 vm-1 lrmd: [3036]: info: RA output:
(p_apache:0:stop:stderr) /usr/lib/ocf/resource.d//heartbeat/apache: 440:
kill:
May 7 13:21:14 vm-1 lrmd: [3036]: info: RA output:
(p_apache:0:stop:stderr) No such process
May 7 13:21:14 vm-1 lrmd: [3036]: info: RA output:
(p_apache:0:stop:stderr)
May 7 13:21:14 vm-1 apache[27992]: INFO: Killing apache PID 27235
May 7 13:21:14 vm-1 apache[27992]: INFO: apache stopped.
May 7 13:21:14 vm-1 lrmd: [3036]: info: operation stop[55] on
p_apache:0 for client 3039: pid 27992 exited with return code 0
May 7 13:21:14 vm-1 crmd: [3039]: info: process_lrm_event: LRM
operation p_apache:0_stop_0 (call=55, rc=0, cib-update=344,
confirmed=true) ok
May 7 13:21:14 vm-1 crmd: [3039]: info: te_rsc_command: Initiating
action 15: start p_apache:0_start_0 on vm-1 (local)
May 7 13:21:14 vm-1 lrmd: [3036]: info: rsc:p_apache:0 start[58] (pid
28166)
May 7 13:21:14 vm-1 stonith: [28062]: info: external/libvirt device OK.
May 7 13:21:14 vm-1 stonith-ng: [3035]: info: log_operation:
p_fence_N2: Performing: stonith -t external/libvirt -S
May 7 13:21:14 vm-1 stonith-ng: [3035]: info: log_operation:
p_fence_N2: success: 0
May 7 13:21:54 vm-1 lrmd: [3036]: WARN: p_apache:0:start process (PID
28166) timed out (try 1). Killing with signal SIGTERM (15).
May 7 13:21:54 vm-1 lrmd: [3036]: WARN: operation start[58] on
p_apache:0 for client 3039: pid 28166 timed out
May 7 13:21:54 vm-1 crmd: [3039]: ERROR: process_lrm_event: LRM
operation p_apache:0_start_0 (58) Timed Out (timeout=40000ms)
May 7 13:21:54 vm-1 crmd: [3039]: WARN: status_from_rc: Action 15
(p_apache:0_start_0) on vm-1 failed (target: 0 vs. rc: -2): Error
May 7 13:21:54 vm-1 crmd: [3039]: WARN: update_failcount: Updating
failcount for p_apache:0 on vm-1 after failed start: rc=-2
(update=INFINITY, time=1430997714)
May 7 13:21:54 vm-1 crmd: [3039]: info: abort_transition_graph:
match_graph_event:277 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=p_apache:0_last_failure_0,
magic=2:-2;15:32:0:da4997d2-70ea-45a7-9617-42963ea29e42, cib=0.502.74) :
Event failed
May 7 13:21:54 vm-1 crmd: [3039]: notice: run_graph: ==== Transition 32
(Complete=11, Pending=0, Fired=0, Skipped=1, Incomplete=0,
Source=/var/lib/pengine/pe-input-1907.bz2): Stopped
May 7 13:21:54 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
May 7 13:21:54 vm-1 attrd: [3037]: notice: attrd_trigger_update:
Sending flush op to all hosts for: fail-count-p_apache:0 (INFINITY)
May 7 13:21:54 vm-1 attrd: [3037]: notice: attrd_perform_update: Sent
update 94: fail-count-p_apache:0=INFINITY
May 7 13:21:54 vm-1 pengine: [3038]: notice: unpack_config: On loss of
CCM Quorum: Ignore
May 7 13:21:54 vm-1 pengine: [3038]: WARN: unpack_rsc_op: Processing
failed op p_apache:0_last_failure_0 on vm-1: unknown exec error (-2)
May 7 13:21:54 vm-1 attrd: [3037]: notice: attrd_trigger_update:
Sending flush op to all hosts for: last-failure-p_apache:0 (1430997714)
May 7 13:21:54 vm-1 pengine: [3038]: notice: LogActions: Recover
p_apache:0#011(Started vm-1)
May 7 13:21:54 vm-1 attrd: [3037]: notice: attrd_perform_update: Sent
update 96: last-failure-p_apache:0=1430997714
May 7 13:21:54 vm-1 crmd: [3039]: info: abort_transition_graph:
te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair,
id=status-vm-1-fail-count-p_apache.0, name=fail-count-p_apache:0,
value=INFINITY, magic=NA, cib=0.502.75) : Transient attribute: update
May 7 13:21:54 vm-1 crmd: [3039]: info: handle_response: pe_calc
calculation pe_calc-dc-1430997714-238 is obsolete
May 7 13:21:54 vm-1 crmd: [3039]: info: abort_transition_graph:
te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair,
id=status-vm-1-last-failure-p_apache.0, name=last-failure-p_apache:0,
value=1430997714, magic=NA, cib=0.502.76) : Transient attribute: update
May 7 13:21:54 vm-1 pengine: [3038]: notice: process_pe_message:
Transition 33: PEngine Input stored in: /var/lib/pengine/pe-input-1908.bz2
May 7 13:21:54 vm-1 pengine: [3038]: notice: unpack_config: On loss of
CCM Quorum: Ignore
May 7 13:21:54 vm-1 pengine: [3038]: WARN: unpack_rsc_op: Processing
failed op p_apache:0_last_failure_0 on vm-1: unknown exec error (-2)
May 7 13:21:54 vm-1 pengine: [3038]: WARN: common_apply_stickiness:
Forcing cl_apache away from vm-1 after 1000000 failures (max=1000000)
May 7 13:21:54 vm-1 pengine: [3038]: WARN: common_apply_stickiness:
Forcing cl_apache away from vm-1 after 1000000 failures (max=1000000)
May 7 13:21:54 vm-1 pengine: [3038]: notice: LogActions: Stop
p_apache:0#011(vm-1)
May 7 13:21:54 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
May 7 13:21:54 vm-1 crmd: [3039]: info: do_te_invoke: Processing graph
34 (ref=pe_calc-dc-1430997714-239) derived from
/var/lib/pengine/pe-input-1909.bz2
May 7 13:21:54 vm-1 crmd: [3039]: info: te_rsc_command: Initiating
action 4: stop p_apache:0_stop_0 on vm-1 (local)
May 7 13:21:54 vm-1 lrmd: [3036]: info: rsc:p_apache:0 stop[59] (pid 28650)
May 7 13:21:54 vm-1 pengine: [3038]: notice: process_pe_message:
Transition 34: PEngine Input stored in: /var/lib/pengine/pe-input-1909.bz2
May 7 13:21:54 vm-1 apache[28650]: INFO: apache is not running.
May 7 13:21:54 vm-1 apache[28650]: INFO: apache children were signalled
(SIGTERM)
May 7 13:21:56 vm-1 apache[28650]: INFO: apache children were signalled
(SIGHUP)
May 7 13:21:57 vm-1 apache[28650]: INFO: apache children were signalled
(SIGKILL)
May 7 13:21:58 vm-1 lrmd: [3036]: info: operation stop[59] on
p_apache:0 for client 3039: pid 28650 exited with return code 0
May 7 13:21:58 vm-1 crmd: [3039]: info: process_lrm_event: LRM
operation p_apache:0_stop_0 (call=59, rc=0, cib-update=349,
confirmed=true) ok
May 7 13:21:58 vm-1 crmd: [3039]: notice: run_graph: ==== Transition 34
(Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-1909.bz2): Complete
May 7 13:21:58 vm-1 crmd: [3039]: notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
May 7 13:22:14 vm-1 stonith-ng: [3035]: info: stonith_command:
Processed st_execute from lrmd: rc=-1
May 7 13:22:15 vm-1 external/libvirt[28943]: [28958]: notice:
qemu+tcp://192.168.122.1/system: Running hypervisor: QEMU 2.0.0
May 7 13:22:16 vm-1 stonith: [28935]: info: external/libvirt device OK.
May 7 13:22:16 vm-1 stonith-ng: [3035]: info: log_operation:
p_fence_N2: Performing: stonith -t external/libvirt -S
May 7 13:22:16 vm-1 stonith-ng: [3035]: info: log_operation:
p_fence_N2: success: 0
Online: [ vm-1 ]
OFFLINE: [ vm-2 ]
Clone Set: cl_gluster_mnt [p_gluster_mnt]
Started: [ vm-1 ]
Stopped: [ p_gluster_mnt:1 ]
Clone Set: cl_IP [IP] (unique)
IP:0 (ocf::heartbeat:IPaddr2): Started vm-1
IP:1 (ocf::heartbeat:IPaddr2): Started vm-1
p_fence_N2 (stonith:external/libvirt): Started vm-1
Failed actions:
p_apache:0_start_0 (node=vm-1, call=58, rc=-2, status=Timed Out):
unknown exec error
More information about the Users
mailing list