Hi Pacemaker mailing list,<br>I'm following the "Clusters from Scratch" guide except I'm using RHEL 5.3 instead of Fedora. Though I doubt that is relevant for this problem, these RHEL 5.3 "machines" are really xen instances. I've looked through the mail archives and have not been able to find anyone with the exact same problem. <br>
<br>I have 2 nodes, pace01 and pace02. On both nodes I can start and stop apache with init scripts and bring up the pages in a browser.<br><br>After adding this configure statement:<br><br>crm configure primitive WebSite ocf:heartbeat:apache \<br>
params configfile=/etc/httpd/conf/httpd.conf \<br>op monitor interval=60<br><br>This is where I am:<br><br>[root@pace01 ~]# crm_mon -fo1<br>============<br>Last updated: Thu Sep 16 10:32:56 2010<br>Stack: openais<br>Current DC: pace01 - partition with quorum<br>
Version: 1.0.9-89bd754939df5150de7cd76835f98fe90851b677<br>2 Nodes configured, 2 expected votes<br>2 Resources configured.<br>============<br><br>Online: [ pace01 pace02 ]<br><br> ClusterIP (ocf::heartbeat:IPaddr2): Started pace01<br>
<br>Operations:<br>* Node pace02: <br> WebSite: migration-threshold=1000000 fail-count=1000000<br> + (4) start: rc=1 (unknown error)<br> + (5) stop: rc=0 (ok)<br>* Node pace01: <br> ClusterIP: migration-threshold=1000000<br>
+ (3) start: rc=0 (ok)<br> + (4) monitor: interval=30000ms rc=0 (ok)<br> WebSite: migration-threshold=1000000 fail-count=1000000<br> + (6) start: rc=1 (unknown error)<br> + (7) stop: rc=0 (ok)<br><br>Failed actions:<br>
WebSite_start_0 (node=pace02, call=4, rc=1, status=complete): unknown error<br> WebSite_start_0 (node=pace01, call=6, rc=1, status=complete): unknown error<br><br><br>----------------------------------------------------------------------------------------------------------------------------<br>
<br>[root@pace01 ~]# crm configure show<br>node pace01<br>node pace02<br>primitive ClusterIP ocf:heartbeat:IPaddr2 \<br> params ip="172.29.84.154" cidr_netmask="32" \<br> op monitor interval="30"<br>
primitive WebSite ocf:heartbeat:apache \<br> params configfile="/etc/httpd/conf/httpd.conf" \<br> op monitor interval="60"<br>property $id="cib-bootstrap-options" \<br> dc-version="1.0.9-89bd754939df5150de7cd76835f98fe90851b677" \<br>
cluster-infrastructure="openais" \<br> expected-quorum-votes="2" \<br> stonith-enabled="false" \<br> no-quorum-policy="ignore"<br>rsc_defaults $id="rsc-options" \<br>
resource-stickiness="100"<br>--------------------------------------------------------------------------------------------------------------------------<br><br>[root@pace01 ~]# grep -i lrmd.*WebSite /var/log/messages <br>
Sep 15 14:00:47 pace01 lrmd: [32149]: info: rsc:WebSite:5: probe<br>Sep 15 14:00:52 pace01 lrmd: [32149]: info: rsc:WebSite:6: start<br>Sep 15 14:00:55 pace01 lrmd: [32149]: info: rsc:WebSite:7: stop<br>Sep 15 14:00:56 pace01 lrmd: [32149]: info: RA output: (WebSite:stop:stderr) /usr/lib/ocf/resource.d//heartbeat/apache: line 437: kill: (32467) - No such p<br>
<br>------------------------------------------------------------------------------------------------------------------------------<br><br>[root@pace01 ~]# crm_verify -LV<br>crm_verify[14253]: 2010/09/16_10:34:17 WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on pace02: unknown error (1)<br>
crm_verify[14253]: 2010/09/16_10:34:17 WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on pace01: unknown error (1)<br>crm_verify[14253]: 2010/09/16_10:34:17 WARN: common_apply_stickiness: Forcing WebSite away from pace01 after 1000000 failures (max=1000000)<br>
crm_verify[14253]: 2010/09/16_10:34:17 WARN: common_apply_stickiness: Forcing WebSite away from pace02 after 1000000 failures (max=1000000)<br><br><br>Sorry if this next bit is unnecessarily verbose, but here is ~15 seconds of /var/log/messages from both nodes when apache is added to the pacemaker config:<br>
<br><br>#################################<br>########PACE02##################<br>#################################<br><br>Sep 15 14:00:43 pace02 cib: [26337]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-88.raw<br>
Sep 15 14:00:43 pace02 cib: [26337]: info: write_cib_contents: Wrote version 0.115.0 of the CIB to disk (digest: fa2d713cb48a259b425133cec51cd077)<br>Sep 15 14:00:43 pace02 cib: [26337]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.VBA3sR (digest: /var/lib/heartbeat/crm/cib.w0QMZR)<br>
Sep 15 14:00:43 pace02 crmd: [26283]: info: update_dc: Unset DC pace01<br>Sep 15 14:00:43 pace02 crmd: [26283]: info: do_state_transition: State transition S_NOT_DC -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_election_count_vote ]<br>
Sep 15 14:00:44 pace02 crmd: [26283]: info: update_dc: Set DC to pace01 (3.0.1)<br>Sep 15 14:00:45 pace02 crmd: [26283]: info: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]<br>
Sep 15 14:00:45 pace02 attrd: [26281]: info: attrd_local_callback: Sending full refresh (origin=crmd)<br>Sep 15 14:00:45 pace02 attrd: [26281]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-WebSite (<null>)<br>
Sep 15 14:00:45 pace02 attrd: [26281]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<br>Sep 15 14:00:45 pace02 attrd: [26281]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)<br>
Sep 15 14:00:45 pace02 attrd: [26281]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-WebSite (<null>)<br>Sep 15 14:00:45 pace02 attrd: [26281]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)<br>
Sep 15 14:00:47 pace02 crmd: [26283]: info: do_lrm_rsc_op: Performing key=7:6:7:b7d1a9ca-8a97-4fd9-88b8-b8d303c3ebbf op=WebSite_monitor_0 )<br>Sep 15 14:00:47 pace02 lrmd: [26280]: info: rsc:WebSite:3: probe<br>Sep 15 14:00:47 pace02 apache[26338]: INFO: apache not running<br>
Sep 15 14:00:47 pace02 crmd: [26283]: info: process_lrm_event: LRM operation WebSite_monitor_0 (call=3, rc=7, cib-update=11, confirmed=true) not running<br>Sep 15 14:00:47 pace02 crmd: [26283]: info: do_lrm_rsc_op: Performing key=10:6:0:b7d1a9ca-8a97-4fd9-88b8-b8d303c3ebbf op=WebSite_start_0 )<br>
Sep 15 14:00:47 pace02 lrmd: [26280]: info: rsc:WebSite:4: start<br>Sep 15 14:00:48 pace02 apache[26373]: INFO: apache not running<br>Sep 15 14:00:48 pace02 apache[26373]: INFO: waiting for apache /etc/httpd/conf/httpd.conf to come up<br>
Sep 15 14:00:49 pace02 crmd: [26283]: info: process_lrm_event: LRM operation WebSite_start_0 (call=4, rc=1, cib-update=12, confirmed=true) unknown error<br>Sep 15 14:00:49 pace02 attrd: [26281]: info: attrd_ais_dispatch: Update relayed from pace01<br>
Sep 15 14:00:49 pace02 attrd: [26281]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-WebSite (INFINITY)<br>Sep 15 14:00:49 pace02 attrd: [26281]: info: attrd_perform_update: Sent update 40: fail-count-WebSite=INFINITY<br>
Sep 15 14:00:49 pace02 attrd: [26281]: info: attrd_ais_dispatch: Update relayed from pace01<br>Sep 15 14:00:49 pace02 attrd: [26281]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-WebSite (1284573649)<br>
Sep 15 14:00:49 pace02 attrd: [26281]: info: attrd_perform_update: Sent update 43: last-failure-WebSite=1284573649<br>Sep 15 14:00:50 pace02 crmd: [26283]: info: do_lrm_rsc_op: Performing key=1:8:0:b7d1a9ca-8a97-4fd9-88b8-b8d303c3ebbf op=WebSite_stop_0 )<br>
Sep 15 14:00:50 pace02 lrmd: [26280]: info: rsc:WebSite:5: stop<br>Sep 15 14:00:52 pace02 lrmd: [26280]: info: RA output: (WebSite:stop:stderr) /usr/lib/ocf/resource.d//heartbeat/apache: line 437: kill: (26406) - No such process <br>
Sep 15 14:00:52 pace02 apache[26438]: INFO: Killing apache PID 26406<br>Sep 15 14:00:52 pace02 apache[26438]: INFO: apache stopped.<br>Sep 15 14:00:52 pace02 crmd: [26283]: info: process_lrm_event: LRM operation WebSite_stop_0 (call=5, rc=0, cib-update=13, confirmed=true) ok<br>
<br>#################################<br>########PACE01##################<br>#################################<br><br>Sep 15 14:00:42 pace01 cibadmin: [32366]: info: Invoked: cibadmin -Ql <br>Sep 15 14:00:42 pace01 cibadmin: [32367]: info: Invoked: cibadmin -Ql <br>
Sep 15 14:00:43 pace01 crm_shadow: [32387]: info: Invoked: crm_shadow -c __crmshell.32345 <br>Sep 15 14:00:43 pace01 cibadmin: [32388]: info: Invoked: cibadmin -p -U <br>Sep 15 14:00:43 pace01 crm_shadow: [32390]: info: Invoked: crm_shadow -C __crmshell.32345 --force <br>
Sep 15 14:00:43 pace01 cib: [32148]: info: cib_replace_notify: Replaced: 0.113.7 -> 0.115.1 from <null><br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: - <cib epoch="113" num_updates="7" admin_epoch="0" /><br>
Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + <cib epoch="115" num_updates="1" admin_epoch="0" ><br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + <configuration ><br>
Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + <resources ><br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + <primitive class="ocf" id="WebSite" provider="heartbeat" type="apache" __crm_diff_marker__="added:top" ><br>
Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + <instance_attributes id="WebSite-instance_attributes" ><br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + <nvpair id="WebSite-instance_attributes-configfile" name="configfile" value="/etc/httpd/conf/httpd.conf" /><br>
Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + </instance_attributes><br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + <operations ><br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + <op id="WebSite-monitor-60" interval="60" name="monitor" /><br>
Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + </operations><br>Sep 15 14:00:43 pace01 crmd: [32152]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change<br>
Sep 15 14:00:43 pace01 attrd: [32150]: info: do_cib_replaced: Sending full refresh<br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + </primitive><br>Sep 15 14:00:43 pace01 crmd: [32152]: info: need_abort: Aborting on change to epoch<br>
Sep 15 14:00:43 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-WebSite (<null>)<br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + </resources><br>
Sep 15 14:00:43 pace01 crmd: [32152]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + </configuration><br>
Sep 15 14:00:43 pace01 crmd: [32152]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.<br>Sep 15 14:00:43 pace01 cib: [32148]: info: log_data_element: cib:diff: + </cib><br>Sep 15 14:00:43 pace01 crmd: [32152]: info: do_pe_invoke: Query 64: Requesting the current CIB: S_POLICY_ENGINE<br>
Sep 15 14:00:43 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_replace for section 'all' (origin=local/crm_shadow/2, version=0.115.1): ok (rc=0)<br>Sep 15 14:00:43 pace01 crmd: [32152]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ]<br>
Sep 15 14:00:43 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/62, version=0.115.1): ok (rc=0)<br>Sep 15 14:00:43 pace01 crm_shadow: [32392]: info: Invoked: crm_shadow -D __crmshell.32345 --force <br>
Sep 15 14:00:43 pace01 crmd: [32152]: info: update_dc: Unset DC pace01<br>Sep 15 14:00:43 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<br>Sep 15 14:00:43 pace01 cib: [32393]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-81.raw<br>
Sep 15 14:00:43 pace01 cibadmin: [32394]: info: Invoked: cibadmin -Ql <br>Sep 15 14:00:43 pace01 crmd: [32152]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]<br>
Sep 15 14:00:43 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)<br>Sep 15 14:00:44 pace01 cib: [32393]: info: write_cib_contents: Wrote version 0.115.0 of the CIB to disk (digest: e7374d53ff60b0a3e814f26c61104159)<br>
Sep 15 14:00:44 pace01 crmd: [32152]: info: do_dc_takeover: Taking over DC status for this partition<br>Sep 15 14:00:44 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-WebSite (<null>)<br>
Sep 15 14:00:44 pace01 cib: [32393]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.on269y (digest: /var/lib/heartbeat/crm/cib.oCdSPA)<br>Sep 15 14:00:44 pace01 cib: [32148]: info: cib_process_readwrite: We are now in R/O mode<br>
Sep 15 14:00:44 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/65, version=0.115.1): ok (rc=0)<br>Sep 15 14:00:44 pace01 cib: [32148]: info: cib_process_readwrite: We are now in R/W mode<br>
Sep 15 14:00:44 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/66, version=0.115.1): ok (rc=0)<br>Sep 15 14:00:44 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/67, version=0.115.1): ok (rc=0)<br>
Sep 15 14:00:44 pace01 cib: [32148]: WARN: G_SIG_dispatch: Dispatch function for SIGCHLD was delayed 110 ms (> 100 ms) before being called (GSource: 0x92b9810)<br>Sep 15 14:00:44 pace01 cib: [32148]: info: G_SIG_dispatch: started at 2020165924 should have started at 2020165913<br>
Sep 15 14:00:44 pace01 cib: [32148]: WARN: G_SIG_dispatch: Dispatch function for SIGCHLD took too long to execute: 70 ms (> 30 ms) (GSource: 0x92b9810)<br>Sep 15 14:00:44 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/69, version=0.115.1): ok (rc=0)<br>
Sep 15 14:00:44 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/71, version=0.115.1): ok (rc=0)<br>Sep 15 14:00:44 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)<br>
Sep 15 14:00:44 pace01 crmd: [32152]: info: do_dc_join_offer_all: join-4: Waiting on 2 outstanding join acks<br>Sep 15 14:00:44 pace01 crmd: [32152]: info: ais_dispatch: Membership 272: quorum retained<br>Sep 15 14:00:44 pace01 crmd: [32152]: info: crm_ais_dispatch: Setting expected votes to 2<br>
Sep 15 14:00:44 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/74, version=0.115.1): ok (rc=0)<br>Sep 15 14:00:44 pace01 crmd: [32152]: info: config_query_callback: Checking for expired actions every 900000ms<br>
Sep 15 14:00:44 pace01 crmd: [32152]: info: config_query_callback: Sending expected-votes=2 to corosync<br>Sep 15 14:00:44 pace01 crmd: [32152]: info: update_dc: Set DC to pace01 (3.0.1)<br>Sep 15 14:00:44 pace01 crmd: [32152]: info: ais_dispatch: Membership 272: quorum retained<br>
Sep 15 14:00:45 pace01 crmd: [32152]: info: crm_ais_dispatch: Setting expected votes to 2<br>Sep 15 14:00:45 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/77, version=0.115.1): ok (rc=0)<br>
Sep 15 14:00:45 pace01 crmd: [32152]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]<br>Sep 15 14:00:45 pace01 crmd: [32152]: info: do_state_transition: All 2 cluster nodes responded to the join offer.<br>
Sep 15 14:00:45 pace01 crmd: [32152]: info: do_dc_join_finalize: join-4: Syncing the CIB from pace01 to the rest of the cluster<br>Sep 15 14:00:45 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/78, version=0.115.1): ok (rc=0)<br>
Sep 15 14:00:45 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/79, version=0.115.1): ok (rc=0)<br>Sep 15 14:00:45 pace01 crmd: [32152]: info: do_dc_join_ack: join-4: Updating node state to member for pace02<br>
Sep 15 14:00:45 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/80, version=0.115.1): ok (rc=0)<br>Sep 15 14:00:45 pace01 crmd: [32152]: info: do_dc_join_ack: join-4: Updating node state to member for pace01<br>
Sep 15 14:00:45 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='pace02']/lrm (origin=local/crmd/81, version=0.115.2): ok (rc=0)<br>Sep 15 14:00:45 pace01 crmd: [32152]: info: erase_xpath_callback: Deletion of "//node_state[@uname='pace02']/lrm": ok (rc=0)<br>
Sep 15 14:00:45 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='pace01']/lrm (origin=local/crmd/83, version=0.115.4): ok (rc=0)<br>Sep 15 14:00:45 pace01 crmd: [32152]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]<br>
Sep 15 14:00:45 pace01 crmd: [32152]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.<br>Sep 15 14:00:45 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/85, version=0.115.5): ok (rc=0)<br>
Sep 15 14:00:45 pace01 crmd: [32152]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date<br>Sep 15 14:00:45 pace01 crmd: [32152]: info: crm_update_quorum: Updating quorum status to true (call=87)<br>
Sep 15 14:00:45 pace01 attrd: [32150]: info: attrd_local_callback: Sending full refresh (origin=crmd)<br>Sep 15 14:00:45 pace01 cib: [32148]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/87, version=0.115.5): ok (rc=0)<br>
Sep 15 14:00:45 pace01 crmd: [32152]: info: abort_transition_graph: do_te_invoke:191 - Triggered transition abort (complete=1) : Peer Cancelled<br>Sep 15 14:00:45 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-WebSite (<null>)<br>
Sep 15 14:00:45 pace01 crmd: [32152]: info: do_pe_invoke: Query 88: Requesting the current CIB: S_POLICY_ENGINE<br>Sep 15 14:00:45 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<br>
Sep 15 14:00:45 pace01 crmd: [32152]: info: erase_xpath_callback: Deletion of "//node_state[@uname='pace01']/lrm": ok (rc=0)<br>Sep 15 14:00:46 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate (<null>)<br>
Sep 15 14:00:46 pace01 crmd: [32152]: info: process_graph_event: Action ClusterIP_monitor_0/5 (0:7;5:3:7:6bbbad8b-ed26-4764-b444-d11cc1eae9de) initiated by a different transitioner<br>Sep 15 14:00:46 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-WebSite (<null>)<br>
Sep 15 14:00:46 pace01 crmd: [32152]: info: abort_transition_graph: process_graph_event:456 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=ClusterIP_monitor_0, magic=0:7;5:3:7:6bbbad8b-ed26-4764-b444-d11cc1eae9de, cib=0.115.5) : Foreign event<br>
Sep 15 14:00:46 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (<null>)<br>Sep 15 14:00:46 pace01 crmd: [32152]: info: process_graph_event: Action ClusterIP_start_0/7 (0:0;7:5:0:6bbbad8b-ed26-4764-b444-d11cc1eae9de) initiated by a different transitioner<br>
Sep 15 14:00:46 pace01 crmd: [32152]: info: abort_transition_graph: process_graph_event:456 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=ClusterIP_start_0, magic=0:0;7:5:0:6bbbad8b-ed26-4764-b444-d11cc1eae9de, cib=0.115.5) : Foreign event<br>
Sep 15 14:00:46 pace01 crmd: [32152]: info: process_graph_event: Action ClusterIP_monitor_30000/8 (0:0;8:5:0:6bbbad8b-ed26-4764-b444-d11cc1eae9de) initiated by a different transitioner<br>Sep 15 14:00:46 pace01 crmd: [32152]: info: abort_transition_graph: process_graph_event:456 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=ClusterIP_monitor_30000, magic=0:0;8:5:0:6bbbad8b-ed26-4764-b444-d11cc1eae9de, cib=0.115.5) : Foreign event<br>
Sep 15 14:00:46 pace01 crmd: [32152]: info: do_pe_invoke_callback: Invoking the PE: query=88, ref=pe_calc-dc-1284573646-36, seq=272, quorate=1<br>Sep 15 14:00:46 pace01 pengine: [32151]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Sep 15 14:00:46 pace01 crmd: [32152]: info: do_pe_invoke: Query 89: Requesting the current CIB: S_POLICY_ENGINE<br>Sep 15 14:00:46 pace01 pengine: [32151]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>
Sep 15 14:00:46 pace01 crmd: [32152]: info: do_pe_invoke: Query 90: Requesting the current CIB: S_POLICY_ENGINE<br>Sep 15 14:00:46 pace01 pengine: [32151]: info: determine_online_status: Node pace02 is online<br>Sep 15 14:00:46 pace01 crmd: [32152]: info: do_pe_invoke: Query 91: Requesting the current CIB: S_POLICY_ENGINE<br>
Sep 15 14:00:46 pace01 pengine: [32151]: info: determine_online_status: Node pace01 is online<br>Sep 15 14:00:46 pace01 crmd: [32152]: info: do_pe_invoke_callback: Invoking the PE: query=91, ref=pe_calc-dc-1284573646-37, seq=272, quorate=1<br>
Sep 15 14:00:46 pace01 pengine: [32151]: notice: native_print: ClusterIP (ocf::heartbeat:IPaddr2): Started pace01<br>Sep 15 14:00:46 pace01 pengine: [32151]: notice: native_print: WebSite (ocf::heartbeat:apache): Stopped <br>
Sep 15 14:00:46 pace01 pengine: [32151]: notice: RecurringOp: Start recurring monitor (60s) for WebSite on pace02<br>Sep 15 14:00:46 pace01 pengine: [32151]: notice: LogActions: Leave resource ClusterIP (Started pace01)<br>
Sep 15 14:00:46 pace01 pengine: [32151]: notice: LogActions: Start WebSite (pace02)<br>Sep 15 14:00:46 pace01 crmd: [32152]: info: handle_response: pe_calc calculation pe_calc-dc-1284573646-36 is obsolete<br>Sep 15 14:00:46 pace01 pengine: [32151]: info: process_pe_message: Transition 5: PEngine Input stored in: /var/lib/pengine/pe-input-3231.bz2<br>
Sep 15 14:00:46 pace01 pengine: [32151]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>Sep 15 14:00:46 pace01 pengine: [32151]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>
Sep 15 14:00:46 pace01 pengine: [32151]: info: determine_online_status: Node pace02 is online<br>Sep 15 14:00:46 pace01 pengine: [32151]: info: determine_online_status: Node pace01 is online<br>Sep 15 14:00:46 pace01 pengine: [32151]: notice: native_print: ClusterIP (ocf::heartbeat:IPaddr2): Started pace01<br>
Sep 15 14:00:47 pace01 pengine: [32151]: notice: native_print: WebSite (ocf::heartbeat:apache): Stopped <br>Sep 15 14:00:47 pace01 pengine: [32151]: notice: RecurringOp: Start recurring monitor (60s) for WebSite on pace02<br>
Sep 15 14:00:47 pace01 pengine: [32151]: notice: LogActions: Leave resource ClusterIP (Started pace01)<br>Sep 15 14:00:47 pace01 pengine: [32151]: notice: LogActions: Start WebSite (pace02)<br>Sep 15 14:00:47 pace01 crmd: [32152]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>
Sep 15 14:00:47 pace01 pengine: [32151]: info: process_pe_message: Transition 6: PEngine Input stored in: /var/lib/pengine/pe-input-3232.bz2<br>Sep 15 14:00:47 pace01 crmd: [32152]: info: unpack_graph: Unpacked transition 6: 7 actions in 7 synapses<br>
Sep 15 14:00:47 pace01 crmd: [32152]: info: do_te_invoke: Processing graph 6 (ref=pe_calc-dc-1284573646-37) derived from /var/lib/pengine/pe-input-3232.bz2<br>Sep 15 14:00:47 pace01 crmd: [32152]: info: te_rsc_command: Initiating action 5: monitor WebSite_monitor_0 on pace01 (local)<br>
Sep 15 14:00:47 pace01 crmd: [32152]: info: do_lrm_rsc_op: Performing key=5:6:7:b7d1a9ca-8a97-4fd9-88b8-b8d303c3ebbf op=WebSite_monitor_0 )<br>Sep 15 14:00:47 pace01 lrmd: [32149]: info: rsc:WebSite:5: probe<br>Sep 15 14:00:47 pace01 crmd: [32152]: info: te_rsc_command: Initiating action 7: monitor WebSite_monitor_0 on pace02<br>
Sep 15 14:00:47 pace01 crmd: [32152]: info: match_graph_event: Action WebSite_monitor_0 (7) confirmed on pace02 (rc=0)<br>Sep 15 14:00:47 pace01 crmd: [32152]: info: te_rsc_command: Initiating action 6: probe_complete probe_complete on pace02 - no waiting<br>
Sep 15 14:00:47 pace01 apache[32399]: INFO: apache not running<br>Sep 15 14:00:47 pace01 crmd: [32152]: info: process_lrm_event: LRM operation WebSite_monitor_0 (call=5, rc=7, cib-update=92, confirmed=true) not running<br>
Sep 15 14:00:47 pace01 crmd: [32152]: info: match_graph_event: Action WebSite_monitor_0 (5) confirmed on pace01 (rc=0)<br>Sep 15 14:00:47 pace01 crmd: [32152]: info: te_rsc_command: Initiating action 4: probe_complete probe_complete on pace01 (local) - no waiting<br>
Sep 15 14:00:47 pace01 crmd: [32152]: info: te_pseudo_action: Pseudo action 3 fired and confirmed<br>Sep 15 14:00:47 pace01 crmd: [32152]: info: te_rsc_command: Initiating action 10: start WebSite_start_0 on pace02<br>Sep 15 14:00:49 pace01 crmd: [32152]: WARN: status_from_rc: Action 10 (WebSite_start_0) on pace02 failed (target: 0 vs. rc: 1): Error<br>
Sep 15 14:00:49 pace01 crmd: [32152]: WARN: update_failcount: Updating failcount for WebSite on pace02 after failed start: rc=1 (update=INFINITY, time=1284573649)<br>Sep 15 14:00:49 pace01 crmd: [32152]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=WebSite_start_0, magic=0:1;10:6:0:b7d1a9ca-8a97-4fd9-88b8-b8d303c3ebbf, cib=0.115.8) : Event failed<br>
Sep 15 14:00:49 pace01 crmd: [32152]: info: update_abort_priority: Abort priority upgraded from 0 to 1<br>Sep 15 14:00:49 pace01 crmd: [32152]: info: update_abort_priority: Abort action done superceeded by restart<br>Sep 15 14:00:49 pace01 crmd: [32152]: info: match_graph_event: Action WebSite_start_0 (10) confirmed on pace02 (rc=4)<br>
Sep 15 14:00:49 pace01 crmd: [32152]: info: run_graph: ====================================================<br>Sep 15 14:00:49 pace01 crmd: [32152]: notice: run_graph: Transition 6 (Complete=6, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-3232.bz2): Stopped<br>
Sep 15 14:00:49 pace01 crmd: [32152]: info: te_graph_trigger: Transition 6 is now complete<br>Sep 15 14:00:49 pace01 crmd: [32152]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
Sep 15 14:00:49 pace01 crmd: [32152]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.<br>Sep 15 14:00:49 pace01 crmd: [32152]: info: do_pe_invoke: Query 93: Requesting the current CIB: S_POLICY_ENGINE<br>
Sep 15 14:00:49 pace01 crmd: [32152]: info: do_pe_invoke_callback: Invoking the PE: query=93, ref=pe_calc-dc-1284573649-43, seq=272, quorate=1<br>Sep 15 14:00:49 pace01 pengine: [32151]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Sep 15 14:00:49 pace01 crmd: [32152]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=pace02, magic=NA, cib=0.115.9) : Transient attribute: update<br>
Sep 15 14:00:49 pace01 pengine: [32151]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>Sep 15 14:00:49 pace01 crmd: [32152]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=pace02, magic=NA, cib=0.115.10) : Transient attribute: update<br>
Sep 15 14:00:49 pace01 pengine: [32151]: info: determine_online_status: Node pace02 is online<br>Sep 15 14:00:49 pace01 crmd: [32152]: info: do_pe_invoke: Query 94: Requesting the current CIB: S_POLICY_ENGINE<br>Sep 15 14:00:49 pace01 pengine: [32151]: WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on pace02: unknown error (1)<br>
Sep 15 14:00:49 pace01 crmd: [32152]: info: do_pe_invoke: Query 95: Requesting the current CIB: S_POLICY_ENGINE<br>Sep 15 14:00:49 pace01 pengine: [32151]: info: determine_online_status: Node pace01 is online<br>Sep 15 14:00:49 pace01 crmd: [32152]: info: do_pe_invoke_callback: Invoking the PE: query=95, ref=pe_calc-dc-1284573649-44, seq=272, quorate=1<br>
Sep 15 14:00:49 pace01 pengine: [32151]: notice: native_print: ClusterIP (ocf::heartbeat:IPaddr2): Started pace01<br>Sep 15 14:00:49 pace01 pengine: [32151]: notice: native_print: WebSite (ocf::heartbeat:apache): Started pace02 FAILED<br>
Sep 15 14:00:50 pace01 pengine: [32151]: notice: RecurringOp: Start recurring monitor (60s) for WebSite on pace02<br>Sep 15 14:00:50 pace01 pengine: [32151]: notice: LogActions: Leave resource ClusterIP (Started pace01)<br>
Sep 15 14:00:50 pace01 pengine: [32151]: notice: LogActions: Recover resource WebSite (Started pace02)<br>Sep 15 14:00:50 pace01 crmd: [32152]: info: handle_response: pe_calc calculation pe_calc-dc-1284573649-43 is obsolete<br>
Sep 15 14:00:50 pace01 pengine: [32151]: info: process_pe_message: Transition 7: PEngine Input stored in: /var/lib/pengine/pe-input-3233.bz2<br>Sep 15 14:00:50 pace01 pengine: [32151]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Sep 15 14:00:50 pace01 pengine: [32151]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>Sep 15 14:00:50 pace01 pengine: [32151]: info: determine_online_status: Node pace02 is online<br>
Sep 15 14:00:50 pace01 pengine: [32151]: WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on pace02: unknown error (1)<br>Sep 15 14:00:50 pace01 pengine: [32151]: info: determine_online_status: Node pace01 is online<br>
Sep 15 14:00:50 pace01 pengine: [32151]: notice: native_print: ClusterIP (ocf::heartbeat:IPaddr2): Started pace01<br>Sep 15 14:00:50 pace01 pengine: [32151]: notice: native_print: WebSite (ocf::heartbeat:apache): Started pace02 FAILED<br>
Sep 15 14:00:50 pace01 pengine: [32151]: info: get_failcount: WebSite has failed INFINITY times on pace02<br>Sep 15 14:00:50 pace01 pengine: [32151]: WARN: common_apply_stickiness: Forcing WebSite away from pace02 after 1000000 failures (max=1000000)<br>
Sep 15 14:00:50 pace01 pengine: [32151]: notice: RecurringOp: Start recurring monitor (60s) for WebSite on pace01<br>Sep 15 14:00:50 pace01 pengine: [32151]: notice: LogActions: Leave resource ClusterIP (Started pace01)<br>
Sep 15 14:00:50 pace01 pengine: [32151]: notice: LogActions: Move resource WebSite (Started pace02 -> pace01)<br>Sep 15 14:00:50 pace01 crmd: [32152]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>
Sep 15 14:00:50 pace01 pengine: [32151]: info: process_pe_message: Transition 8: PEngine Input stored in: /var/lib/pengine/pe-input-3234.bz2<br>Sep 15 14:00:50 pace01 crmd: [32152]: info: unpack_graph: Unpacked transition 8: 4 actions in 4 synapses<br>
Sep 15 14:00:50 pace01 crmd: [32152]: info: do_te_invoke: Processing graph 8 (ref=pe_calc-dc-1284573649-44) derived from /var/lib/pengine/pe-input-3234.bz2<br>Sep 15 14:00:50 pace01 crmd: [32152]: info: te_rsc_command: Initiating action 1: stop WebSite_stop_0 on pace02<br>
Sep 15 14:00:52 pace01 crmd: [32152]: info: match_graph_event: Action WebSite_stop_0 (1) confirmed on pace02 (rc=0)<br>Sep 15 14:00:52 pace01 crmd: [32152]: info: te_rsc_command: Initiating action 9: start WebSite_start_0 on pace01 (local)<br>
Sep 15 14:00:52 pace01 crmd: [32152]: info: do_lrm_rsc_op: Performing key=9:8:0:b7d1a9ca-8a97-4fd9-88b8-b8d303c3ebbf op=WebSite_start_0 )<br>Sep 15 14:00:52 pace01 lrmd: [32149]: info: rsc:WebSite:6: start<br>Sep 15 14:00:52 pace01 crmd: [32152]: info: te_pseudo_action: Pseudo action 3 fired and confirmed<br>
Sep 15 14:00:52 pace01 apache[32434]: INFO: apache not running<br>Sep 15 14:00:52 pace01 apache[32434]: INFO: waiting for apache /etc/httpd/conf/httpd.conf to come up<br>Sep 15 14:00:53 pace01 crmd: [32152]: info: process_lrm_event: LRM operation WebSite_start_0 (call=6, rc=1, cib-update=96, confirmed=true) unknown error<br>
Sep 15 14:00:53 pace01 crmd: [32152]: WARN: status_from_rc: Action 9 (WebSite_start_0) on pace01 failed (target: 0 vs. rc: 1): Error<br>Sep 15 14:00:53 pace01 crmd: [32152]: WARN: update_failcount: Updating failcount for WebSite on pace01 after failed start: rc=1 (update=INFINITY, time=1284573653)<br>
Sep 15 14:00:53 pace01 crmd: [32152]: info: abort_transition_graph: match_graph_event:272 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=WebSite_start_0, magic=0:1;9:8:0:b7d1a9ca-8a97-4fd9-88b8-b8d303c3ebbf, cib=0.115.12) : Event failed<br>
Sep 15 14:00:53 pace01 crmd: [32152]: info: update_abort_priority: Abort priority upgraded from 0 to 1<br>Sep 15 14:00:53 pace01 crmd: [32152]: info: update_abort_priority: Abort action done superceeded by restart<br>Sep 15 14:00:53 pace01 crmd: [32152]: info: match_graph_event: Action WebSite_start_0 (9) confirmed on pace01 (rc=4)<br>
Sep 15 14:00:53 pace01 crmd: [32152]: info: run_graph: ====================================================<br>Sep 15 14:00:53 pace01 crmd: [32152]: notice: run_graph: Transition 8 (Complete=3, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pengine/pe-input-3234.bz2): Stopped<br>
Sep 15 14:00:53 pace01 crmd: [32152]: info: te_graph_trigger: Transition 8 is now complete<br>Sep 15 14:00:53 pace01 crmd: [32152]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
Sep 15 14:00:53 pace01 crmd: [32152]: info: do_state_transition: All 2 cluster nodes are eligible to run resources.<br>Sep 15 14:00:53 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-WebSite (INFINITY)<br>
Sep 15 14:00:53 pace01 crmd: [32152]: info: do_pe_invoke: Query 97: Requesting the current CIB: S_POLICY_ENGINE<br>Sep 15 14:00:53 pace01 attrd: [32150]: info: attrd_perform_update: Sent update 74: fail-count-WebSite=INFINITY<br>
Sep 15 14:00:53 pace01 crmd: [32152]: info: do_pe_invoke_callback: Invoking the PE: query=97, ref=pe_calc-dc-1284573653-47, seq=272, quorate=1<br>Sep 15 14:00:53 pace01 pengine: [32151]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Sep 15 14:00:53 pace01 attrd: [32150]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-WebSite (1284573653)<br>Sep 15 14:00:54 pace01 crmd: [32152]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=pace01, magic=NA, cib=0.115.13) : Transient attribute: update<br>
Sep 15 14:00:54 pace01 pengine: [32151]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>Sep 15 14:00:54 pace01 attrd: [32150]: info: attrd_perform_update: Sent update 77: last-failure-WebSite=1284573653<br>
Sep 15 14:00:54 pace01 crmd: [32152]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=pace01, magic=NA, cib=0.115.14) : Transient attribute: update<br>
Sep 15 14:00:54 pace01 pengine: [32151]: info: determine_online_status: Node pace02 is online<br>Sep 15 14:00:54 pace01 crmd: [32152]: info: do_pe_invoke: Query 98: Requesting the current CIB: S_POLICY_ENGINE<br>Sep 15 14:00:54 pace01 pengine: [32151]: WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on pace02: unknown error (1)<br>
Sep 15 14:00:54 pace01 crmd: [32152]: info: do_pe_invoke: Query 99: Requesting the current CIB: S_POLICY_ENGINE<br>Sep 15 14:00:54 pace01 pengine: [32151]: info: determine_online_status: Node pace01 is online<br>Sep 15 14:00:54 pace01 crmd: [32152]: info: do_pe_invoke_callback: Invoking the PE: query=99, ref=pe_calc-dc-1284573654-48, seq=272, quorate=1<br>
Sep 15 14:00:54 pace01 pengine: [32151]: WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on pace01: unknown error (1)<br>Sep 15 14:00:54 pace01 pengine: [32151]: notice: native_print: ClusterIP (ocf::heartbeat:IPaddr2): Started pace01<br>
Sep 15 14:00:54 pace01 pengine: [32151]: notice: native_print: WebSite (ocf::heartbeat:apache): Started pace01 FAILED<br>Sep 15 14:00:54 pace01 pengine: [32151]: info: get_failcount: WebSite has failed INFINITY times on pace02<br>
Sep 15 14:00:54 pace01 pengine: [32151]: WARN: common_apply_stickiness: Forcing WebSite away from pace02 after 1000000 failures (max=1000000)<br>Sep 15 14:00:54 pace01 pengine: [32151]: notice: RecurringOp: Start recurring monitor (60s) for WebSite on pace01<br>
Sep 15 14:00:54 pace01 pengine: [32151]: notice: LogActions: Leave resource ClusterIP (Started pace01)<br>Sep 15 14:00:54 pace01 pengine: [32151]: notice: LogActions: Recover resource WebSite (Started pace01)<br>Sep 15 14:00:54 pace01 crmd: [32152]: info: handle_response: pe_calc calculation pe_calc-dc-1284573653-47 is obsolete<br>
Sep 15 14:00:54 pace01 pengine: [32151]: info: process_pe_message: Transition 9: PEngine Input stored in: /var/lib/pengine/pe-input-3235.bz2<br>Sep 15 14:00:54 pace01 pengine: [32151]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
Sep 15 14:00:54 pace01 pengine: [32151]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0<br>Sep 15 14:00:54 pace01 pengine: [32151]: info: determine_online_status: Node pace02 is online<br>
Sep 15 14:00:54 pace01 pengine: [32151]: WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on pace02: unknown error (1)<br>Sep 15 14:00:54 pace01 pengine: [32151]: info: determine_online_status: Node pace01 is online<br>
Sep 15 14:00:54 pace01 pengine: [32151]: WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on pace01: unknown error (1)<br>Sep 15 14:00:54 pace01 pengine: [32151]: notice: native_print: ClusterIP (ocf::heartbeat:IPaddr2): Started pace01<br>
Sep 15 14:00:55 pace01 pengine: [32151]: notice: native_print: WebSite (ocf::heartbeat:apache): Started pace01 FAILED<br>Sep 15 14:00:55 pace01 pengine: [32151]: info: get_failcount: WebSite has failed INFINITY times on pace01<br>
Sep 15 14:00:55 pace01 pengine: [32151]: WARN: common_apply_stickiness: Forcing WebSite away from pace01 after 1000000 failures (max=1000000)<br>Sep 15 14:00:55 pace01 pengine: [32151]: info: get_failcount: WebSite has failed INFINITY times on pace02<br>
Sep 15 14:00:55 pace01 pengine: [32151]: WARN: common_apply_stickiness: Forcing WebSite away from pace02 after 1000000 failures (max=1000000)<br>Sep 15 14:00:55 pace01 pengine: [32151]: info: native_color: Resource WebSite cannot run anywhere<br>
Sep 15 14:00:55 pace01 pengine: [32151]: notice: LogActions: Leave resource ClusterIP (Started pace01)<br>Sep 15 14:00:55 pace01 pengine: [32151]: notice: LogActions: Stop resource WebSite (pace01)<br>Sep 15 14:00:55 pace01 crmd: [32152]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<br>
Sep 15 14:00:55 pace01 crmd: [32152]: info: unpack_graph: Unpacked transition 10: 2 actions in 2 synapses<br>Sep 15 14:00:55 pace01 pengine: [32151]: info: process_pe_message: Transition 10: PEngine Input stored in: /var/lib/pengine/pe-input-3236.bz2<br>
Sep 15 14:00:55 pace01 crmd: [32152]: info: do_te_invoke: Processing graph 10 (ref=pe_calc-dc-1284573654-48) derived from /var/lib/pengine/pe-input-3236.bz2<br>Sep 15 14:00:55 pace01 crmd: [32152]: info: te_rsc_command: Initiating action 2: stop WebSite_stop_0 on pace01 (local)<br>
Sep 15 14:00:55 pace01 crmd: [32152]: info: do_lrm_rsc_op: Performing key=2:10:0:b7d1a9ca-8a97-4fd9-88b8-b8d303c3ebbf op=WebSite_stop_0 )<br>Sep 15 14:00:55 pace01 lrmd: [32149]: info: rsc:WebSite:7: stop<br>Sep 15 14:00:56 pace01 lrmd: [32149]: info: RA output: (WebSite:stop:stderr) /usr/lib/ocf/resource.d//heartbeat/apache: line 437: kill: (32467) - No such process <br>
Sep 15 14:00:56 pace01 apache[32529]: INFO: Killing apache PID 32467<br>Sep 15 14:00:56 pace01 apache[32529]: INFO: apache stopped.<br>Sep 15 14:00:57 pace01 crmd: [32152]: info: process_lrm_event: LRM operation WebSite_stop_0 (call=7, rc=0, cib-update=100, confirmed=true) ok<br>
Sep 15 14:00:57 pace01 crmd: [32152]: info: match_graph_event: Action WebSite_stop_0 (2) confirmed on pace01 (rc=0)<br>Sep 15 14:00:57 pace01 crmd: [32152]: info: te_pseudo_action: Pseudo action 3 fired and confirmed<br>Sep 15 14:00:57 pace01 crmd: [32152]: info: run_graph: ====================================================<br>
Sep 15 14:00:57 pace01 crmd: [32152]: notice: run_graph: Transition 10 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-3236.bz2): Complete<br>Sep 15 14:00:57 pace01 crmd: [32152]: info: te_graph_trigger: Transition 10 is now complete<br>
Sep 15 14:00:57 pace01 crmd: [32152]: info: notify_crmd: Transition 10 status: done - <null><br>Sep 15 14:00:57 pace01 crmd: [32152]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
Sep 15 14:00:57 pace01 crmd: [32152]: info: do_state_transition: Starting PEngine Recheck Timer<br><br><br>Thanks in advance for your help!<br>~Brandon<br><br><br><br>