Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: process_graph_event:520 - Triggered transition abort (complete=1, node=nbgprepdb6, tag=lrm_rsc_op, id=pgsql_last_failure_0, magic=2:1;24:4076:8:48387f7e-8de3-4eed-bab6-3db414426226, cib=0.36.7) : Old event Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: warning: update_failcount: Updating failcount for pgsql on nbgprepdb6 after failed monitor: rc=1 (update=value++, time=1452765464) Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: info: process_graph_event: Detected action (4076.24) pgsql_monitor_10000.92=unknown error: failed Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] Jan 14 10:57:44 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/crmd/32, version=0.36.7) Jan 14 10:57:44 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/4241, version=0.36.7) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepwitness56 is online Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb5 is online Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb6 is online Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: warning: unpack_rsc_op: Processing failed op monitor for pgsql:1 on nbgprepdb6: unknown error (1) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: ClusterMon (ocf::pacemaker:ClusterMon): Started nbgprepwitness56 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: vip-slave (ocf::heartbeat:IPaddr2): Started nbgprepdb5 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: group_print: Resource Group: master-group Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: vip-master (ocf::heartbeat:IPaddr2): Started nbgprepdb6 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: vip-rep (ocf::heartbeat:IPaddr2): Started nbgprepdb6 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: clone_print: Master/Slave Set: msPostgresql [pgsql] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: pgsql (ocf::heartbeat:pgsql): Master nbgprepdb6 FAILED Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: short_print: Slaves: [ nbgprepdb5 ] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: short_print: Stopped: [ nbgprepwitness56 ] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: clone_print: Clone Set: clnPingCheck [pingCheck] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: short_print: Started: [ nbgprepdb5 nbgprepdb6 nbgprepwitness56 ] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: master_color: Promoting pgsql:1 (Master nbgprepdb6) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: master_color: msPostgresql: Promoted 1 instances of a possible 1 to master Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:1 on nbgprepdb6 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:1 on nbgprepdb6 Jan 14 10:57:44 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='fail-count-pgsql']: No such device or address (rc=-6, origin=local/attrd/112, version=0.36.7) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave ClusterMon (Started nbgprepwitness56) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave vip-slave (Started nbgprepdb5) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave vip-master (Started nbgprepdb6) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave vip-rep (Started nbgprepdb6) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pgsql:0 (Slave nbgprepdb5) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: notice: LogActions: Recover pgsql:1 (Master nbgprepdb6) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:0 (Started nbgprepwitness56) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:1 (Started nbgprepdb5) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:2 (Started nbgprepdb6) Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=1, node=nbgprepdb6, tag=nvpair, id=status-739251316-fail-count-pgsql, name=fail-count-pgsql, value=1, magic=NA, cib=0.36.8) : Transient attribute: update Jan 14 10:57:44 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/112, version=0.36.8) Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=1, node=nbgprepdb6, tag=nvpair, id=status-739251316-last-failure-pgsql, name=last-failure-pgsql, value=1452765464, magic=NA, cib=0.36.9) : Transient attribute: update Jan 14 10:57:44 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/115, version=0.36.9) Jan 14 10:57:44 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/4242, version=0.36.9) Jan 14 10:57:44 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/4243, version=0.36.9) Jan 14 10:57:44 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='last-failure-pgsql']: No such device or address (rc=-6, origin=local/attrd/113, version=0.36.9) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: notice: crm_ipc_prepare: Message exceeds the configured ipc limit (51200 bytes), consider configuring PCMK_ipc_buffer to 110392 or higher to avoid compression overheads Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: crm_compress_string: Compressed 55156 bytes into 2509 (ratio 21:1) in 15ms Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: notice: process_pe_message: Calculated Transition 4103: /var/lib/pacemaker/pengine/pe-input-102.bz2 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepwitness56 is online Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb5 is online Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb6 is online Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: warning: unpack_rsc_op: Processing failed op monitor for pgsql:1 on nbgprepdb6: unknown error (1) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: ClusterMon (ocf::pacemaker:ClusterMon): Started nbgprepwitness56 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: vip-slave (ocf::heartbeat:IPaddr2): Started nbgprepdb5 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: group_print: Resource Group: master-group Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: vip-master (ocf::heartbeat:IPaddr2): Started nbgprepdb6 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: vip-rep (ocf::heartbeat:IPaddr2): Started nbgprepdb6 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: clone_print: Master/Slave Set: msPostgresql [pgsql] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: native_print: pgsql (ocf::heartbeat:pgsql): Master nbgprepdb6 FAILED Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: short_print: Slaves: [ nbgprepdb5 ] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: short_print: Stopped: [ nbgprepwitness56 ] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: clone_print: Clone Set: clnPingCheck [pingCheck] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: short_print: Started: [ nbgprepdb5 nbgprepdb6 nbgprepwitness56 ] Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:0 has failed 1 times on nbgprepdb6 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: common_apply_stickiness: msPostgresql can fail 999999 more times on nbgprepdb6 before being forced off Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:1 has failed 1 times on nbgprepdb6 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: common_apply_stickiness: msPostgresql can fail 999999 more times on nbgprepdb6 before being forced off Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: master_color: Promoting pgsql:1 (Master nbgprepdb6) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: master_color: msPostgresql: Promoted 1 instances of a possible 1 to master Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:1 on nbgprepdb6 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:1 on nbgprepdb6 Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave ClusterMon (Started nbgprepwitness56) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave vip-slave (Started nbgprepdb5) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave vip-master (Started nbgprepdb6) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave vip-rep (Started nbgprepdb6) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pgsql:0 (Slave nbgprepdb5) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: notice: LogActions: Recover pgsql:1 (Master nbgprepdb6) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:0 (Started nbgprepwitness56) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:1 (Started nbgprepdb5) Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:2 (Started nbgprepdb6) Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: info: handle_response: pe_calc calculation pe_calc-dc-1452765464-4317 is obsolete Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: info: crm_compress_string: Compressed 55156 bytes into 2508 (ratio 21:1) in 9ms Jan 14 10:57:44 [31406] nbgprepwitness56 pengine: notice: process_pe_message: Calculated Transition 4104: /var/lib/pacemaker/pengine/pe-input-103.bz2 Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: info: do_te_invoke: Processing graph 4104 (ref=pe_calc-dc-1452765464-4318) derived from /var/lib/pacemaker/pengine/pe-input-103.bz2 Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 80: notify pgsql_pre_notify_demote_0 on nbgprepdb5 Jan 14 10:57:44 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 82: notify pgsql_pre_notify_demote_0 on nbgprepdb6 Jan 14 10:57:45 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (80) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:45 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (82) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:45 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 30: demote pgsql_demote_0 on nbgprepdb6 Jan 14 10:57:45 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='master-pgsql']: No such device or address (rc=-6, origin=local/attrd/114, version=0.36.9) Jan 14 10:57:45 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=0, node=nbgprepdb6, tag=nvpair, id=status-739251316-master-pgsql, name=master-pgsql, value=-INFINITY, magic=NA, cib=0.36.10) : Transient attribute: update Jan 14 10:57:45 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/117, version=0.36.10) Jan 14 10:57:45 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/attrd/26, version=0.36.10) Jan 14 10:57:45 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='pgsql-master-baseline']: No such device or address (rc=-6, origin=local/attrd/115, version=0.36.10) Jan 14 10:57:45 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:188 - Triggered transition abort (complete=0, node=nbgprepdb6, tag=transient_attributes, id=739251316, magic=NA, cib=0.36.11) : Transient attribute: removal Jan 14 10:57:45 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/119, version=0.36.11) Jan 14 10:57:45 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/121, version=0.36.11) Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=0, node=nbgprepdb6, tag=nvpair, id=status-739251316-pgsql-status, name=pgsql-status, value=STOP, magic=NA, cib=0.36.12) : Transient attribute: update Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/123, version=0.36.12) Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_demote_0 (30) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 81: notify pgsql_post_notify_demote_0 on nbgprepdb5 Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 83: notify pgsql_post_notify_demote_0 on nbgprepdb6 Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/crmd/33, version=0.36.13) Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/attrd/29, version=0.36.13) Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='pgsql-status']: No such device or address (rc=-6, origin=local/attrd/116, version=0.36.13) Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (83) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='pgsql-master-baseline']: No such device or address (rc=-6, origin=local/attrd/117, version=0.36.13) Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=0, node=nbgprepdb5, tag=nvpair, id=status-739251315-pgsql-master-baseline, name=pgsql-master-baseline, value=000004AFF4000090, magic=NA, cib=0.36.14) : Transient attribute: update Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/attrd/32, version=0.36.14) Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='pgsql-status']: No such device or address (rc=-6, origin=local/attrd/118, version=0.36.14) Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=0, node=nbgprepdb5, tag=nvpair, id=status-739251315-pgsql-status, name=pgsql-status, value=HS:alone, magic=NA, cib=0.36.15) : Transient attribute: update Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (81) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: notice: run_graph: Transition 4104 (Complete=11, Pending=0, Fired=0, Skipped=19, Incomplete=14, Source=/var/lib/pacemaker/pengine/pe-input-103.bz2): Stopped Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/attrd/34, version=0.36.15) Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/126, version=0.36.15) Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/4244, version=0.36.15) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepwitness56 is online Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb5 is online Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb6 is online Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: warning: unpack_rsc_op: Processing failed op monitor for pgsql:1 on nbgprepdb6: unknown error (1) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: native_print: ClusterMon (ocf::pacemaker:ClusterMon): Started nbgprepwitness56 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: native_print: vip-slave (ocf::heartbeat:IPaddr2): Started nbgprepdb5 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: group_print: Resource Group: master-group Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: native_print: vip-master (ocf::heartbeat:IPaddr2): Started nbgprepdb6 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: native_print: vip-rep (ocf::heartbeat:IPaddr2): Started nbgprepdb6 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: clone_print: Master/Slave Set: msPostgresql [pgsql] Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: native_print: pgsql (ocf::heartbeat:pgsql): Slave nbgprepdb6 FAILED Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: short_print: Slaves: [ nbgprepdb5 ] Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: short_print: Stopped: [ nbgprepwitness56 ] Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: clone_print: Clone Set: clnPingCheck [pingCheck] Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: short_print: Started: [ nbgprepdb5 nbgprepdb6 nbgprepwitness56 ] Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:0 has failed 1 times on nbgprepdb6 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: common_apply_stickiness: msPostgresql can fail 999999 more times on nbgprepdb6 before being forced off Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:1 has failed 1 times on nbgprepdb6 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: common_apply_stickiness: msPostgresql can fail 999999 more times on nbgprepdb6 before being forced off Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: native_color: Resource vip-slave cannot run anywhere Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: master_color: Promoting pgsql:0 (Slave nbgprepdb5) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: master_color: msPostgresql: Promoted 1 instances of a possible 1 to master Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for vip-master on nbgprepdb5 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for vip-rep on nbgprepdb5 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: RecurringOp: Cancelling action pgsql:0_monitor_30000 (Slave vs. Master) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:0 on nbgprepdb5 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (30s) for pgsql:1 on nbgprepdb6 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: RecurringOp: Cancelling action pgsql:0_monitor_30000 (Slave vs. Master) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:0 on nbgprepdb5 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (30s) for pgsql:1 on nbgprepdb6 Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: LogActions: Leave ClusterMon (Started nbgprepwitness56) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: notice: LogActions: Stop vip-slave (nbgprepdb5) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: notice: LogActions: Move vip-master (Started nbgprepdb6 -> nbgprepdb5) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: notice: LogActions: Move vip-rep (Started nbgprepdb6 -> nbgprepdb5) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: notice: LogActions: Promote pgsql:0 (Slave -> Master nbgprepdb5) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: notice: LogActions: Recover pgsql:1 (Slave nbgprepdb6) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:0 (Started nbgprepwitness56) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:1 (Started nbgprepdb5) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:2 (Started nbgprepdb6) Jan 14 10:57:54 [31406] nbgprepwitness56 pengine: notice: process_pe_message: Calculated Transition 4105: /var/lib/pacemaker/pengine/pe-input-104.bz2 Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: do_te_invoke: Processing graph 4105 (ref=pe_calc-dc-1452765474-4324) derived from /var/lib/pacemaker/pengine/pe-input-104.bz2 Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 17: stop vip-slave_stop_0 on nbgprepdb5 Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 21: stop vip-rep_stop_0 on nbgprepdb6 Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 5: cancel pgsql_cancel_30000 on nbgprepdb5 Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 75: notify pgsql_pre_notify_stop_0 on nbgprepdb5 Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 77: notify pgsql_pre_notify_stop_0 on nbgprepdb6 Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_monitor_30000 (5) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:54 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:258 - Triggered transition abort (complete=0, node=nbgprepdb5, tag=lrm_rsc_op, id=pgsql_monitor_30000, magic=0:0;28:4099:0:48387f7e-8de3-4eed-bab6-3db414426226, cib=0.36.16) : Resource op removal Jan 14 10:57:54 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/22, version=0.36.16) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action vip-slave_stop_0 (17) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:55 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/23, version=0.36.17) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action vip-rep_stop_0 (21) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:55 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/crmd/34, version=0.36.18) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (77) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (75) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: notice: run_graph: Transition 4105 (Complete=7, Pending=0, Fired=0, Skipped=25, Incomplete=14, Source=/var/lib/pacemaker/pengine/pe-input-104.bz2): Stopped Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 14 10:57:55 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/4245, version=0.36.18) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepwitness56 is online Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb5 is online Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb6 is online Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: warning: unpack_rsc_op: Processing failed op monitor for pgsql:1 on nbgprepdb6: unknown error (1) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: native_print: ClusterMon (ocf::pacemaker:ClusterMon): Started nbgprepwitness56 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: native_print: vip-slave (ocf::heartbeat:IPaddr2): Stopped Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: group_print: Resource Group: master-group Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: native_print: vip-master (ocf::heartbeat:IPaddr2): Started nbgprepdb6 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: native_print: vip-rep (ocf::heartbeat:IPaddr2): Stopped Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: clone_print: Master/Slave Set: msPostgresql [pgsql] Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: native_print: pgsql (ocf::heartbeat:pgsql): Slave nbgprepdb6 FAILED Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: short_print: Slaves: [ nbgprepdb5 ] Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: short_print: Stopped: [ nbgprepwitness56 ] Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: clone_print: Clone Set: clnPingCheck [pingCheck] Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: short_print: Started: [ nbgprepdb5 nbgprepdb6 nbgprepwitness56 ] Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:0 has failed 1 times on nbgprepdb6 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: common_apply_stickiness: msPostgresql can fail 999999 more times on nbgprepdb6 before being forced off Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:1 has failed 1 times on nbgprepdb6 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: common_apply_stickiness: msPostgresql can fail 999999 more times on nbgprepdb6 before being forced off Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: native_color: Resource vip-slave cannot run anywhere Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: master_color: Promoting pgsql:0 (Slave nbgprepdb5) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: master_color: msPostgresql: Promoted 1 instances of a possible 1 to master Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for vip-master on nbgprepdb5 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for vip-rep on nbgprepdb5 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:0 on nbgprepdb5 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (30s) for pgsql:1 on nbgprepdb6 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:0 on nbgprepdb5 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (30s) for pgsql:1 on nbgprepdb6 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: LogActions: Leave ClusterMon (Started nbgprepwitness56) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: LogActions: Leave vip-slave (Stopped) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: notice: LogActions: Move vip-master (Started nbgprepdb6 -> nbgprepdb5) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: notice: LogActions: Start vip-rep (nbgprepdb5) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: notice: LogActions: Promote pgsql:0 (Slave -> Master nbgprepdb5) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: notice: LogActions: Recover pgsql:1 (Slave nbgprepdb6) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:0 (Started nbgprepwitness56) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:1 (Started nbgprepdb5) Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:2 (Started nbgprepdb6) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: do_te_invoke: Processing graph 4106 (ref=pe_calc-dc-1452765475-4330) derived from /var/lib/pacemaker/pengine/pe-input-105.bz2 Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 14: stop vip-master_stop_0 on nbgprepdb6 Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 70: notify pgsql_pre_notify_stop_0 on nbgprepdb5 Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 72: notify pgsql_pre_notify_stop_0 on nbgprepdb6 Jan 14 10:57:55 [31406] nbgprepwitness56 pengine: notice: process_pe_message: Calculated Transition 4106: /var/lib/pacemaker/pengine/pe-input-105.bz2 Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action vip-master_stop_0 (14) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:55 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/crmd/35, version=0.36.19) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (72) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (70) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 6: stop pgsql_stop_0 on nbgprepdb6 Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_stop_0 (6) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 71: notify pgsql_post_notify_stop_0 on nbgprepdb5 Jan 14 10:57:55 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/crmd/36, version=0.36.20) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (71) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:55 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 27: start pgsql_start_0 on nbgprepdb6 Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: warning: status_from_rc: Action 27 (pgsql_start_0) on nbgprepdb6 failed (target: 0 vs. rc: 1): Error Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: warning: update_failcount: Updating failcount for pgsql on nbgprepdb6 after failed start: rc=1 (update=INFINITY, time=1452765476) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: match_graph_event:313 - Triggered transition abort (complete=0, node=nbgprepdb6, tag=lrm_rsc_op, id=pgsql_last_failure_0, magic=0:1;27:4106:0:48387f7e-8de3-4eed-bab6-3db414426226, cib=0.36.21) : Event failed Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_start_0 (27) confirmed on nbgprepdb6 (rc=4) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: warning: update_failcount: Updating failcount for pgsql on nbgprepdb6 after failed start: rc=1 (update=INFINITY, time=1452765476) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: process_graph_event: Detected action (4106.27) pgsql_start_0.175=unknown error: failed Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 68: notify pgsql_post_notify_start_0 on nbgprepdb5 Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 69: notify pgsql_post_notify_start_0 on nbgprepdb6 Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/crmd/37, version=0.36.21) Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='fail-count-pgsql']: No such device or address (rc=-6, origin=local/attrd/119, version=0.36.21) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=0, node=nbgprepdb6, tag=nvpair, id=status-739251316-fail-count-pgsql, name=fail-count-pgsql, value=INFINITY, magic=NA, cib=0.36.22) : Transient attribute: update Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/128, version=0.36.22) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=0, node=nbgprepdb6, tag=nvpair, id=status-739251316-last-failure-pgsql, name=last-failure-pgsql, value=1452765476, magic=NA, cib=0.36.23) : Transient attribute: update Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/130, version=0.36.23) Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/132, version=0.36.23) Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/134, version=0.36.23) Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='last-failure-pgsql']: No such device or address (rc=-6, origin=local/attrd/120, version=0.36.23) Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='fail-count-pgsql']: No such device or address (rc=-6, origin=local/attrd/121, version=0.36.23) Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='last-failure-pgsql']: No such device or address (rc=-6, origin=local/attrd/122, version=0.36.23) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (68) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (69) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: notice: run_graph: Transition 4106 (Complete=24, Pending=0, Fired=0, Skipped=14, Incomplete=5, Source=/var/lib/pacemaker/pengine/pe-input-105.bz2): Stopped Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/4246, version=0.36.23) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepwitness56 is online Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb5 is online Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb6 is online Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: warning: unpack_rsc_op: Processing failed op start for pgsql:1 on nbgprepdb6: unknown error (1) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: native_print: ClusterMon (ocf::pacemaker:ClusterMon): Started nbgprepwitness56 Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: native_print: vip-slave (ocf::heartbeat:IPaddr2): Stopped Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: group_print: Resource Group: master-group Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: native_print: vip-master (ocf::heartbeat:IPaddr2): Stopped Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: native_print: vip-rep (ocf::heartbeat:IPaddr2): Stopped Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: clone_print: Master/Slave Set: msPostgresql [pgsql] Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: native_print: pgsql (ocf::heartbeat:pgsql): Slave nbgprepdb6 FAILED Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: short_print: Slaves: [ nbgprepdb5 ] Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: short_print: Stopped: [ nbgprepwitness56 ] Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: clone_print: Clone Set: clnPingCheck [pingCheck] Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: short_print: Started: [ nbgprepdb5 nbgprepdb6 nbgprepwitness56 ] Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:0 has failed INFINITY times on nbgprepdb6 Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: warning: common_apply_stickiness: Forcing msPostgresql away from nbgprepdb6 after 1000000 failures (max=1000000) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:1 has failed INFINITY times on nbgprepdb6 Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: warning: common_apply_stickiness: Forcing msPostgresql away from nbgprepdb6 after 1000000 failures (max=1000000) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: native_color: Resource vip-slave cannot run anywhere Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: rsc_merge_weights: pgsql:1: Rolling back scores from vip-master Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: native_color: Resource pgsql:1 cannot run anywhere Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: master_color: Promoting pgsql:0 (Slave nbgprepdb5) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: master_color: msPostgresql: Promoted 1 instances of a possible 1 to master Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for vip-master on nbgprepdb5 Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for vip-rep on nbgprepdb5 Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:0 on nbgprepdb5 Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:0 on nbgprepdb5 Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: LogActions: Leave ClusterMon (Started nbgprepwitness56) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: LogActions: Leave vip-slave (Stopped) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: notice: LogActions: Start vip-master (nbgprepdb5) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: notice: LogActions: Start vip-rep (nbgprepdb5) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: notice: LogActions: Promote pgsql:0 (Slave -> Master nbgprepdb5) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: notice: LogActions: Stop pgsql:1 (nbgprepdb6) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:0 (Started nbgprepwitness56) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:1 (Started nbgprepdb5) Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:2 (Started nbgprepdb6) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: do_te_invoke: Processing graph 4107 (ref=pe_calc-dc-1452765476-4340) derived from /var/lib/pacemaker/pengine/pe-input-106.bz2 Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 66: notify pgsql_pre_notify_stop_0 on nbgprepdb5 Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 68: notify pgsql_pre_notify_stop_0 on nbgprepdb6 Jan 14 10:57:56 [31406] nbgprepwitness56 pengine: notice: process_pe_message: Calculated Transition 4107: /var/lib/pacemaker/pengine/pe-input-106.bz2 Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (68) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (66) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 5: stop pgsql_stop_0 on nbgprepdb6 Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_stop_0 (5) confirmed on nbgprepdb6 (rc=0) Jan 14 10:57:56 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 67: notify pgsql_post_notify_stop_0 on nbgprepdb5 Jan 14 10:57:56 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/crmd/38, version=0.36.24) Jan 14 10:57:57 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (67) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:57 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 69: notify pgsql_pre_notify_promote_0 on nbgprepdb5 Jan 14 10:57:57 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (69) confirmed on nbgprepdb5 (rc=0) Jan 14 10:57:57 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 23: promote pgsql_promote_0 on nbgprepdb5 Jan 14 10:57:57 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=0, node=, tag=diff, id=(null), magic=NA, cib=0.37.1) : Non-status change Jan 14 10:57:57 [31402] nbgprepwitness56 cib: notice: cib:diff: Diff: --- 0.36.24 Jan 14 10:57:57 [31402] nbgprepwitness56 cib: notice: cib:diff: Diff: +++ 0.37.1 f513cdbbfcc6512ae86fa52fd8317062 Jan 14 10:57:57 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=nbgprepdb5/crm_attribute/4, version=0.37.1) Jan 14 10:57:57 [31402] nbgprepwitness56 cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-39.raw Jan 14 10:57:57 [31402] nbgprepwitness56 cib: info: write_cib_contents: Wrote version 0.37.0 of the CIB to disk (digest: 519cff930134aac8abef98a7be07e5c4) Jan 14 10:57:57 [31402] nbgprepwitness56 cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.lT09vB (digest: /var/lib/pacemaker/cib/cib.fYHwfu) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=0, node=, tag=diff, id=(null), magic=NA, cib=0.38.1) : Non-status change Jan 14 10:58:02 [31402] nbgprepwitness56 cib: notice: log_cib_diff: cib:diff: Local-only Change: 0.38.1 Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=nbgprepdb5/crm_attribute/4, version=0.38.1) Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-40.raw Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: write_cib_contents: Wrote version 0.38.0 of the CIB to disk (digest: 82486246c3f72ea5d30cea540a0c849b) Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.owOveL (digest: /var/lib/pacemaker/cib/cib.oT1X4Q) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=0, node=nbgprepdb5, tag=nvpair, id=status-739251315-master-pgsql, name=master-pgsql, value=1000, magic=NA, cib=0.38.2) : Transient attribute: update Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/attrd/40, version=0.38.2) Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/136, version=0.38.2) Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='master-pgsql']: No such device or address (rc=-6, origin=local/attrd/123, version=0.38.2) Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='pgsql-status']: No such device or address (rc=-6, origin=local/attrd/124, version=0.38.2) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=0, node=nbgprepdb5, tag=nvpair, id=status-739251315-pgsql-status, name=pgsql-status, value=PRI, magic=NA, cib=0.38.3) : Transient attribute: update Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/attrd/40, version=0.38.2) Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/136, version=0.38.2) Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='master-pgsql']: No such device or address (rc=-6, origin=local/attrd/123, version=0.38.2) Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='739251256']//transient_attributes//nvpair[@name='pgsql-status']: No such device or address (rc=-6, origin=local/attrd/124, version=0.38.2) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: abort_transition_graph: te_update_diff:172 - Triggered transition abort (complete=0, node=nbgprepdb5, tag=nvpair, id=status-739251315-pgsql-status, name=pgsql-status, value=PRI, magic=NA, cib=0.38.3) : Transient attribute: update Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/attrd/42, version=0.38.3) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_promote_0 (23) confirmed on nbgprepdb5 (rc=0) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 70: notify pgsql_post_notify_promote_0 on nbgprepdb5 Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/24, version=0.38.4) Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb6/attrd/138, version=0.38.4) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_notify_0 (70) confirmed on nbgprepdb5 (rc=0) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: notice: run_graph: Transition 4107 (Complete=20, Pending=0, Fired=0, Skipped=7, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-106.bz2): Stopped Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/4247, version=0.38.4) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepwitness56 is online Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb5 is online Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: determine_online_status: Node nbgprepdb6 is online Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: warning: unpack_rsc_op: Processing failed op start for pgsql:1 on nbgprepdb6: unknown error (1) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: native_print: ClusterMon (ocf::pacemaker:ClusterMon): Started nbgprepwitness56 Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: native_print: vip-slave (ocf::heartbeat:IPaddr2): Stopped Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: group_print: Resource Group: master-group Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: native_print: vip-master (ocf::heartbeat:IPaddr2): Stopped Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: native_print: vip-rep (ocf::heartbeat:IPaddr2): Stopped Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: clone_print: Master/Slave Set: msPostgresql [pgsql] Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: short_print: Masters: [ nbgprepdb5 ] Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: short_print: Stopped: [ nbgprepdb6 nbgprepwitness56 ] Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: clone_print: Clone Set: clnPingCheck [pingCheck] Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: short_print: Started: [ nbgprepdb5 nbgprepdb6 nbgprepwitness56 ] Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:0 has failed INFINITY times on nbgprepdb6 Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: warning: common_apply_stickiness: Forcing msPostgresql away from nbgprepdb6 after 1000000 failures (max=1000000) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: get_failcount_full: pgsql:1 has failed INFINITY times on nbgprepdb6 Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: warning: common_apply_stickiness: Forcing msPostgresql away from nbgprepdb6 after 1000000 failures (max=1000000) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: rsc_merge_weights: pgsql:1: Rolling back scores from vip-master Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: native_color: Resource pgsql:1 cannot run anywhere Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: master_color: Promoting pgsql:0 (Master nbgprepdb5) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: master_color: msPostgresql: Promoted 1 instances of a possible 1 to master Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for vip-slave on nbgprepdb5 Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for vip-master on nbgprepdb5 Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for vip-rep on nbgprepdb5 Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:0 on nbgprepdb5 Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: RecurringOp: Start recurring monitor (10s) for pgsql:0 on nbgprepdb5 Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: LogActions: Leave ClusterMon (Started nbgprepwitness56) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: notice: LogActions: Start vip-slave (nbgprepdb5) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: notice: LogActions: Start vip-master (nbgprepdb5) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: notice: LogActions: Start vip-rep (nbgprepdb5) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pgsql:0 (Master nbgprepdb5) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pgsql:1 (Stopped) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:0 (Started nbgprepwitness56) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:1 (Started nbgprepdb5) Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: info: LogActions: Leave pingCheck:2 (Started nbgprepdb6) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: do_te_invoke: Processing graph 4108 (ref=pe_calc-dc-1452765482-4348) derived from /var/lib/pacemaker/pengine/pe-input-107.bz2 Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 12: start vip-slave_start_0 on nbgprepdb5 Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 14: start vip-master_start_0 on nbgprepdb5 Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 26: monitor pgsql_monitor_10000 on nbgprepdb5 Jan 14 10:58:02 [31406] nbgprepwitness56 pengine: notice: process_pe_message: Calculated Transition 4108: /var/lib/pacemaker/pengine/pe-input-107.bz2 Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action vip-master_start_0 (14) confirmed on nbgprepdb5 (rc=0) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 15: monitor vip-master_monitor_10000 on nbgprepdb5 Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 16: start vip-rep_start_0 on nbgprepdb5 Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/25, version=0.38.5) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action vip-slave_start_0 (12) confirmed on nbgprepdb5 (rc=0) Jan 14 10:58:02 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 13: monitor vip-slave_monitor_10000 on nbgprepdb5 Jan 14 10:58:02 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/26, version=0.38.6) Jan 14 10:58:03 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action vip-slave_monitor_10000 (13) confirmed on nbgprepdb5 (rc=0) Jan 14 10:58:03 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/27, version=0.38.7) Jan 14 10:58:03 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action vip-master_monitor_10000 (15) confirmed on nbgprepdb5 (rc=0) Jan 14 10:58:03 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/28, version=0.38.8) Jan 14 10:58:03 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action vip-rep_start_0 (16) confirmed on nbgprepdb5 (rc=0) Jan 14 10:58:03 [31407] nbgprepwitness56 crmd: notice: te_rsc_command: Initiating action 17: monitor vip-rep_monitor_10000 on nbgprepdb5 Jan 14 10:58:03 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/29, version=0.38.9) Jan 14 10:58:03 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action vip-rep_monitor_10000 (17) confirmed on nbgprepdb5 (rc=0) Jan 14 10:58:03 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/30, version=0.38.10) Jan 14 10:58:03 [31407] nbgprepwitness56 crmd: info: match_graph_event: Action pgsql_monitor_10000 (26) confirmed on nbgprepdb5 (rc=0) Jan 14 10:58:03 [31407] nbgprepwitness56 crmd: notice: run_graph: Transition 4108 (Complete=9, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-107.bz2): Complete Jan 14 10:58:03 [31407] nbgprepwitness56 crmd: info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE Jan 14 10:58:03 [31407] nbgprepwitness56 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 14 10:58:03 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=nbgprepdb5/crmd/31, version=0.38.11) Jan 14 10:59:59 [31402] nbgprepwitness56 cib: info: crm_client_new: Connecting 0x7ff411d07d30 for uid=0 gid=0 pid=25945 id=bddc6ba6-79db-4c96-a17b-d14877877173 Jan 14 10:59:59 [31402] nbgprepwitness56 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.38.11) Jan 14 10:59:59 [31402] nbgprepwitness56 cib: info: crm_client_destroy: Destroying 0 events