[ClusterLabs] Fwd: Postgres pacemaker cluster failure
Danka Ivanović
danka.ivanovic at gmail.com
Thu Apr 25 12:57:55 EDT 2019
Hi,
Here are the logs when pacemaker fails to start postgres service on master.
It manage to start only postgres slave.
I tried different configuration with pgslqms and pgsql resource agents.
Those errors are when I use pgsqlms agent, which configuration I have sent
in first mail:
Apr 25 16:40:23 [4213] master lrmd: info: log_execute: executing
- rsc:PGSQL action:start call_id:51
launching as "postgres" command "/usr/lib/postgresql/9.5/bin/pg_ctl
--pgdata /var/lib/postgresql/9.5/main -w --timeout 120 start -o -c
config_file=/etc/postgresql/9.5/main/postgresql.conf"
Apr 25 16:40:24 [4211] master cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='PGSQL']/lrm_rsc_op[@id='PGSQL_last_0']:
@operation_key=PGSQL_start_0, @operation=start,
@transition-key=12:30:0:078c2b66-b095-49c4-947b-2427dd7852bf,
@transition-magic=0:0;12:30:0:078c2b66-b095-49c4-947b-2427dd7852bf,
@call-id=176, @rc-code=0, @exec-time=1146, @queue-time=0
Apr 25 16:40:53 [4216] master crmd: debug: crm_timer_start: Started
Shutdown Escalation (I_STOP:1200000ms), src=53
Apr 25 16:41:23 [4213] master lrmd: warning:
child_timeout_callback: PGSQL_start_0
process (PID 5986) timed out
Part of the log is attached.
On Tue, 23 Apr 2019 at 17:28, Danka Ivanović <danka.ivanovic at gmail.com>
wrote:
> Hi,
> It seems that ldap timeout caused cluster failure. Cluster is checking
> status every 15s on master and 16s on slave. Cluster needs postgres user
> for authentication, but ldap first query user on ldap server and then
> localy on host. When connection to ldap server was interrupted, cluster
> couldn't find postgres user and authenticate on db to check state. Problem
> is solved with reconfiguring /etc/ldap.conf and /etc/nslcd.conf. Following
> variable is added: nss_initgroups_ignoreusers with specified local users
> which should be ignored when querying ldap server. Thanks for your help. :)
> Another problem is that I cannot start postgres master with pacemaker.
> When I start postgres manually (with systemd) and then start pacemaker on
> slave, pacemaker is able to recognize master and start slave and failover
> works.
> That is another problem which I didn't manage to solve. Should I send a
> new mail for that issue or we can continue in this thread?
>
> On Fri, 19 Apr 2019 at 19:19, Jehan-Guillaume de Rorthais <jgdr at dalibo.com>
> wrote:
>
>> On Fri, 19 Apr 2019 17:26:14 +0200
>> Danka Ivanović <danka.ivanovic at gmail.com> wrote:
>> ...
>> > Should I change any of those timeout parameters in order to avoid
>> timeout?
>>
>> You can try to raise the timeout, indeed. But as far as we don't know
>> **why**
>> your VMs froze for some time, it is difficult to guess how high should be
>> these timeouts.
>>
>> Not to mention that it will raise your RTO.
>>
>
>
> --
> Pozdrav
> Danka Ivanovic
>
--
Pozdrav
Danka Ivanovic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20190425/c78b14ad/attachment-0001.html>
-------------- next part --------------
Apr 25 16:39:50 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444e8e80 for uid=0 gid=0 pid=5791 id=c93d535d-77d8-4556-9a63-d9a1c2b45de9
Apr 25 16:39:50 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5791-13)
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5791]
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5791-13)
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5791-13) state:2
Apr 25 16:39:50 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5791-13-header
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5791-13-header
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5791-13-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5837-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5837]
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5837-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5837-24) state:2
Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9)
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-15544-5837-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-15544-5837-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-15544-5837-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5837-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5837]
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5837-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5837-24) state:2
Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9)
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-15544-5837-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-15544-5837-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-15544-5837-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5837-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5837]
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5837-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5837-24) state:2
Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9)
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-15544-5837-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-15544-5837-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-15544-5837-24-header
Apr 25 16:39:50 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444e8e80 for uid=0 gid=0 pid=5841 id=92b5b3a5-7575-4ed9-a26b-5877b2ab1bab
Apr 25 16:39:50 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5841-13)
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5841]
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5841-13)
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5841-13) state:2
Apr 25 16:39:50 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5841-13-header
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5841-13-header
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5841-13-header
Apr 25 16:39:50 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444e8e80 for uid=0 gid=0 pid=5847 id=6f50e4fb-5cb0-43c9-98f8-15c3cacc7102
Apr 25 16:39:50 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5847-13)
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5847]
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to master (origin=local/cibadmin/2)
Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:39:50 [4211] master cib: debug: xml_patch_version_check: Can apply patch 5.758.3 to 5.758.2
Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: --- 5.758.2 2
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: +++ 5.759.0 (null)
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: + /cib: @epoch=759, @num_updates=0
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ /cib/configuration/resources: <primitive id="PGSQL" class="ocf" provider="heartbeat" type="pgsqlms"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <instance_attributes id="PGSQL-instance_attributes">
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="pgdata" value="/var/lib/postgresql/9.5/main" id="PGSQL-instance_attributes-pgdata"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="bindir" value="/usr/lib/postgresql/9.5/bin" id="PGSQL-instance_attributes-bindir"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="pghost" value="/var/run/postgresql/" id="PGSQL-instance_attributes-pghost"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="recovery_template" value="/etc/postgresql/9.5/main/recovery.conf.pcmk" id="PGSQL-instance_attributes-recovery_template"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <nvpair name="start_opts" value="-c config_file=/etc/postgresql/9.5/main/postgresql.conf" id="PGSQL-instance_attributes-start_opts"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </instance_attributes>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <operations>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="start" timeout="60s" interval="0" id="PGSQL-start-0"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="stop" timeout="60s" interval="0" id="PGSQL-stop-0"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="promote" timeout="15s" interval="0" id="PGSQL-promote-0"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="demote" timeout="120s" interval="0" id="PGSQL-demote-0"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="monitor" interval="15s" timeout="10s" role="Master" id="PGSQL-monitor-15s"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="monitor" interval="16s" timeout="10s" role="Slave" id="PGSQL-monitor-16s"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <op name="notify" timeout="60" interval="0" id="PGSQL-notify-0"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </operations>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </primitive>
Apr 25 16:39:50 [4211] master cib: debug: activateCibXml: Triggering CIB write for cib_apply_diff op
Apr 25 16:39:50 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.759.0 to 5.758.2
Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: STONITH timeout: 60000
Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: STONITH of failed nodes is enabled
Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: Stop all active resources: false
Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default
Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: Default stickiness: 0
Apr 25 16:39:50 [4212] master stonith-ng: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 25 16:39:50 [4212] master stonith-ng: debug: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Apr 25 16:39:50 [4212] master stonith-ng: debug: cib_device_update: Device fencing-test-rsyslog is allowed on master: score=0
Apr 25 16:39:50 [4212] master stonith-ng: debug: stonith_device_register: Device 'fencing-test-rsyslog' already existed in device list (1 active devices)
Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=master/cibadmin/2, version=5.759.0)
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5847-13)
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5847-13) state:2
Apr 25 16:39:50 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5847-13-header
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5847-13-header
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5847-13-header
Apr 25 16:39:50 [4211] master cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-11.raw
Apr 25 16:39:50 [4211] master cib: debug: cib_file_write_with_digest: Writing CIB to disk
Apr 25 16:39:50 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_info operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=0, reply=0, notify=0, exit=-439087056
Apr 25 16:39:50 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_register operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=0, reply=1, notify=1, exit=-439087056
Apr 25 16:39:50 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_info operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=0, reply=0, notify=0, exit=-439087056
Apr 25 16:39:50 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_exec operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=43, reply=1, notify=0, exit=-439087056
Apr 25 16:39:50 [4213] master lrmd: debug: log_execute: executing - rsc:PGSQL action:monitor call_id:43
Apr 25 16:39:50 [4211] master cib: info: cib_file_write_with_digest: Wrote version 5.759.0 of the CIB to disk (digest: 16f02c16f467c9ef3114138a3febd8c5)
Apr 25 16:39:50 [4211] master cib: debug: cib_file_write_with_digest: Wrote digest 16f02c16f467c9ef3114138a3febd8c5 to disk
Apr 25 16:39:50 [4211] master cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.G7Oz1H (digest: /var/lib/pacemaker/cib/cib.VFtf2I)
Apr 25 16:39:50 [4211] master cib: debug: cib_file_write_with_digest: Activating /var/lib/pacemaker/cib/cib.G7Oz1H
Apr 25 16:39:50 [5856] master crm_node: info: get_cluster_type: Verifying cluster type: 'corosync'
Apr 25 16:39:50 [5856] master crm_node: info: get_cluster_type: Assuming an active 'corosync' cluster
Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5856-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5856]
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5856] master crm_node: debug: qb_ipcc_disconnect: qb_ipcc_disconnect()
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-request-15544-5856-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5856-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5856-24) state:2
Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9)
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-response-15544-5856-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-15544-5856-24-header
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-event-15544-5856-24-header
Apr 25 16:39:50 [5856] master crm_node: debug: get_local_nodeid: Local nodeid is 1
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-15544-5856-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-15544-5856-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5856-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5856]
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5856] master crm_node: debug: qb_ipcc_disconnect: qb_ipcc_disconnect()
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-request-15544-5856-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5856-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5856-24) state:2
Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9)
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-15544-5856-24-header
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-response-15544-5856-24-header
Apr 25 16:39:50 [5856] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-event-15544-5856-24-header
Apr 25 16:39:50 [5856] master crm_node: info: crm_xml_cleanup: Cleaning up memory from libxml2
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-15544-5856-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-15544-5856-24-header
Apr 25 16:39:50 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5858 id=97335234-7d26-4466-a47a-fb474c95ac16
Apr 25 16:39:50 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5858-13)
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5858]
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:39:50 [5858] master crm_resource: debug: cib_native_signon_raw: Connection to CIB successful
Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: --- 5.759.0 2
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: +++ 5.759.1 (null)
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: + /cib: @num_updates=1
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources: <lrm_resource id="PGSQL" type="pgsqlms" class="ocf" provider="heartbeat"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <lrm_rsc_op id="PGSQL_last_failure_0" operation_key="PGSQL_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="5:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" transition-magic="0:5;5:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" exit-reason="You must set meta parameter notify=true for your master resou
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <lrm_rsc_op id="PGSQL_last_0" operation_key="PGSQL_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="5:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" transition-magic="0:5;5:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" exit-reason="You must set meta parameter notify=true for your master resource" on_
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </lrm_resource>
Apr 25 16:39:50 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.759.1 to 5.759.0
Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=secondary/crmd/277, version=5.759.1)
Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: STONITH timeout: 60000
Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: STONITH of failed nodes is enabled
Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: Stop all active resources: false
Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default
Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: Default stickiness: 0
Apr 25 16:39:50 [5858] master crm_resource: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 25 16:39:50 [5858] master crm_resource: debug: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Apr 25 16:39:50 [5858] master crm_resource: info: determine_online_status_fencing: Node master is active
Apr 25 16:39:50 [5858] master crm_resource: info: determine_online_status: Node master is online
Apr 25 16:39:50 [5858] master crm_resource: debug: populate_hash: Overwriting value of master-PGSQL: -1000 -> -INFINITY
Apr 25 16:39:50 [5858] master crm_resource: info: determine_online_status_fencing: Node secondary is active
Apr 25 16:39:50 [5858] master crm_resource: info: determine_online_status: Node secondary is online
Apr 25 16:39:50 [5858] master crm_resource: debug: determine_op_status: AWSVIP_monitor_0 on master returned 'ok' (0) instead of the expected value: 'not running' (7)
Apr 25 16:39:50 [5858] master crm_resource: info: determine_op_status: Operation monitor found resource AWSVIP active on master
Apr 25 16:39:50 [5858] master crm_resource: debug: determine_op_status: AWSVIP_monitor_0 on master returned 'ok' (0) instead of the expected value: 'not running' (7)
Apr 25 16:39:50 [5858] master crm_resource: info: determine_op_status: Operation monitor found resource AWSVIP active on master
Apr 25 16:39:50 [5858] master crm_resource: debug: cli_resource_print_attribute: Looking up notify in PGSQL
Apr 25 16:39:50 [5858] master crm_resource: debug: cib_native_signoff: Signing out of the CIB Service
Apr 25 16:39:50 [5858] master crm_resource: debug: qb_ipcc_disconnect: qb_ipcc_disconnect()
Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5858-13-header
Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5858-13-header
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5858-13)
Apr 25 16:39:50 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5858-13) state:2
Apr 25 16:39:50 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5858-13-header
Apr 25 16:39:50 [5858] master crm_resource: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5858-13-header
Apr 25 16:39:50 [5858] master crm_resource: info: crm_xml_cleanup: Cleaning up memory from libxml2
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5858-13-header
Apr 25 16:39:50 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5858-13-header
pgsqlms(PGSQL)[5849]: 2019/04/25_16:39:50 ERROR: You must set meta parameter notify=true for your master resource
Apr 25 16:39:50 [4213] master lrmd: debug: operation_finished: PGSQL_monitor_0:5849 - exited with rc=5
Apr 25 16:39:50 [4213] master lrmd: notice: operation_finished: PGSQL_monitor_0:5849:stderr [ ocf-exit-reason:You must set meta parameter notify=true for your master resource ]
Apr 25 16:39:50 [4213] master lrmd: debug: operation_finished: PGSQL_monitor_0:5849:stdout [ -- empty -- ]
Apr 25 16:39:50 [4213] master lrmd: debug: log_finished: finished - rsc:PGSQL action:monitor call_id:43 pid:5849 exit-code:5 exec-time:91ms queue-time:0ms
Apr 25 16:39:50 [4216] master crmd: debug: create_operation_update: do_update_resource: Updating resource PGSQL after monitor op complete (interval=0)
Apr 25 16:39:50 [5866] master crm_node: info: get_cluster_type: Verifying cluster type: 'corosync'
Apr 25 16:39:50 [5866] master crm_node: info: get_cluster_type: Assuming an active 'corosync' cluster
Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5866-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5866]
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5866] master crm_node: debug: qb_ipcc_disconnect: qb_ipcc_disconnect()
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-request-15544-5866-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5866-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5866-24) state:2
Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9)
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-response-15544-5866-24-header
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-response-15544-5866-24-header
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cpg-event-15544-5866-24-header
Apr 25 16:39:50 [5866] master crm_node: debug: get_local_nodeid: Local nodeid is 1
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-event-15544-5866-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cpg-request-15544-5866-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] IPC credentials authenticated (15544-5866-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] connecting to client [5866]
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [15544] master corosync debug [QB ] shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_open_2: shm size:1048589; real_size:1052672; rb->word_size:263168
Apr 25 16:39:50 [5866] master crm_node: debug: qb_ipcc_disconnect: qb_ipcc_disconnect()
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-request-15544-5866-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] HUP conn (15544-5866-24)
Apr 25 16:39:50 [15544] master corosync debug [QB ] qb_ipcs_disconnect(15544-5866-24) state:2
Apr 25 16:39:50 [15544] master corosync debug [QB ] epoll_ctl(del): Bad file descriptor (9)
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-15544-5866-24-header
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-response-15544-5866-24-header
Apr 25 16:39:50 [5866] master crm_node: debug: qb_rb_close: Closing ringbuffer: /dev/shm/qb-cmap-event-15544-5866-24-header
Apr 25 16:39:50 [5866] master crm_node: info: crm_xml_cleanup: Cleaning up memory from libxml2
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-15544-5866-24-header
Apr 25 16:39:50 [15544] master corosync debug [QB ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-15544-5866-24-header
Apr 25 16:39:50 [4216] master crmd: info: action_synced_wait: Managed pgsqlms_meta-data_0 process 5859 exited with rc=0
Apr 25 16:39:50 [4216] master crmd: notice: process_lrm_event: Operation PGSQL_monitor_0: not installed (node=master, call=43, rc=5, cib-update=40, confirmed=true)
Apr 25 16:39:50 [4216] master crmd: notice: process_lrm_event: master-PGSQL_monitor_0:43 [ ocf-exit-reason:You must set meta parameter notify=true for your master resource\n ]
Apr 25 16:39:50 [4216] master crmd: debug: update_history_cache: Updating history for 'PGSQL' with monitor op
Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/40)
Apr 25 16:39:50 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: --- 5.759.1 2
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: Diff: +++ 5.759.2 (null)
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: + /cib: @num_updates=2
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources: <lrm_resource id="PGSQL" type="pgsqlms" class="ocf" provider="heartbeat"/>
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <lrm_rsc_op id="PGSQL_last_failure_0" operation_key="PGSQL_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="4:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" transition-magic="0:5;4:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" exit-reason="You must set meta parameter notify=true for your master resou
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ <lrm_rsc_op id="PGSQL_last_0" operation_key="PGSQL_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="4:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" transition-magic="0:5;4:25:7:078c2b66-b095-49c4-947b-2427dd7852bf" exit-reason="You must set meta parameter notify=true for your master resource" on_
Apr 25 16:39:50 [4211] master cib: info: cib_perform_op: ++ </lrm_resource>
Apr 25 16:39:50 [4211] master cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=master/crmd/40, version=5.759.2)
Apr 25 16:39:50 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.759.2 to 5.759.1
Apr 25 16:39:55 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:03 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444eda80 for uid=0 gid=0 pid=5871 id=77319e05-d144-4d61-a6a1-7b02ce9d7eaa
Apr 25 16:40:03 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5871-13)
Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5871]
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:03 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5871-13)
Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5871-13) state:2
Apr 25 16:40:03 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5871-13-header
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5871-13-header
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5871-13-header
Apr 25 16:40:03 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444eda80 for uid=0 gid=0 pid=5881 id=c3aae380-3d8d-4800-a167-7d0f676010c8
Apr 25 16:40:03 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5881-13)
Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5881]
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:03 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5881-13)
Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5881-13) state:2
Apr 25 16:40:03 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5881-13-header
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5881-13-header
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5881-13-header
Apr 25 16:40:03 [4211] master cib: debug: crm_client_new: Connecting 0x55d8444eda80 for uid=0 gid=0 pid=5887 id=eb5ba3d7-d898-4a55-988a-ba46894ff6b8
Apr 25 16:40:03 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5887-13)
Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5887]
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:03 [4211] master cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to master (origin=local/cibadmin/2)
Apr 25 16:40:03 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:03 [4211] master cib: debug: xml_patch_version_check: Can apply patch 5.759.3 to 5.759.2
Apr 25 16:40:03 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: Diff: --- 5.759.2 2
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: Diff: +++ 5.760.0 771ea8eb8446b4c081f2915124d06d63
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: -- /cib/configuration/resources/primitive[@id='PGSQL']
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: + /cib: @epoch=760, @num_updates=0
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ /cib/configuration/resources: <master id="PGSQL-HA"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <meta_attributes id="PGSQL-HA-meta_attributes">
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="master-max" value="1" id="PGSQL-HA-meta_attributes-master-max"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="master-node-max" value="1" id="PGSQL-HA-meta_attributes-master-node-max"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="clone-max" value="2" id="PGSQL-HA-meta_attributes-clone-max"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="clone-node-max" value="1" id="PGSQL-HA-meta_attributes-clone-node-max"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="notify" value="true" id="PGSQL-HA-meta_attributes-notify"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="interleave" value="true" id="PGSQL-HA-meta_attributes-interleave"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </meta_attributes>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <primitive id="PGSQL" class="ocf" provider="heartbeat" type="pgsqlms">
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <instance_attributes id="PGSQL-instance_attributes">
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="pgdata" value="/var/lib/postgresql/9.5/main" id="PGSQL-instance_attributes-pgdata"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="bindir" value="/usr/lib/postgresql/9.5/bin" id="PGSQL-instance_attributes-bindir"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="pghost" value="/var/run/postgresql/" id="PGSQL-instance_attributes-pghost"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="recovery_template" value="/etc/postgresql/9.5/main/recovery.conf.pcmk" id="PGSQL-instance_attributes-recovery_template"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <nvpair name="start_opts" value="-c config_file=/etc/postgresql/9.5/main/postgresql.conf" id="PGSQL-instance_attributes-start_opts"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </instance_attributes>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <operations>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="start" timeout="60s" interval="0" id="PGSQL-start-0"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="stop" timeout="60s" interval="0" id="PGSQL-stop-0"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="promote" timeout="15s" interval="0" id="PGSQL-promote-0"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="demote" timeout="120s" interval="0" id="PGSQL-demote-0"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="monitor" interval="15s" timeout="10s" role="Master" id="PGSQL-monitor-15s"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="monitor" interval="16s" timeout="10s" role="Slave" id="PGSQL-monitor-16s"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ <op name="notify" timeout="60" interval="0" id="PGSQL-notify-0"/>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </operations>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </primitive>
Apr 25 16:40:03 [4211] master cib: info: cib_perform_op: ++ </master>
Apr 25 16:40:03 [4211] master cib: debug: activateCibXml: Triggering CIB write for cib_apply_diff op
Apr 25 16:40:03 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.760.0 to 5.759.2
Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: STONITH timeout: 60000
Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: STONITH of failed nodes is enabled
Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: Stop all active resources: false
Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: Cluster is symmetric - resources can run anywhere by default
Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: Default stickiness: 0
Apr 25 16:40:03 [4212] master stonith-ng: notice: unpack_config: On loss of CCM Quorum: Ignore
Apr 25 16:40:03 [4212] master stonith-ng: debug: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0
Apr 25 16:40:03 [4212] master stonith-ng: debug: cib_device_update: Device fencing-test-rsyslog is allowed on master: score=0
Apr 25 16:40:03 [4212] master stonith-ng: debug: stonith_device_register: Device 'fencing-test-rsyslog' already existed in device list (1 active devices)
Apr 25 16:40:03 [4211] master cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=master/cibadmin/2, version=5.760.0)
Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5887-13)
Apr 25 16:40:03 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5887-13) state:2
Apr 25 16:40:03 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5887-13-header
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5887-13-header
Apr 25 16:40:03 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5887-13-header
Apr 25 16:40:03 [4211] master cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-12.raw
Apr 25 16:40:03 [4211] master cib: debug: cib_file_write_with_digest: Writing CIB to disk
Apr 25 16:40:03 [4211] master cib: info: cib_file_write_with_digest: Wrote version 5.760.0 of the CIB to disk (digest: 85843593b18ef9bd7a6b33630ead2936)
Apr 25 16:40:03 [4211] master cib: debug: cib_file_write_with_digest: Wrote digest 85843593b18ef9bd7a6b33630ead2936 to disk
Apr 25 16:40:03 [4211] master cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.N7t074 (digest: /var/lib/pacemaker/cib/cib.FeFUqC)
Apr 25 16:40:03 [4211] master cib: debug: cib_file_write_with_digest: Activating /var/lib/pacemaker/cib/cib.N7t074
Apr 25 16:40:04 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5893 id=21032fff-7a5b-4e0f-86b9-f8cd87ffa267
Apr 25 16:40:04 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5893-13)
Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5893]
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:04 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5893-13)
Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5893-13) state:2
Apr 25 16:40:04 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5893-13-header
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5893-13-header
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5893-13-header
Apr 25 16:40:04 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5903 id=b145010e-eaf7-40f0-a699-b5a4a0dbdad1
Apr 25 16:40:04 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5903-13)
Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5903]
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:04 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5903-13)
Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5903-13) state:2
Apr 25 16:40:04 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5903-13-header
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5903-13-header
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5903-13-header
Apr 25 16:40:04 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5909 id=527ca328-9436-4720-b202-2a82d415cafb
Apr 25 16:40:04 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5909-13)
Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5909]
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:04 [4211] master cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to master (origin=local/cibadmin/2)
Apr 25 16:40:04 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:04 [4211] master cib: debug: xml_patch_version_check: Can apply patch 5.760.1 to 5.760.0
Apr 25 16:40:04 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:04 [4211] master cib: info: cib_perform_op: Diff: --- 5.760.0 2
Apr 25 16:40:04 [4211] master cib: info: cib_perform_op: Diff: +++ 5.761.0 (null)
Apr 25 16:40:04 [4211] master cib: info: cib_perform_op: + /cib: @epoch=761
Apr 25 16:40:04 [4211] master cib: info: cib_perform_op: ++ /cib/configuration/constraints: <rsc_colocation id="IPAWSIP-WITH-MASTER" score="INFINITY" rsc="AWSVIP" with-rsc="PGSQL-HA" with-rsc-role="Master"/>
Apr 25 16:40:04 [4211] master cib: debug: activateCibXml: Triggering CIB write for cib_apply_diff op
Apr 25 16:40:04 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.761.0 to 5.760.0
Apr 25 16:40:04 [4211] master cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=master/cibadmin/2, version=5.761.0)
Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5909-13)
Apr 25 16:40:04 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5909-13) state:2
Apr 25 16:40:04 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-response-4211-5909-13-header
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-event-4211-5909-13-header
Apr 25 16:40:04 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_rw-request-4211-5909-13-header
Apr 25 16:40:04 [4211] master cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-13.raw
Apr 25 16:40:04 [4211] master cib: debug: cib_file_write_with_digest: Writing CIB to disk
Apr 25 16:40:04 [4211] master cib: info: cib_file_write_with_digest: Wrote version 5.761.0 of the CIB to disk (digest: cbf371ff24027c60389dd1df088dda45)
Apr 25 16:40:04 [4216] master crmd: debug: do_lrm_rsc_op: Stopped 0 recurring operations in preparation for AWSVIP_stop_0
Apr 25 16:40:04 [4213] master lrmd: debug: process_lrmd_message: Processed lrmd_rsc_exec operation from a0d15606-efa1-42dc-8892-885bfe52b70f: rc=44, reply=1, notify=0, exit=-439087056
Apr 25 16:40:04 [4213] master lrmd: info: log_execute: executing - rsc:AWSVIP action:stop call_id:44
Apr 25 16:40:04 [4211] master cib: debug: cib_file_write_with_digest: Wrote digest cbf371ff24027c60389dd1df088dda45 to disk
Apr 25 16:40:04 [4211] master cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.8CQj3j (digest: /var/lib/pacemaker/cib/cib.IpBJgU)
Apr 25 16:40:04 [4211] master cib: debug: cib_file_write_with_digest: Activating /var/lib/pacemaker/cib/cib.8CQj3j
Apr 25 16:40:06 [4216] master crmd: debug: throttle_cib_load: cib load: 0.001000 (3 ticks in 30s)
Apr 25 16:40:06 [4216] master crmd: debug: throttle_load_avg: Current load is 0.040000 (full: 0.04 0.07 0.02 1/229 5948)
Apr 25 16:40:06 [4216] master crmd: debug: throttle_io_load: Current IO load is 0.000000
Apr 25 16:40:07 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5960 id=c11cc076-8c66-41f6-abdf-a980e63a8e1e
Apr 25 16:40:07 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5960-13)
Apr 25 16:40:07 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5960]
Apr 25 16:40:07 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:07 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:07 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:07 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:07 [4211] master cib: debug: qb_ipcs_dispatch_connection_request: HUP conn (4211-5960-13)
Apr 25 16:40:07 [4211] master cib: debug: qb_ipcs_disconnect: qb_ipcs_disconnect(4211-5960-13) state:2
Apr 25 16:40:07 [4211] master cib: debug: crm_client_destroy: Destroying 0 events
Apr 25 16:40:07 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_ro-response-4211-5960-13-header
Apr 25 16:40:07 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_ro-event-4211-5960-13-header
Apr 25 16:40:07 [4211] master cib: debug: qb_rb_close: Free'ing ringbuffer: /dev/shm/qb-cib_ro-request-4211-5960-13-header
Apr 25 16:40:09 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
awsvip(AWSVIP)[5911]: 2019/04/25_16:40:10 INFO: secondary_private_ip has been successfully brought down (10.x.x.x)
awsvip(AWSVIP)[5911]: 2019/04/25_16:40:10 DEBUG: AWSVIP stop : 0
Apr 25 16:40:10 [4213] master lrmd: debug: operation_finished: AWSVIP_stop_0:5911 - exited with rc=0
Apr 25 16:40:10 [4213] master lrmd: debug: operation_finished: AWSVIP_stop_0:5911:stderr [ -- empty -- ]
Apr 25 16:40:10 [4213] master lrmd: debug: operation_finished: AWSVIP_stop_0:5911:stdout [ -- empty -- ]
Apr 25 16:40:10 [4213] master lrmd: info: log_finished: finished - rsc:AWSVIP action:stop call_id:44 pid:5911 exit-code:0 exec-time:6302ms queue-time:0ms
Apr 25 16:40:10 [4216] master crmd: debug: create_operation_update: do_update_resource: Updating resource AWSVIP after stop op complete (interval=0)
Apr 25 16:40:10 [4216] master crmd: notice: process_lrm_event: Operation AWSVIP_stop_0: ok (node=master, call=44, rc=0, cib-update=41, confirmed=true)
Apr 25 16:40:10 [4216] master crmd: debug: update_history_cache: Updating history for 'AWSVIP' with stop op
Apr 25 16:40:10 [4211] master cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/41)
Apr 25 16:40:10 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:10 [4211] master cib: info: cib_perform_op: Diff: --- 5.761.0 2
Apr 25 16:40:10 [4211] master cib: info: cib_perform_op: Diff: +++ 5.761.1 (null)
Apr 25 16:40:10 [4211] master cib: info: cib_perform_op: + /cib: @num_updates=1
Apr 25 16:40:10 [4211] master cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='AWSVIP']/lrm_rsc_op[@id='AWSVIP_last_0']: @operation_key=AWSVIP_stop_0, @operation=stop, @transition-key=8:28:0:078c2b66-b095-49c4-947b-2427dd7852bf, @transition-magic=0:0;8:28:0:078c2b66-b095-49c4-947b-2427dd7852bf, @call-id=44, @last-run=1556210404, @last-rc-change=1556210404, @exec-time=6302
Apr 25 16:40:10 [4211] master cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=master/crmd/41, version=5.761.1)
Apr 25 16:40:10 [4212] master stonith-ng: debug: xml_patch_version_check: Can apply patch 5.761.1 to 5.761.0
Apr 25 16:40:15 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:23 [4211] master cib: debug: crm_client_new: Connecting 0x55d8443698c0 for uid=0 gid=0 pid=5971 id=6d820cd5-f375-4af0-b4a8-99ee56f25447
Apr 25 16:40:23 [4211] master cib: debug: handle_new_connection: IPC credentials authenticated (4211-5971-13)
Apr 25 16:40:23 [4211] master cib: debug: qb_ipcs_shm_connect: connecting to client [5971]
Apr 25 16:40:23 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:23 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:23 [4211] master cib: debug: qb_rb_open_2: shm size:524301; real_size:528384; rb->word_size:132096
Apr 25 16:40:23 [4211] master cib: debug: cib_acl_enabled: CIB ACL is disabled
Apr 25 16:40:23 [4216] master crmd: debug: crm_client_new: Connecting 0x559525e82730 for uid=0 gid=0 pid=5971 id=68c1b0cc-17a2-4013-924f-f29860582058
Apr 25 16:40:23 [4216] master crmd: debug: handle_new_connection: IPC credentials authenticated (4216-5971-13)
Apr 25 16:40:23 [4216] master crmd: debug: qb_ipcs_shm_connect: connecting to client [5971]
Apr 25 16:40:23 [4216] master crmd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792
Apr 25 16:40:23 [4216] master crmd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792
Apr 25 16:40:23 [4216] master crmd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792
Apr 25 16:40:23 [4214] master attrd: debug: crm_client_new: Connecting 0x55d565951a50 for uid=0 gid=0 pid=5971 id=fdb1c5ce-ecd1-4f8e-ac74-07b7d7989c29
Apr 25 16:40:23 [4214] master attrd: debug: handle_new_connection: IPC credentials authenticated (4214-5971-10)
Apr 25 16:40:23 [4214] master attrd: debug: qb_ipcs_shm_connect: connecting to client [5971]
Apr 25 16:40:23 [4214] master attrd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792
Apr 25 16:40:23 [4214] master attrd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792
Apr 25 16:40:23 [4214] master attrd: debug: qb_rb_open_2: shm size:131085; real_size:135168; rb->word_size:33792
Apr 25 16:40:23 [4214] master attrd: debug: attrd_client_update: Broadcasting fail-count-PGSQL[master] = (null)
Apr 25 16:40:23 [4211] master cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='master']//lrm_resource[@id='PGSQL'] to master (origin=local/crmd/42)
More information about the Users
mailing list