[Pacemaker] Corosync and Pacemaker Hangs

Vladislav Bogdanov bubble at hoster-ok.com
Mon Sep 15 03:55:10 UTC 2014


15.09.2014 04:24, Norbert Kiam Maclang wrote:
> Hi Vladislav and Andrew,
> 
> After adding fencing/stonith (resource level) and fencing handlers on
> drbd, I am not getting monitor timeouts on drbd but I am experiencing a
> different problem now. As per my understanding, logs on node01 showed
> that it detects node02 to be disconnected (and moved the resources to
> itself) but crm_mon shows that the resources are still started on node02
> which is not.

That is probably the root of your issues.
That _may_ be caused by the fact that VMs are not rescheduled to run on
host CPUs fair enough. You'd need either to re-think the whole
architecture of your cluster or to some-how tune your cluster messaging
layer to deal with that. Increasing 'totem.token' would be the first
step. See man corosync.conf and cman docs.

> 
> Node01:
> 
> node01 crmd[952]:   notice: do_state_transition: State transition S_IDLE
> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
> origin=crm_timer_popped ]
> node01 pengine[951]:   notice: unpack_config: On loss of CCM Quorum: Ignore
> node01 crmd[952]:   notice: run_graph: Transition 260 (Complete=0,
> Pending=0, Fired=0, Skipped=0, Incomplete=0,
> Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
> node01 crmd[952]:   notice: do_state_transition: State transition
> S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL
> origin=notify_crmd ]
> node01 pengine[951]:   notice: process_pe_message: Calculated Transition
> 260: /var/lib/pacemaker/pengine/pe-input-78.bz2
> node01 corosync[917]:   [TOTEM ] A processor failed, forming new
> configuration.
> node01 corosync[917]:   [TOTEM ] A new membership (10.2.131.20:352
> <http://10.2.131.20:352>) was formed. Members left: 167936789
> node01 crmd[952]:  warning: match_down_event: No match for shutdown
> action on 167936789
> node01 crmd[952]:   notice: peer_update_callback: Stonith/shutdown of
> node02 not matched
> node01 crmd[952]:   notice: do_state_transition: State transition S_IDLE
> -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
> origin=abort_transition_graph ]
> node01 pengine[951]:   notice: unpack_config: On loss of CCM Quorum: Ignore
> node01 pengine[951]:  warning: pe_fence_node: Node node02 will be fenced
> because our peer process is no longer available
> node01 pengine[951]:  warning: determine_online_status: Node node02 is
> unclean
> node01 pengine[951]:  warning: stage6: Scheduling Node node02 for STONITH
> node01 pengine[951]:   notice: LogActions: Move    fs_pg#011(Started
> node02 -> node01)
> node01 pengine[951]:   notice: LogActions: Move    ip_pg#011(Started
> node02 -> node01)
> node01 pengine[951]:   notice: LogActions: Move    lsb_pg#011(Started
> node02 -> node01)
> node01 pengine[951]:   notice: LogActions: Demote  drbd_pg:0#011(Master
> -> Stopped node02)
> node01 pengine[951]:   notice: LogActions: Promote drbd_pg:1#011(Slave
> -> Master node01)
> node01 pengine[951]:   notice: LogActions: Stop    p_fence:0#011(node02)
> node01 crmd[952]:   notice: te_rsc_command: Initiating action 2: cancel
> drbd_pg_cancel_31000 on node01 (local)
> node01 crmd[952]:   notice: te_fence_node: Executing reboot fencing
> operation (54) on node02 (timeout=60000)
> node01 stonith-ng[948]:   notice: handle_request: Client
> crmd.952.6d7ac808 wants to fence (reboot) 'node02' with device '(any)'
> node01 stonith-ng[948]:   notice: initiate_remote_stonith_op: Initiating
> remote operation reboot for node02: 96530c7b-1c80-42c4-82cf-840bf3d5bb5f (0)
> node01 crmd[952]:   notice: te_rsc_command: Initiating action 68: notify
> drbd_pg_pre_notify_demote_0 on node02
> node01 crmd[952]:   notice: te_rsc_command: Initiating action 70: notify
> drbd_pg_pre_notify_demote_0 on node01 (local)
> node01 pengine[951]:  warning: process_pe_message: Calculated Transition
> 261: /var/lib/pacemaker/pengine/pe-warn-0.bz2
> node01 crmd[952]:   notice: process_lrm_event: LRM operation
> drbd_pg_notify_0 (call=63, rc=0, cib-update=0, confirmed=true) ok
> node01 kernel: [230495.836024] d-con pg: PingAck did not arrive in time.
> node01 kernel: [230495.836176] d-con pg: peer( Primary -> Unknown )
> conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown ) 
> node01 kernel: [230495.837204] d-con pg: asender terminated
> node01 kernel: [230495.837216] d-con pg: Terminating drbd_a_pg
> node01 kernel: [230495.837286] d-con pg: Connection closed
> node01 kernel: [230495.837298] d-con pg: conn( NetworkFailure ->
> Unconnected ) 
> node01 kernel: [230495.837299] d-con pg: receiver terminated
> node01 kernel: [230495.837300] d-con pg: Restarting receiver thread
> node01 kernel: [230495.837304] d-con pg: receiver (re)started
> node01 kernel: [230495.837314] d-con pg: conn( Unconnected ->
> WFConnection ) 
> node01 crmd[952]:  warning: action_timer_callback: Timer popped
> (timeout=20000, abort_level=1000000, complete=false)
> node01 crmd[952]:    error: print_synapse: [Action    2]: Completed rsc
> op drbd_pg_cancel_31000              on node01 (priority: 0, waiting: none)
> node01 crmd[952]:  warning: action_timer_callback: Timer popped
> (timeout=20000, abort_level=1000000, complete=false)
> node01 crmd[952]:    error: print_synapse: [Action   68]: In-flight rsc
> op drbd_pg_pre_notify_demote_0       on node02 (priority: 0, waiting: none)
> node01 crmd[952]:  warning: cib_action_update: rsc_op 68:
> drbd_pg_pre_notify_demote_0 on node02 timed out
> node01 crmd[952]:    error: cib_action_updated: Update 297 FAILED: Timer
> expired
> node01 crmd[952]:    error: stonith_async_timeout_handler: Async call 2
> timed out after 120000ms
> node01 crmd[952]:   notice: tengine_stonith_callback: Stonith operation
> 2/54:261:0:6978227d-ce2d-4dc6-955a-eb9313f112a5: Timer expired (-62)
> node01 crmd[952]:   notice: tengine_stonith_callback: Stonith operation
> 2 for node02 failed (Timer expired): aborting transition.
> node01 crmd[952]:   notice: run_graph: Transition 261 (Complete=6,
> Pending=0, Fired=0, Skipped=29, Incomplete=15,
> Source=/var/lib/pacemaker/pengine/pe-warn-0.bz2): Stopped
> node01 pengine[951]:   notice: unpack_config: On loss of CCM Quorum: Ignore
> node01 pengine[951]:  warning: pe_fence_node: Node node02 will be fenced
> because our peer process is no longer available
> node01 pengine[951]:  warning: determine_online_status: Node node02 is
> unclean
> node01 pengine[951]:  warning: stage6: Scheduling Node node02 for STONITH
> node01 pengine[951]:   notice: LogActions: Move    fs_pg#011(Started
> node02 -> node01)
> node01 pengine[951]:   notice: LogActions: Move    ip_pg#011(Started
> node02 -> node01)
> node01 pengine[951]:   notice: LogActions: Move    lsb_pg#011(Started
> node02 -> node01)
> node01 pengine[951]:   notice: LogActions: Demote  drbd_pg:0#011(Master
> -> Stopped node02)
> node01 pengine[951]:   notice: LogActions: Promote drbd_pg:1#011(Slave
> -> Master node01)
> node01 pengine[951]:   notice: LogActions: Stop    p_fence:0#011(node02)
> node01 crmd[952]:   notice: te_fence_node: Executing reboot fencing
> operation (53) on node02 (timeout=60000)
> node01 stonith-ng[948]:   notice: handle_request: Client
> crmd.952.6d7ac808 wants to fence (reboot) 'node02' with device '(any)'
> node01 stonith-ng[948]:   notice: initiate_remote_stonith_op: Initiating
> remote operation reboot for node02: a4fae8ce-3a6c-4fe5-a934-b5b83ae123cb (0)
> node01 crmd[952]:   notice: te_rsc_command: Initiating action 67: notify
> drbd_pg_pre_notify_demote_0 on node02
> node01 crmd[952]:   notice: te_rsc_command: Initiating action 69: notify
> drbd_pg_pre_notify_demote_0 on node01 (local)
> node01 pengine[951]:  warning: process_pe_message: Calculated Transition
> 262: /var/lib/pacemaker/pengine/pe-warn-1.bz2
> node01 crmd[952]:   notice: process_lrm_event: LRM operation
> drbd_pg_notify_0 (call=66, rc=0, cib-update=0, confirmed=true) ok
> 
> Last updated: Mon Sep 15 01:15:59 2014
> Last change: Sat Sep 13 15:23:45 2014 via cibadmin on node01
> Stack: corosync
> Current DC: node01 (167936788) - partition with quorum
> Version: 1.1.10-42f2063
> 2 Nodes configured
> 7 Resources configured
> 
> 
> Node node02 (167936789): UNCLEAN (online)
> Online: [ node01 ]
> 
>  Resource Group: PGServer
>      fs_pg      (ocf::heartbeat:Filesystem):    Started node02
>      ip_pg      (ocf::heartbeat:IPaddr2):       Started node02
>      lsb_pg     (lsb:postgresql):       Started node02
>  Master/Slave Set: ms_drbd_pg [drbd_pg]
>      Masters: [ node02 ]
>      Slaves: [ node01 ]
>  Clone Set: cln_p_fence [p_fence]
>      Started: [ node01 node02 ]
> 
> Thank you,
> Norbert
> 
> On Fri, Sep 12, 2014 at 12:06 PM, Vladislav Bogdanov
> <bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>> wrote:
> 
>     12.09.2014 05:00, Norbert Kiam Maclang wrote:
>     > Hi,
>     >
>     > After adding resource level fencing on drbd, I still ended up having
>     > problems with timeouts on drbd. Is there a recommended settings for
>     > this? I followed what is written in the drbd documentation -
>     >
>     http://www.drbd.org/users-guide-emb/s-pacemaker-crm-drbd-backed-service.html
>     > , Another thing I can't understand is why during initial tests, even I
>     > reboot the vms several times, failover works. But after I soak it
>     for a
>     > couple of hours (say for example 8 hours or more) and continue
>     with the
>     > tests, it will not failover and experience split brain. I confirmed it
>     > though that everything is healthy before performing a reboot. Disk
>     > health and network is good, drbd is synced, time beetween servers
>     is good.
> 
>     I recall I've seen something similar a year ago (near the time your
>     pacemaker version is dated). I do not remember what was the exact
>     problem cause, but I saw that drbd RA timeouts because it waits for
>     something (fencing) in the kernel space to be done. drbd calls userspace
>     scripts from within kernelspace, and you'll see them in the process list
>     with the drbd kernel thread as a parent.
> 
>     I'd also upgrade your corosync configuration from "member" to "nodelist"
>     syntax, specifying "name" parameter together with ring0_addr for nodes
>     (that parameter is not referenced in corosync docs but should be
>     somewhere in the Pacemaker Explained - it is used only by the
>     pacemaker).
> 
>     Also there is trace_ra functionality support in both pacemaker and crmsh
>     (cannot say if that is supported in versions you have though, probably
>     yes) so you may want to play with that to get the exact picture from the
>     resource agent.
> 
>     Anyways, upgrading to 1.1.12 and more recent crmsh is nice to have for
>     you because you may be just hitting a long-ago solved and forgotten
>     bug/issue.
> 
>     Concerning your
>     >       expected-quorum-votes="1"
> 
>     You need to configure votequorum in corosync with two_node: 1 instead of
>     that line.
> 
>     >
>     > # Logs:
>     > node01 lrmd[1036]:  warning: child_timeout_callback:
>     > drbd_pg_monitor_29000 process (PID 27744) timed out
>     > node01 lrmd[1036]:  warning: operation_finished:
>     > drbd_pg_monitor_29000:27744 - timed out after 20000ms
>     > node01 crmd[1039]:    error: process_lrm_event: LRM operation
>     > drbd_pg_monitor_29000 (69) Timed Out (timeout=20000ms)
>     > node01 crmd[1039]:  warning: update_failcount: Updating failcount for
>     > drbd_pg on tyo1mqdb01p after failed monitor: rc=1 (update=value++,
>     > time=1410486352)
>     >
>     > Thanks,
>     > Kiam
>     >
>     > On Thu, Sep 11, 2014 at 6:58 PM, Norbert Kiam Maclang
>     > <norbert.kiam.maclang at gmail.com
>     <mailto:norbert.kiam.maclang at gmail.com>
>     <mailto:norbert.kiam.maclang at gmail.com
>     <mailto:norbert.kiam.maclang at gmail.com>>>
>     > wrote:
>     >
>     >     Thank you Vladislav.
>     >
>     >     I have configured resource level fencing on drbd and removed
>     >     wfc-timeout and defr-wfc-timeout (is this required?). My drbd
>     >     configuration is now:
>     >
>     >     resource pg {
>     >       device /dev/drbd0;
>     >       disk /dev/vdb;
>     >       meta-disk internal;
>     >       disk {
>     >         fencing resource-only;
>     >         on-io-error detach;
>     >         resync-rate 40M;
>     >       }
>     >       handlers {
>     >         fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
>     >         after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
>     >         split-brain "/usr/lib/drbd/notify-split-brain.sh nkbm";
>     >       }
>     >       on node01 {
>     >         address 10.2.136.52:7789 <http://10.2.136.52:7789>
>     <http://10.2.136.52:7789>;
>     >       }
>     >       on node02 {
>     >         address 10.2.136.55:7789 <http://10.2.136.55:7789>
>     <http://10.2.136.55:7789>;
>     >       }
>     >       net {
>     >         verify-alg md5;
>     >         after-sb-0pri discard-zero-changes;
>     >         after-sb-1pri discard-secondary;
>     >         after-sb-2pri disconnect;
>     >       }
>     >     }
>     >
>     >     Failover works on my initial test (restarting both nodes
>     alternately
>     >     - this always works). Will wait for a couple of hours after
>     doing a
>     >     failover test again (Which always fail on my previous setup).
>     >
>     >     Thank you!
>     >     Kiam
>     >
>     >     On Thu, Sep 11, 2014 at 2:14 PM, Vladislav Bogdanov
>     >     <bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>
>     <mailto:bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>>> wrote:
>     >
>     >         11.09.2014 05:57, Norbert Kiam Maclang wrote:
>     >         > Is this something to do with quorum? But I already set
>     >
>     >         You'd need to configure fencing at the drbd resources level.
>     >
>     >       
>      http://www.drbd.org/users-guide-emb/s-pacemaker-fencing.html#s-pacemaker-fencing-cib
>     >
>     >
>     >         >
>     >         > property no-quorum-policy="ignore" \
>     >         > expected-quorum-votes="1"
>     >         >
>     >         > Thanks in advance,
>     >         > Kiam
>     >         >
>     >         > On Thu, Sep 11, 2014 at 10:09 AM, Norbert Kiam Maclang
>     >         > <norbert.kiam.maclang at gmail.com
>     <mailto:norbert.kiam.maclang at gmail.com>
>     >         <mailto:norbert.kiam.maclang at gmail.com
>     <mailto:norbert.kiam.maclang at gmail.com>>
>     >         <mailto:norbert.kiam.maclang at gmail.com
>     <mailto:norbert.kiam.maclang at gmail.com>
>     >         <mailto:norbert.kiam.maclang at gmail.com
>     <mailto:norbert.kiam.maclang at gmail.com>>>>
>     >         > wrote:
>     >         >
>     >         >     Hi,
>     >         >
>     >         >     Please help me understand what is causing the problem. I
>     >         have a 2
>     >         >     node cluster running on vms using KVM. Each vm (I am
>     using
>     >         Ubuntu
>     >         >     14.04) runs on a separate hypervisor on separate
>     machines.
>     >         All are
>     >         >     working well during testing (I restarted the vms
>     >         alternately), but
>     >         >     after a day when I kill the other node, I always end up
>     >         corosync and
>     >         >     pacemaker hangs on the surviving node. Date and time on
>     >         the vms are
>     >         >     in sync, I use unicast, tcpdump shows both nodes
>     exchanges,
>     >         >     confirmed that DRBD is healthy and crm_mon show good
>     >         status before I
>     >         >     kill the other node. Below are my configurations and
>     >         versions I used:
>     >         >
>     >         >     corosync             2.3.3-1ubuntu1
>     >         >     crmsh                1.2.5+hg1034-1ubuntu3
>     >         >     drbd8-utils          2:8.4.4-1ubuntu1
>     >         >     libcorosync-common4  2.3.3-1ubuntu1
>     >         >     libcrmcluster4       1.1.10+git20130802-1ubuntu2
>     >         >     libcrmcommon3        1.1.10+git20130802-1ubuntu2
>     >         >     libcrmservice1       1.1.10+git20130802-1ubuntu2
>     >         >     pacemaker            1.1.10+git20130802-1ubuntu2
>     >         >     pacemaker-cli-utils  1.1.10+git20130802-1ubuntu2
>     >         >     postgresql-9.3       9.3.5-0ubuntu0.14.04.1
>     >         >
>     >         >     # /etc/corosync/corosync:
>     >         >     totem {
>     >         >     version: 2
>     >         >     token: 3000
>     >         >     token_retransmits_before_loss_const: 10
>     >         >     join: 60
>     >         >     consensus: 3600
>     >         >     vsftype: none
>     >         >     max_messages: 20
>     >         >     clear_node_high_bit: yes
>     >         >      secauth: off
>     >         >      threads: 0
>     >         >      rrp_mode: none
>     >         >      interface {
>     >         >                     member {
>     >         >                             memberaddr: 10.2.136.56
>     >         >                     }
>     >         >                     member {
>     >         >                             memberaddr: 10.2.136.57
>     >         >                     }
>     >         >                     ringnumber: 0
>     >         >                     bindnetaddr: 10.2.136.0
>     >         >                     mcastport: 5405
>     >         >             }
>     >         >             transport: udpu
>     >         >     }
>     >         >     amf {
>     >         >     mode: disabled
>     >         >     }
>     >         >     quorum {
>     >         >     provider: corosync_votequorum
>     >         >     expected_votes: 1
>     >         >     }
>     >         >     aisexec {
>     >         >             user:   root
>     >         >             group:  root
>     >         >     }
>     >         >     logging {
>     >         >             fileline: off
>     >         >             to_stderr: yes
>     >         >             to_logfile: no
>     >         >             to_syslog: yes
>     >         >     syslog_facility: daemon
>     >         >             debug: off
>     >         >             timestamp: on
>     >         >             logger_subsys {
>     >         >                     subsys: AMF
>     >         >                     debug: off
>     >         >                     tags:
>     >         enter|leave|trace1|trace2|trace3|trace4|trace6
>     >         >             }
>     >         >     }
>     >         >
>     >         >     # /etc/corosync/service.d/pcmk:
>     >         >     service {
>     >         >       name: pacemaker
>     >         >       ver: 1
>     >         >     }
>     >         >
>     >         >     /etc/drbd.d/global_common.conf:
>     >         >     global {
>     >         >     usage-count no;
>     >         >     }
>     >         >
>     >         >     common {
>     >         >     net {
>     >         >                     protocol C;
>     >         >     }
>     >         >     }
>     >         >
>     >         >     # /etc/drbd.d/pg.res:
>     >         >     resource pg {
>     >         >       device /dev/drbd0;
>     >         >       disk /dev/vdb;
>     >         >       meta-disk internal;
>     >         >       startup {
>     >         >         wfc-timeout 15;
>     >         >         degr-wfc-timeout 60;
>     >         >       }
>     >         >       disk {
>     >         >         on-io-error detach;
>     >         >         resync-rate 40M;
>     >         >       }
>     >         >       on node01 {
>     >         >         address 10.2.136.56:7789
>     <http://10.2.136.56:7789> <http://10.2.136.56:7789>
>     >         <http://10.2.136.56:7789>;
>     >         >       }
>     >         >       on node02 {
>     >         >         address 10.2.136.57:7789
>     <http://10.2.136.57:7789> <http://10.2.136.57:7789>
>     >         <http://10.2.136.57:7789>;
>     >         >       }
>     >         >       net {
>     >         >         verify-alg md5;
>     >         >         after-sb-0pri discard-zero-changes;
>     >         >         after-sb-1pri discard-secondary;
>     >         >         after-sb-2pri disconnect;
>     >         >       }
>     >         >     }
>     >         >
>     >         >     # Pacemaker configuration:
>     >         >     node $id="167938104" node01
>     >         >     node $id="167938105" node02
>     >         >     primitive drbd_pg ocf:linbit:drbd \
>     >         >     params drbd_resource="pg" \
>     >         >     op monitor interval="29s" role="Master" \
>     >         >     op monitor interval="31s" role="Slave"
>     >         >     primitive fs_pg ocf:heartbeat:Filesystem \
>     >         >     params device="/dev/drbd0"
>     >         directory="/var/lib/postgresql/9.3/main"
>     >         >     fstype="ext4"
>     >         >     primitive ip_pg ocf:heartbeat:IPaddr2 \
>     >         >     params ip="10.2.136.59" cidr_netmask="24" nic="eth0"
>     >         >     primitive lsb_pg lsb:postgresql
>     >         >     group PGServer fs_pg lsb_pg ip_pg
>     >         >     ms ms_drbd_pg drbd_pg \
>     >         >     meta master-max="1" master-node-max="1" clone-max="2"
>     >         >     clone-node-max="1" notify="true"
>     >         >     colocation pg_on_drbd inf: PGServer ms_drbd_pg:Master
>     >         >     order pg_after_drbd inf: ms_drbd_pg:promote
>     PGServer:start
>     >         >     property $id="cib-bootstrap-options" \
>     >         >     dc-version="1.1.10-42f2063" \
>     >         >     cluster-infrastructure="corosync" \
>     >         >     stonith-enabled="false" \
>     >         >     no-quorum-policy="ignore"
>     >         >     rsc_defaults $id="rsc-options" \
>     >         >     resource-stickiness="100"
>     >         >
>     >         >     # Logs on node01
>     >         >     Sep 10 10:25:33 node01 crmd[1019]:   notice:
>     >         peer_update_callback:
>     >         >     Our peer on the DC is dead
>     >         >     Sep 10 10:25:33 node01 crmd[1019]:   notice:
>     >         do_state_transition:
>     >         >     State transition S_NOT_DC -> S_ELECTION [
>     input=I_ELECTION
>     >         >     cause=C_CRMD_STATUS_CALLBACK
>     origin=peer_update_callback ]
>     >         >     Sep 10 10:25:33 node01 crmd[1019]:   notice:
>     >         do_state_transition:
>     >         >     State transition S_ELECTION -> S_INTEGRATION [
>     >         input=I_ELECTION_DC
>     >         >     cause=C_FSA_INTERNAL origin=do_election_check ]
>     >         >     Sep 10 10:25:33 node01 corosync[940]:   [TOTEM ] A new
>     >         membership
>     >         >     (10.2.136.56:52 <http://10.2.136.56:52>
>     <http://10.2.136.56:52>
>     >         <http://10.2.136.56:52>) was formed. Members left:
>     >         >     167938105
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.740024] d-con pg:
>     >         PingAck did
>     >         >     not arrive in time.
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.740169] d-con
>     pg: peer(
>     >         >     Primary -> Unknown ) conn( Connected ->
>     NetworkFailure ) pdsk(
>     >         >     UpToDate -> DUnknown )
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.740987] d-con pg:
>     >         asender
>     >         >     terminated
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.740999] d-con pg:
>     >         Terminating
>     >         >     drbd_a_pg
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.741235] d-con pg:
>     >         Connection
>     >         >     closed
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.741259] d-con
>     pg: conn(
>     >         >     NetworkFailure -> Unconnected )
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.741260] d-con pg:
>     >         receiver
>     >         >     terminated
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.741261] d-con pg:
>     >         Restarting
>     >         >     receiver thread
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.741262] d-con pg:
>     >         receiver
>     >         >     (re)started
>     >         >     Sep 10 10:25:45 node01 kernel: [74452.741269] d-con
>     pg: conn(
>     >         >     Unconnected -> WFConnection )
>     >         >     Sep 10 10:26:12 node01 lrmd[1016]:  warning:
>     >         child_timeout_callback:
>     >         >     drbd_pg_monitor_31000 process (PID 8445) timed out
>     >         >     Sep 10 10:26:12 node01 lrmd[1016]:  warning:
>     >         operation_finished:
>     >         >     drbd_pg_monitor_31000:8445 - timed out after 20000ms
>     >         >     Sep 10 10:26:12 node01 crmd[1019]:    error:
>     >         process_lrm_event: LRM
>     >         >     operation drbd_pg_monitor_31000 (30) Timed Out
>     >         (timeout=20000ms)
>     >         >     Sep 10 10:26:32 node01 crmd[1019]:  warning:
>     cib_rsc_callback:
>     >         >     Resource update 23 failed: (rc=-62) Timer expired
>     >         >     Sep 10 10:27:03 node01 lrmd[1016]:  warning:
>     >         child_timeout_callback:
>     >         >     drbd_pg_monitor_31000 process (PID 8693) timed out
>     >         >     Sep 10 10:27:03 node01 lrmd[1016]:  warning:
>     >         operation_finished:
>     >         >     drbd_pg_monitor_31000:8693 - timed out after 20000ms
>     >         >     Sep 10 10:27:54 node01 lrmd[1016]:  warning:
>     >         child_timeout_callback:
>     >         >     drbd_pg_monitor_31000 process (PID 8938) timed out
>     >         >     Sep 10 10:27:54 node01 lrmd[1016]:  warning:
>     >         operation_finished:
>     >         >     drbd_pg_monitor_31000:8938 - timed out after 20000ms
>     >         >     Sep 10 10:28:33 node01 crmd[1019]:    error:
>     crm_timer_popped:
>     >         >     Integration Timer (I_INTEGRATED) just popped in state
>     >         S_INTEGRATION!
>     >         >     (180000ms)
>     >         >     Sep 10 10:28:33 node01 crmd[1019]:  warning:
>     >         do_state_transition:
>     >         >     Progressed to state S_FINALIZE_JOIN after C_TIMER_POPPED
>     >         >     Sep 10 10:28:33 node01 crmd[1019]:  warning:
>     >         do_state_transition: 1
>     >         >     cluster nodes failed to respond to the join offer.
>     >         >     Sep 10 10:28:33 node01 crmd[1019]:   notice:
>     >         crmd_join_phase_log:
>     >         >     join-1: node02=none
>     >         >     Sep 10 10:28:33 node01 crmd[1019]:   notice:
>     >         crmd_join_phase_log:
>     >         >     join-1: node01=welcomed
>     >         >     Sep 10 10:28:45 node01 lrmd[1016]:  warning:
>     >         child_timeout_callback:
>     >         >     drbd_pg_monitor_31000 process (PID 9185) timed out
>     >         >     Sep 10 10:28:45 node01 lrmd[1016]:  warning:
>     >         operation_finished:
>     >         >     drbd_pg_monitor_31000:9185 - timed out after 20000ms
>     >         >     Sep 10 10:29:36 node01 lrmd[1016]:  warning:
>     >         child_timeout_callback:
>     >         >     drbd_pg_monitor_31000 process (PID 9432) timed out
>     >         >     Sep 10 10:29:36 node01 lrmd[1016]:  warning:
>     >         operation_finished:
>     >         >     drbd_pg_monitor_31000:9432 - timed out after 20000ms
>     >         >     Sep 10 10:30:27 node01 lrmd[1016]:  warning:
>     >         child_timeout_callback:
>     >         >     drbd_pg_monitor_31000 process (PID 9680) timed out
>     >         >     Sep 10 10:30:27 node01 lrmd[1016]:  warning:
>     >         operation_finished:
>     >         >     drbd_pg_monitor_31000:9680 - timed out after 20000ms
>     >         >     Sep 10 10:31:18 node01 lrmd[1016]:  warning:
>     >         child_timeout_callback:
>     >         >     drbd_pg_monitor_31000 process (PID 9927) timed out
>     >         >     Sep 10 10:31:18 node01 lrmd[1016]:  warning:
>     >         operation_finished:
>     >         >     drbd_pg_monitor_31000:9927 - timed out after 20000ms
>     >         >     Sep 10 10:32:09 node01 lrmd[1016]:  warning:
>     >         child_timeout_callback:
>     >         >     drbd_pg_monitor_31000 process (PID 10174) timed out
>     >         >     Sep 10 10:32:09 node01 lrmd[1016]:  warning:
>     >         operation_finished:
>     >         >     drbd_pg_monitor_31000:10174 - timed out after 20000ms
>     >         >
>     >         >     #crm_mon on node01 before I kill the other vm:
>     >         >     Stack: corosync
>     >         >     Current DC: node02 (167938104) - partition with quorum
>     >         >     Version: 1.1.10-42f2063
>     >         >     2 Nodes configured
>     >         >     5 Resources configured
>     >         >
>     >         >     Online: [ node01 node02 ]
>     >         >
>     >         >      Resource Group: PGServer
>     >         >          fs_pg      (ocf::heartbeat:Filesystem):   
>     Started node02
>     >         >          lsb_pg     (lsb:postgresql):       Started node02
>     >         >          ip_pg      (ocf::heartbeat:IPaddr2):     
>      Started node02
>     >         >      Master/Slave Set: ms_drbd_pg [drbd_pg]
>     >         >          Masters: [ node02 ]
>     >         >          Slaves: [ node01 ]
>     >         >
>     >         >     Thank you,
>     >         >     Kiam
>     >         >
>     >         >
>     >         >
>     >         >
>     >         > _______________________________________________
>     >         > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>     <mailto:Pacemaker at oss.clusterlabs.org>
>     >         <mailto:Pacemaker at oss.clusterlabs.org
>     <mailto:Pacemaker at oss.clusterlabs.org>>
>     >         > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>     >         >
>     >         > Project Home: http://www.clusterlabs.org
>     >         > Getting started:
>     >         http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>     >         > Bugs: http://bugs.clusterlabs.org
>     >         >
>     >
>     >
>     >         _______________________________________________
>     >         Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>     <mailto:Pacemaker at oss.clusterlabs.org>
>     >         <mailto:Pacemaker at oss.clusterlabs.org
>     <mailto:Pacemaker at oss.clusterlabs.org>>
>     >         http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>     >
>     >         Project Home: http://www.clusterlabs.org
>     >         Getting started:
>     >         http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>     >         Bugs: http://bugs.clusterlabs.org
>     >
>     >
>     >
>     >
>     >
>     > _______________________________________________
>     > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>     <mailto:Pacemaker at oss.clusterlabs.org>
>     > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>     >
>     > Project Home: http://www.clusterlabs.org
>     > Getting started:
>     http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>     > Bugs: http://bugs.clusterlabs.org
>     >
> 
> 
>     _______________________________________________
>     Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>     <mailto:Pacemaker at oss.clusterlabs.org>
>     http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
>     Project Home: http://www.clusterlabs.org
>     Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>     Bugs: http://bugs.clusterlabs.org
> 
> 
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 





More information about the Pacemaker mailing list