[ClusterLabs] Cluster node getting stopped from other node(resending mail)

Ken Gaillot kgaillot at redhat.com
Tue Jul 7 14:23:03 UTC 2015


On 07/07/2015 03:58 AM, Arjun Pandey wrote:
> Hi Ken
> 
> If i look at the logs on the other node around the same time i see this. I
> can't figure out the reason based on these.Attaching the corosync.log for
> the other node as well.

I don't see anything there either. The relevant part could be earlier or
later.

Are you using the same version of pacemaker and corosync on both nodes?

> Jun 01 13:55:44 [1965] messi       crmd:     info: do_dc_join_offer_one:
>      An unknown node joined - (re-)offer to any unconfirmed nodes
> Jun 01 13:55:44 [1965] messi       crmd:     info: join_make_offer:
> Making join offers based on membership 224
> Jun 01 13:55:44 [1965] messi       crmd:     info: join_make_offer:
> Skipping messi: already known 4
> Jun 01 13:55:44 [1965] messi       crmd:     info: join_make_offer:
> join-2: Sending offer to ronaldo
> Jun 01 13:55:44 [1960] messi        cib:     info: cib_process_request:
>     Forwarding cib_modify operation for section status to master
> (origin=local/crmd/138)
> Jun 01 13:55:44 [1960] messi        cib:     info: cib_perform_op:
>  Diff: --- 0.80.2 2
> Jun 01 13:55:44 [1960] messi        cib:     info: cib_perform_op:
>  Diff: +++ 0.80.3 (null)
> Jun 01 13:55:44 [1965] messi       crmd:     info: crm_update_peer_join:
>      join_make_offer: Node ronaldo[2] - join-2 phase 0 -> 1
> Jun 01 13:55:44 [1965] messi       crmd:     info: abort_transition_graph:
>      Transition aborted: Peer Halt (source=do_te_invoke:158, 1)
> Jun 01 13:55:44 [1960] messi        cib:     info: cib_perform_op:      +
>  /cib:  @num_updates=3
> Jun 01 13:55:44 [1960] messi        cib:     info: cib_perform_op:      +
>  /cib/status/node_state[@id='ronaldo']:  @crmd=online,
> @crm-debug-origin=peer_update_callback
> Jun 01 13:55:44 [1960] messi        cib:     info: cib_process_request:
>     Completed cib_modify operation for section status: OK (rc=0,
> origin=messi/crmd/138, version=0.80.3)
> Jun 01 13:55:44 [1960] messi        cib:     info: cib_process_request:
>     Completed cib_modify operation for section nodes: OK (rc=0,
> origin=ronaldo/crmd/3, version=0.80.3)
> Jun 01 13:55:45 [1965] messi       crmd:     info: do_dc_join_offer_one:
>      join-2: Processing join_announce request from ronaldo in state
> S_INTEGRATION
> Jun 01 13:55:45 [1965] messi       crmd:     info: crm_update_peer_join:
>      do_dc_join_offer_one: Node ronaldo[2] - join-2 phase 1 -> 0
> Jun 01 13:55:45 [1965] messi       crmd:     info: join_make_offer:
> join-2: Sending offer to ronaldo
> Jun 01 13:55:45 [1965] messi       crmd:     info: crm_update_peer_join:
>      join_make_offer: Node ronaldo[2] - join-2 phase 0 -> 1
> Jun 01 13:55:45 [1965] messi       crmd:     info: crm_update_peer_join:
>      join_make_offer: Node messi[1] - join-2 phase 4 -> 0
> Jun 01 13:55:45 [1965] messi       crmd:     info: join_make_offer:
> join-2: Sending offer to messi
> Jun 01 13:55:45 [1965] messi       crmd:     info: crm_update_peer_join:
>      join_make_offer: Node messi[1] - join-2 phase 0 -> 1
> Jun 01 13:55:45 [1965] messi       crmd:     info: abort_transition_graph:
>      Transition aborted: Node join (source=do_dc_join_offer_one:236, 1)
> Jun 01 13:55:45 [1965] messi       crmd:     info: crm_update_peer_join:
>      do_dc_join_filter_offer: Node messi[1] - join-2 phase 1 -> 2
> Jun 01 13:55:46 [1965] messi       crmd:     info: crm_update_peer_join:
>      do_dc_join_filter_offer: Node ronaldo[2] - join-2 phase 1 -> 2
> Jun 01 13:55:46 [1965] messi       crmd:     info:
> crm_update_peer_expected:    do_dc_join_filter_offer: Node ronaldo[2] -
> expected state is now member (was down)
> Jun 01 13:55:46 [1965] messi       crmd:     info: do_state_transition:
>     State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
> cause=C_FSA_INTERNAL origin=check_join_state ]
> Jun 01 13:55:46 [1965] messi       crmd:     info: crmd_join_phase_log:
>     join-2: messi=integrated
> Jun 01 13:55:46 [1965] messi       crmd:     info: crmd_join_phase_log:
>     join-2: ronaldo=integrated
> Jun 01 13:55:46 [1965] messi       crmd:     info: do_dc_join_finalize:
>     join-2: Syncing our CIB to the rest of the cluster
> Jun 01 13:55:46 [1965] messi       crmd:     info: crm_update_peer_join:
>      finalize_join_for: Node messi[1] - join-2 phase 2 -> 3
> Jun 01 13:55:46 [1965] messi       crmd:     info: crm_update_peer_join:
>      finalize_join_for: Node ronaldo[2] - join-2 phase 2 -> 3
> Jun 01 13:55:46 [1965] messi       crmd:     info: crm_update_peer_join:
>      do_dc_join_ack: Node messi[1] - join-2 phase 3 -> 4
> Jun 01 13:55:46 [1965] messi       crmd:     info: do_dc_join_ack:
>  join-2: Updating node state to member for messi
> Jun 01 13:55:46 [1965] messi       crmd:     info: erase_status_tag:
>  Deleting xpath: //node_state[@uname='messi']/lrm
> Jun 01 13:55:46 [1960] messi        cib:     info: cib_process_replace:
>     Digest matched on replace from messi: 5138b696984c7b834dd2b528dadabe0d
> Jun 01 13:55:46 [1960] messi        cib:     info: cib_process_replace:
>     Replaced 0.80.3 with 0.80.3 from messi
> Jun 01 13:55:46 [1965] messi       crmd:     info: crm_update_peer_join:
>      do_dc_join_ack: Node ronaldo[2] - join-2 phase 3 -> 4
> 
> 
> Regards
> Arjun
> 
> On Wed, Jul 1, 2015 at 7:17 PM, Ken Gaillot <kgaillot at redhat.com> wrote:
> 
>> On 06/30/2015 11:30 PM, Arjun Pandey wrote:
>>> Hi
>>>
>>> I am running a 2 node cluster with this config on centos 6.5/6.6
>>>
>>> Master/Slave Set: foo-master [foo]
>>> Masters: [ messi ]
>>> Stopped: [ronaldo ]
>>>  eth1-CP        (ocf::pw:IPaddr):       Started messi
>>>  eth2-UP        (ocf::pw:IPaddr):       Started messi
>>>  eth3-UPCP      (ocf::pw:IPaddr):       Started messi
>>>
>>> where i have a multi-state resource foo being run in master/slave mode
>> and
>>>  IPaddr RA is just modified IPAddr2 RA. Additionally i have a
>>> collocation constraint for the IP addr to be collocated with the master.
>>>
>>> Sometimes when i setup the cluster , i find that one of the nodes (the
>>> second node that joins ) gets stopped and i find this log.
>>>
>>> 2015-06-01T13:55:46.153941+05:30 ronaldo pacemaker: Starting Pacemaker
>>> Cluster Manager
>>> 2015-06-01T13:55:46.233639+05:30 ronaldo attrd[25988]:   notice:
>>> attrd_trigger_update: Sending flush op to all hosts for: shutdown (0)
>>> 2015-06-01T13:55:46.234162+05:30 ronaldo crmd[25990]:   notice:
>>> do_state_transition: State transition S_PENDING -> S_NOT_DC [
>>> input=I_NOT_DC cause=C_HA_MESSAG
>>> E origin=do_cl_join_finalize_respond ]
>>> 2015-06-01T13:55:46.234701+05:30 ronaldo attrd[25988]:   notice:
>>> attrd_local_callback: Sending full refresh (origin=crmd)
>>> 2015-06-01T13:55:46.234708+05:30 ronaldo attrd[25988]:   notice:
>>> attrd_trigger_update: Sending flush op to all hosts for: shutdown (0)
>>> ************************ This looks to be the likely
>>> reason*******************************************
>>> 2015-06-01T13:55:46.254310+05:30 ronaldo crmd[25990]:    error:
>>> handle_request: We didn't ask to be shut down, yet our DC is telling us
>> too
>>> .
>>>
>> *********************************************************************************************************
>>
>> Hi Arjun,
>>
>> I'd check the other node's logs at this time, to see why it requested
>> the shutdown.
>>
>>> 2015-06-01T13:55:46.254577+05:30 ronaldo crmd[25990]:   notice:
>>> do_state_transition: State transition S_NOT_DC -> S_STOPPING [
>> input=I_STOP
>>> cause=C_HA_MESSAGE
>>>  origin=route_message ]
>>> 2015-06-01T13:55:46.255134+05:30 ronaldo crmd[25990]:   notice:
>>> lrm_state_verify_stopped: Stopped 0 recurring operations at shutdown...
>>> waiting (2 ops remaining)
>>>
>>> Based on the logs , pacemaker on active was stopping the secondary cloud
>>> everytime it joins cluster. This issue seems similar to
>>>
>> http://pacemaker.oss.clusterlabs.narkive.com/rVvN8May/node-sends-shutdown-request-to-other-node-error
>>>
>>> Packages used :-
>>> pacemaker-1.1.12-4.el6.x86_64
>>> pacemaker-libs-1.1.12-4.el6.x86_64
>>> pacemaker-cli-1.1.12-4.el6.x86_64
>>> pacemaker-cluster-libs-1.1.12-4.el6.x86_64
>>> pacemaker-debuginfo-1.1.10-14.el6.x86_64
>>> pcsc-lite-libs-1.5.2-13.el6_4.x86_64
>>> pcs-0.9.90-2.el6.centos.2.noarch
>>> pcsc-lite-1.5.2-13.el6_4.x86_64
>>> pcsc-lite-openct-0.6.19-4.el6.x86_64
>>> corosync-1.4.1-17.el6.x86_64
>>> corosynclib-1.4.1-17.el6.x86_64





More information about the Users mailing list