[ClusterLabs] Antw: Re: [Question] About movement of pacemaker_remote.

Andrew Beekhof andrew at beekhof.net
Mon Apr 13 05:11:57 UTC 2015


> On 8 Apr 2015, at 12:27 pm, renayama19661014 at ybb.ne.jp wrote:
> 
> Hi All,
> 
> Let me confirm the first question once again.
> 
> I confirmed the next movement in Pacemaker1.1.13-rc1.
> Stonith does not set it.
> 
> -------------------------------------------------------------
> property no-quorum-policy="ignore" \
>         stonith-enabled="false" \
>         startup-fencing="false" \
> 
> rsc_defaults resource-stickiness="INFINITY" \
>         migration-threshold="1"
> 
> primitive snmp1 ocf:pacemaker:remote \
>         params \
>                 server="snmp1" \
>         op start interval="0s" timeout="60s" on-fail="restart" \
>         op monitor interval="3s" timeout="15s" \
>         op stop interval="0s" timeout="60s" on-fail="ignore"
> 
> primitive snmp2 ocf:pacemaker:remote \
>         params \
>                 server="snmp2" \
>         op start interval="0s" timeout="60s" on-fail="restart" \
>         op monitor interval="3s" timeout="15s" \
>         op stop interval="0s" timeout="60s" on-fail="stop"
> 
> primitive Host-rsc1 ocf:heartbeat:Dummy \
>         op start interval="0s" timeout="60s" on-fail="restart" \
>         op monitor interval="10s" timeout="60s" on-fail="restart" \
>         op stop interval="0s" timeout="60s" on-fail="ignore"
> 
> primitive Remote-rsc1 ocf:heartbeat:Dummy \
>         op start interval="0s" timeout="60s" on-fail="restart" \
>         op monitor interval="10s" timeout="60s" on-fail="restart" \
>         op stop interval="0s" timeout="60s" on-fail="ignore"
> 
> primitive Remote-rsc2 ocf:heartbeat:Dummy \
>         op start interval="0s" timeout="60s" on-fail="restart" \
>         op monitor interval="10s" timeout="60s" on-fail="restart" \
>         op stop interval="0s" timeout="60s" on-fail="ignore"
> 
> location loc1 Remote-rsc1 \
>         rule 200: #uname eq snmp1 \
>         rule 100: #uname eq snmp2
> location loc2 Remote-rsc2 \        rule 200: #uname eq snmp2 \        rule 100: #uname eq snmp1
> location loc3 Host-rsc1 \
>         rule 200: #uname eq sl7-01
> 
> -------------------------------------------------------------
> 
> Step 1) We use two remote nodes and constitute a cluster.
> -------------------------------------------------------------
> Version: 1.1.12-3e93bc1
> 3 Nodes configured
> 5 Resources configured
> 
> 
> Online: [ sl7-01 ]
> RemoteOnline: [ snmp1 snmp2 ]
> 
>  Host-rsc1      (ocf::heartbeat:Dummy): Started sl7-01 
>  Remote-rsc1    (ocf::heartbeat:Dummy): Started snmp1 
>  Remote-rsc2    (ocf::heartbeat:Dummy): Started snmp2 
>  snmp1  (ocf::pacemaker:remote):        Started sl7-01 
>  snmp2  (ocf::pacemaker:remote):        Started sl7-01 
> 
> Node Attributes:
> * Node sl7-01:
> * Node snmp1:
> * Node snmp2:
> 
> Migration summary:
> * Node sl7-01: 
> * Node snmp1: 
> * Node snmp2: 
> -------------------------------------------------------------
> 
> Step 2) We stop pacemaker_remoted in one remote.
> -------------------------------------------------------------
> Current DC: sl7-01 - partition WITHOUT quorum
> Version: 1.1.12-3e93bc1
> 3 Nodes configured
> 5 Resources configured
> 
> 
> Online: [ sl7-01 ]
> RemoteOnline: [ snmp1 ]
> RemoteOFFLINE: [ snmp2 ]
> 
>  Host-rsc1      (ocf::heartbeat:Dummy): Started sl7-01 
>  Remote-rsc1    (ocf::heartbeat:Dummy): Started snmp1 
>  snmp1  (ocf::pacemaker:remote):        Started sl7-01 
>  snmp2  (ocf::pacemaker:remote):        FAILED sl7-01 
> 
> Node Attributes:
> * Node sl7-01:
> * Node snmp1:
> 
> Migration summary:
> * Node sl7-01: 
>    snmp2: migration-threshold=1 fail-count=1 last-failure='Fri Apr  3 12:56:12 2015'
> * Node snmp1: 
> 
> Failed actions:
>     snmp2_monitor_3000 on sl7-01 'unknown error' (1): call=6, status=Error, exit-reason='none', last-rc-change='Fri Apr  3 12:56:12 2015', queued=0ms, exec=0ms

Ideally we’d have fencing configured and reboot the remote node here.
But for the sake of argument, ok :)


> -------------------------------------------------------------
> 
> Step 3) We reboot pacemaker_remoted which stopped.

As in you reboot the node on which pacemaker_remoted is stopped and pacemaker_remoted is configured to start at boot?

> 
> Step 4) We clear snmp2 of remote by crm_resource command,

Was pacemaker_remoted running at this point?
I mentioned this earlier today, we need to improve the experience in this area.

Probably a good excuse to fix on-fail=ignore for start actions. 

> but remote cannot participate in a cluster.
> -------------------------------------------------------------
> Version: 1.1.12-3e93bc1
> 3 Nodes configured
> 5 Resources configured
> 
> 
> Online: [ sl7-01 ]
> RemoteOnline: [ snmp1 ]
> RemoteOFFLINE: [ snmp2 ]
> 
>  Host-rsc1      (ocf::heartbeat:Dummy): Started sl7-01 
>  Remote-rsc1    (ocf::heartbeat:Dummy): Started snmp1 
>  snmp1  (ocf::pacemaker:remote):        Started sl7-01 
>  snmp2  (ocf::pacemaker:remote):        FAILED sl7-01 
> 
> Node Attributes:
> * Node sl7-01:
> * Node snmp1:
> 
> Migration summary:
> * Node sl7-01: 
>    snmp2: migration-threshold=1 fail-count=1000000 last-failure='Wed Apr  8 11:21:09 2015'
> * Node snmp1: 
> 
> Failed actions:
>     snmp2_start_0 on sl7-01 'unknown error' (1): call=8, status=Timed Out, exit-reason='none', last-rc-change='Wed Apr  8 11:20:11 2015', queued=0ms, exec=0ms
> -------------------------------------------------------------
> 
> 
> Node of pacemaker and the remote node output the following log repeatedly.
> 
> -------------------------------------------------------------
> Apr  8 11:20:38 sl7-01 crmd[17101]: info: crm_remote_tcp_connect_async: Attempting to connect to remote server at 192.168.40.110:3121
> Apr  8 11:20:38 sl7-01 crmd[17101]: info: lrmd_tcp_connect_cb: Remote lrmd client TLS connection established with server snmp2:3121
> Apr  8 11:20:38 sl7-01 crmd[17101]: error: lrmd_tls_recv_reply: Unable to receive expected reply, disconnecting.
> Apr  8 11:20:38 sl7-01 crmd[17101]: error: lrmd_tls_send_recv: Remote lrmd server disconnected while waiting for reply with id 101.
> Apr  8 11:20:38 sl7-01 crmd[17101]: info: lrmd_tls_connection_destroy: TLS connection destroyed
> Apr  8 11:20:38 sl7-01 crmd[17101]: info: lrmd_api_disconnect: Disconnecting from lrmd service
> -------------------------------------------------------------
> Apr  8 11:20:36 snmp2 pacemaker_remoted[1502]:   notice: lrmd_remote_client_destroy: LRMD client disconnecting remote client - name: remote-lrmd-snmp2:3121 id: 8fbbc3cd-daa5-406b-942d-21be868cfc62
> Apr  8 11:20:37 snmp2 pacemaker_remoted[1502]:   notice: lrmd_remote_listen: LRMD client connection established. 0xbb7ca0 id: a59392c9-6575-40ed-9b53-98a68de00409
> Apr  8 11:20:38 snmp2 pacemaker_remoted[1502]:     info: lrmd_remote_client_msg: Client disconnect detected in tls msg dispatcher.
> Apr  8 11:20:38 snmp2 pacemaker_remoted[1502]:   notice: lrmd_remote_client_destroy: LRMD client disconnecting remote client - name: remote-lrmd-snmp2:3121 id: a59392c9-6575-40ed-9b53-98a68de00409
> Apr  8 11:20:39 snmp2 pacemaker_remoted[1502]:   notice: lrmd_remote_listen: LRMD client connection established. 0xbb7ca0 id: 0e58614c-b1c5-4e37-a917-1f8e3de5de24
> Apr  8 11:20:39 snmp2 pacemaker_remoted[1502]:     info: lrmd_remote_client_msg: Client disconnect detected in tls msg dispatcher.
> Apr  8 11:20:39 snmp2 pacemaker_remoted[1502]:   notice: lrmd_remote_client_destroy: LRMD client disconnecting remote client - name: remote-lrmd-snmp2:3121 id: 0e58614c-b1c5-4e37-a917-1f8e3de5de24
> Apr  8 11:20:40 snmp2 pacemaker_remoted[1502]:   notice: lrmd_remote_listen: LRMD client connection established. 0xbb7ca0 id: 518bcca5-5f83-47fb-93ea-2ece33690111
> -------------------------------------------------------------
> 
> Is this movement right?
> 
> Best Regards,
> Hideo Yamauchi.
> 
> 
> 
> ----- Original Message -----
>> From: Ulrich Windl <Ulrich.Windl at rz.uni-regensburg.de>
>> To: users at clusterlabs.org
>> Cc: 
>> Date: 2015/4/2, Thu 22:30
>> Subject: [ClusterLabs] Antw: Re: [Question] About movement of pacemaker_remote.
>> 
>>>>> David Vossel <dvossel at redhat.com> schrieb am 02.04.2015 um 
>> 14:58 in Nachricht
>> <796820123.6644200.1427979523554.JavaMail.zimbra at redhat.com>:
>> 
>>> 
>>> ----- Original Message -----
>>>> 
>>>>> On 14 Mar 2015, at 10:14 am, David Vossel 
>> <dvossel at redhat.com> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>> ----- Original Message -----
>>>>>> 
>>>>>> Failed actions:
>>>>>>      snmp2_start_0 on sl7-01 'unknown error' (1): 
>> call=8, status=Timed Out,
>>>>>>      exit-reason='none', last-rc-change='Thu Mar 12 
>> 14:26:26 2015',
>>>>>>      queued=0ms, exec=0ms
>>>>>>      snmp2_start_0 on sl7-01 'unknown error' (1): 
>> call=8, status=Timed Out,
>>>>>>      exit-reason='none', last-rc-change='Thu Mar 12 
>> 14:26:26 2015',
>>>>>>      queued=0ms, exec=0ms
>>>>>> -----------------------
>>>>> 
>>>>> Pacemaker is attempting to restore connection to the remote node 
>> here, are
>>>>> you
>>>>> sure the remote is accessible? The "Timed Out" error 
>> means that pacemaker
>>>>> was
>>>>> unable to establish the connection during the timeout period.
>>>> 
>>>> Random question: Are we smart enough not to try and start 
>> pacemaker-remote
>>>> resources for node's we've just fenced?
>>> 
>>> we try and re-connect to remote nodes after fencing. if the fence operation
>>> was 'off' instead of 'reboot', this would make no sense. 
>> I'm not entirely 
>>> sure how to handle this. We want the remote-node re-integrated into the 
>>> cluster,
>>> but i'd like to optimize the case where we know the node will not be 
>> coming
>>> back online.
>> 
>> Beware: Even if the fencing action is "off" (for software), a human 
>> may decide to boot the node anyway, also starting the cluster software.
>> 
>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Users mailing list: Users at clusterlabs.org 
>>>> http://clusterlabs.org/mailman/listinfo/users 
>>>> 
>>>> Project Home: http://www.clusterlabs.org 
>>>> Getting started: 
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>>>> Bugs: http://bugs.clusterlabs.org 
>>>> 
>>> 
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org 
>>> http://clusterlabs.org/mailman/listinfo/users 
>>> 
>>> Project Home: http://www.clusterlabs.org 
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>>> Bugs: http://bugs.clusterlabs.org 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>> 
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>> 
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org





More information about the Users mailing list