[ClusterLabs] Antw: Re: Antw: Re: [Question] About movement of pacemaker_remote.

renayama19661014 at ybb.ne.jp renayama19661014 at ybb.ne.jp
Tue May 12 02:12:55 UTC 2015


Hi All,

The problem is like a buffer becoming NULL after crm_resouce -C practice somehow or other after having rebooted remote node.

I incorporated log in a source code and confirmed it.

------------------------------------------------
crm_remote_recv_once(crm_remote_t * remote)
{
(snip)
    /* automatically grow the buffer when needed */
    if(remote->buffer_size < read_len) {
           remote->buffer_size = 2 * read_len;
        crm_trace("Expanding buffer to %u bytes", remote->buffer_size);

        remote->buffer = realloc_safe(remote->buffer, remote->buffer_size + 1);
        CRM_ASSERT(remote->buffer != NULL);
    }

#ifdef HAVE_GNUTLS_GNUTLS_H
    if (remote->tls_session) {
        if (remote->buffer == NULL) {
        crm_info("### YAMAUCHI buffer is NULL [buffer_zie[%d] readlen[%d]", remote->buffer_size, read_len);
        }
        rc = gnutls_record_recv(*(remote->tls_session),
                                remote->buffer + remote->buffer_offset,
                                remote->buffer_size - remote->buffer_offset);
(snip)
------------------------------------------------

May 12 10:54:01 sl7-01 crmd[30447]: info: crm_remote_recv_once: ### YAMAUCHI buffer is NULL [buffer_zie[1326] readlen[40]
May 12 10:54:02 sl7-01 crmd[30447]: info: crm_remote_recv_once: ### YAMAUCHI buffer is NULL [buffer_zie[1326] readlen[40]
May 12 10:54:04 sl7-01 crmd[30447]: info: crm_remote_recv_once: ### YAMAUCHI buffer is NULL [buffer_zie[1326] readlen[40]

------------------------------------------------

gnutls_record_recv processes an empty buffer and becomes the error.

------------------------------------------------
(snip)
ssize_t
_gnutls_recv_int(gnutls_session_t session, content_type_t type,
 gnutls_handshake_description_t htype,
 gnutls_packet_t *packet,
 uint8_t * data, size_t data_size, void *seq,
 unsigned int ms)
{
int ret;

if (packet == NULL && (type != GNUTLS_ALERT && type != GNUTLS_HEARTBEAT)
    && (data_size == 0 || data == NULL))
return gnutls_assert_val(GNUTLS_E_INVALID_REQUEST);

(sip)
ssize_t
gnutls_record_recv(gnutls_session_t session, void *data, size_t data_size)
{
return _gnutls_recv_int(session, GNUTLS_APPLICATION_DATA, -1, NULL,
data, data_size, NULL,
session->internals.record_timeout_ms);
}
(snip)
------------------------------------------------

Best Regards,
Hideo Yamauchi.



----- Original Message -----
> From: "renayama19661014 at ybb.ne.jp" <renayama19661014 at ybb.ne.jp>
> To: "users at clusterlabs.org" <users at clusterlabs.org>
> Cc: 
> Date: 2015/5/11, Mon 16:45
> Subject: Re: [ClusterLabs] Antw: Re: Antw: Re: [Question] About movement of pacemaker_remote.
> 
> Hi Ulrich,
> 
> Thank you for comments.
> 
>>  So your host and you resource are both named "snmp1"? I also 
> don't 
>>  have much experience with cleaning up resources for a node that is offline. 
> What 
>>  change should it make (while the node is offline)?
> 
> 
> The name of the remote resource and the name of the remote node make same 
> "snmp1".
> 
> 
> (snip)
> primitive snmp1 ocf:pacemaker:remote \
>         params \
>                 server="snmp1" \
>         op start interval="0s" timeout="60s" 
> on-fail="ignore" \
>         op monitor interval="3s" timeout="15s" \
>         op stop interval="0s" timeout="60s" 
> on-fail="ignore"
> 
> primitive Host-rsc1 ocf:heartbeat:Dummy \
>         op start interval="0s" timeout="60s" 
> on-fail="restart" \
>         op monitor interval="10s" timeout="60s" 
> on-fail="restart" \
>         op stop interval="0s" timeout="60s" 
> on-fail="ignore"
> 
> primitive Remote-rsc1 ocf:heartbeat:Dummy \
>         op start interval="0s" timeout="60s" 
> on-fail="restart" \
>         op monitor interval="10s" timeout="60s" 
> on-fail="restart" \
>         op stop interval="0s" timeout="60s" 
> on-fail="ignore"
> 
> location loc1 Remote-rsc1 \
>         rule 200: #uname eq snmp1
> location loc3 Host-rsc1 \
>         rule 200: #uname eq bl460g8n1
> (snip)
> 
> The pacemaker_remoted of the snmp1 node stops in SIGTERM.
> I reboot pacemaker_remoted of the snmp1 node afterwards.
> And I execute crm_resource command, but the snmp1 node remains off-line.
> 
> After having executed crm_resource command, the remote node thinks that it is 
> right movement to become the snmp1 online.
> 
> 
> 
> Best Regards,
> Hideo Yamauchi.
> 
> 
> 
> 
> 
> ----- Original Message -----
>>  From: Ulrich Windl <Ulrich.Windl at rz.uni-regensburg.de>
>>  To: users at clusterlabs.org; renayama19661014 at ybb.ne.jp
>>  Cc: 
>>  Date: 2015/5/11, Mon 15:39
>>  Subject: Antw: Re: [ClusterLabs] Antw: Re: [Question] About movement of 
> pacemaker_remote.
>> 
>>>>>   <renayama19661014 at ybb.ne.jp> schrieb am 11.05.2015 um 
> 06:22 
>>  in Nachricht
>>  <361916.15877.qm at web200006.mail.kks.yahoo.co.jp>:
>>>   Hi All,
>>> 
>>>   I matched the OS version of the remote node with a host once again and 
> 
>>>   confirmed it in Pacemaker1.1.13-rc2.
>>> 
>>>   It was the same even if I made a host RHEL7.1.(bl460g8n1)
>>>   I made the remote host RHEL7.1.(snmp1)
>>> 
>>>   The first crm_resource -C fails.
>>>   --------------------------------
>>>   [root at bl460g8n1 ~]# crm_resource -C -r snmp1
>>>   Cleaning up snmp1 on bl460g8n1
>>>   Waiting for 1 replies from the CRMd. OK
>>> 
>>>   [root at bl460g8n1 ~]# crm_mon -1 -Af
>>>   Last updated: Mon May 11 12:44:31 2015
>>>   Last change: Mon May 11 12:43:30 2015
>>>   Stack: corosync
>>>   Current DC: bl460g8n1 - partition WITHOUT quorum
>>>   Version: 1.1.12-7a2e3ae
>>>   2 Nodes configured
>>>   3 Resources configured
>>> 
>>> 
>>>   Online: [ bl460g8n1 ]
>>>   RemoteOFFLINE: [ snmp1 ]
>> 
>>  So your host and you resource are both named "snmp1"? I also 
> don't 
>>  have much experience with cleaning up resources for a node that is offline. 
> What 
>>  change should it make (while the node is offline)?
>> 
>>> 
>>>    Host-rsc1      (ocf::heartbeat:Dummy): Started bl460g8n1
>>>    Remote-rsc1    (ocf::heartbeat:Dummy): Started bl460g8n1 (failure 
> ignored)
>>> 
>>>   Node Attributes:
>>>   * Node bl460g8n1:
>>>       + ringnumber_0                      : 192.168.101.21 is UP
>>>       + ringnumber_1                      : 192.168.102.21 is UP
>>> 
>>>   Migration summary:
>>>   * Node bl460g8n1:
>>>      snmp1: migration-threshold=1 fail-count=1000000 
> last-failure='Mon 
>>  May 11 
>>>   12:44:28 2015'
>>> 
>>>   Failed actions:
>>>       snmp1_start_0 on bl460g8n1 'unknown error' (1): call=5, 
>>  status=Timed 
>>>   Out, exit-reason='none', last-rc-change='Mon May 11 
> 12:43:31 
>>  2015', queued=0ms, 
>>>   exec=0ms
>>>   --------------------------------
>>> 
>>> 
>>>   The second crm_resource -C succeeded and was connected to the remote 
> host.
>> 
>>  Then the node was online it seems.
>> 
>>  Regards,
>>  Ulrich
>> 
>>>   --------------------------------
>>>   [root at bl460g8n1 ~]# crm_mon -1 -Af
>>>   Last updated: Mon May 11 12:44:54 2015
>>>   Last change: Mon May 11 12:44:48 2015
>>>   Stack: corosync
>>>   Current DC: bl460g8n1 - partition WITHOUT quorum
>>>   Version: 1.1.12-7a2e3ae
>>>   2 Nodes configured
>>>   3 Resources configured
>>> 
>>> 
>>>   Online: [ bl460g8n1 ]
>>>   RemoteOnline: [ snmp1 ]
>>> 
>>>    Host-rsc1      (ocf::heartbeat:Dummy): Started bl460g8n1
>>>    Remote-rsc1    (ocf::heartbeat:Dummy): Started snmp1
>>>    snmp1  (ocf::pacemaker:remote):        Started bl460g8n1
>>> 
>>>   Node Attributes:
>>>   * Node bl460g8n1:
>>>       + ringnumber_0                      : 192.168.101.21 is UP
>>>       + ringnumber_1                      : 192.168.102.21 is UP
>>>   * Node snmp1:
>>> 
>>>   Migration summary:
>>>   * Node bl460g8n1:
>>>   * Node snmp1:
>>>   --------------------------------
>>> 
>>>   The gnutls of a host and the remote node was the next version.
>>> 
>>>   gnutls-devel-3.3.8-12.el7.x86_64
>>>   gnutls-dane-3.3.8-12.el7.x86_64
>>>   gnutls-c++-3.3.8-12.el7.x86_64
>>>   gnutls-3.3.8-12.el7.x86_64
>>>   gnutls-utils-3.3.8-12.el7.x86_64
>>> 
>>> 
>>>   Best Regards,
>>>   Hideo Yamauchi.
>>> 
>>> 
>>> 
>>> 
>>>   ----- Original Message -----
>>>>   From: "renayama19661014 at ybb.ne.jp" 
>>  <renayama19661014 at ybb.ne.jp>
>>>>   To: Cluster Labs - All topics related to open-source clustering 
>>  welcomed 
>>>   <users at clusterlabs.org>
>>>>   Cc: 
>>>>   Date: 2015/4/28, Tue 14:06
>>>>   Subject: Re: [ClusterLabs] Antw: Re: [Question] About movement of 
>>>   pacemaker_remote.
>>>> 
>>>>   Hi David,
>>>> 
>>>>   Even if the result changed the remote node to RHEL7.1, it was the 
> same.
>>>> 
>>>> 
>>>>   I try it with a host node of pacemaker as RHEL7.1 this time.
>>>> 
>>>> 
>>>>   I noticed an interesting phenomenon.
>>>>   The remote node fails in a reconnection in the first crm_resource.
>>>>   However, the remote node succeeds in a reconnection in the second 
>>>   crm_resource.
>>>> 
>>>>   I think that I have some problem with the point where I cut the 
>>  connection 
>>>   with 
>>>>   the remote node first.
>>>> 
>>>>   Best Regards,
>>>>   Hideo Yamauchi.
>>>> 
>>>> 
>>>>   ----- Original Message -----
>>>>>    From: "renayama19661014 at ybb.ne.jp" 
>>>>   <renayama19661014 at ybb.ne.jp>
>>>>>    To: Cluster Labs - All topics related to open-source 
> clustering 
>>  welcomed 
>>>>   <users at clusterlabs.org>
>>>>>    Cc: 
>>>>>    Date: 2015/4/28, Tue 11:52
>>>>>    Subject: Re: [ClusterLabs] Antw: Re: [Question] About 
> movement of 
>>>>   pacemaker_remote.
>>>>> 
>>>>>    Hi David,
>>>>>    Thank you for comments.
>>>>>>    At first glance this looks gnutls related.  GNUTLS is 
>>  returning -50 
>>>>   during 
>>>>>    receive
>>>>> 
>>>>>>    on the client side (pacemaker's side). -50 maps to 
>>  'invalid 
>>>>>    request'. >debug: crm_remote_recv_once:     TLS 
> receive 
>>  failed: The 
>>>>>    request is invalid. >We treat this error as fatal and 
> destroy 
>>  the 
>>>>   connection. 
>>>>>    I've never encountered
>>>>>>    this error and I don't know what causes it. It's 
>>  possible 
>>>>>    there's a bug in
>>>>>>    our gnutls usage... it's also possible there's a 
> bug 
>>  in the 
>>>>   version 
>>>>>    of gnutls
>>>>>>    that is in use as well. 
>>>>>    We built the remote node in RHEL6.5.
>>>>>    Because it may be a problem of gnutls, I confirm it in 
> RHEL7.1.
>>>>> 
>>>>>    Best Regards,
>>>>>    Hideo Yamauchi.
>>>>> 
>>>>>    _______________________________________________
>>>>>    Users mailing list: Users at clusterlabs.org 
>>>>>    http://clusterlabs.org/mailman/listinfo/users 
>>>>> 
>>>>>    Project Home: http://www.clusterlabs.org 
>>>>>    Getting started: 
>>  http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>>>>>    Bugs: http://bugs.clusterlabs.org 
>>>>> 
>>>> 
>>>>   _______________________________________________
>>>>   Users mailing list: Users at clusterlabs.org 
>>>>   http://clusterlabs.org/mailman/listinfo/users 
>>>> 
>>>>   Project Home: http://www.clusterlabs.org 
>>>>   Getting started: 
>>  http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>>>>   Bugs: http://bugs.clusterlabs.org 
>>>> 
>>> 
>>>   _______________________________________________
>>>   Users mailing list: Users at clusterlabs.org 
>>>   http://clusterlabs.org/mailman/listinfo/users 
>>> 
>>>   Project Home: http://www.clusterlabs.org 
>>>   Getting started: 
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
>>>   Bugs: http://bugs.clusterlabs.org 
>> 
> 
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 




More information about the Users mailing list