[Pacemaker] Trying to figure out a constraint

Digimer lists at alteeve.ca
Thu Jun 19 04:06:44 UTC 2014


On 18/06/14 11:42 PM, Digimer wrote:
> On 18/06/14 12:47 AM, Andrew Beekhof wrote:
>>
>> On 18 Jun 2014, at 2:03 pm, Digimer <lists at alteeve.ca> wrote:
>>
>>> Hi all,
>>>
>>>   I am trying to setup a basic pacemaker 1.1.10 on RHEL 6.5 with DRBD
>>> 8.3.16.
>>>
>>>   I've setup DRBD and configured one clustered LVM volume group using
>>> that drbd resource as the PV. With DRBD configured alone, I can
>>> stop/start pacemaker repeatedly without issue. However, when I add
>>> the LVM VG using ocf:heartbeat:LVM and setup a constraint, subsequent
>>> restarts of pacemaker almost always end up with a fence. I have to
>>> think then that I am messing up my constraints...
>>
>> find out who is calling stonith_admin:
>>
>> Jun 17 23:56:06 an-a04n01 kernel: block drbd0: helper command:
>> /sbin/drbdadm fence-peer minor-0
>> Jun 17 23:56:07 an-a04n01 kernel: block drbd0: Handshake successful:
>> Agreed network protocol version 97
>> Jun 17 23:56:07 an-a04n01 stonith_admin[28637]:   notice:
>> crm_log_args: Invoked: stonith_admin --fence an-a04n02.alteeve.ca
>> Jun 17 23:56:07 an-a04n01 stonith-ng[28356]:   notice: handle_request:
>> Client stonith_admin.28637.6ed13ba6 wants to fence (off)
>> 'an-a04n02.alteeve.ca' with device '(any)'
>>
>> Double check fence_pcmk includes "--tag cman" as an argument to
>> stonith_admin (since that will rule it out as a source).
>> Could drbd be initiating it?
>
> Following up on #linux-ha discussion...
>
> DRBD was triggering this. When I used
> /usr/lib/drbd/stonith_admin-fence-peer.sh, I saw both nodes sit at
> 'WFConnection' before it fenced, which normally tells me there is a
> network issue. However manually start DRBD never had a problem, and
> after the fenced node comes back up, starting pacemaker causes it to
> start fine.
>
> So I decided to try '/usr/lib/drbd/crm-fence-peer.sh' instead. Now,
> instead of fencing, an-a04n02 (node 2) fails to promote. However, if I
> try running:
>
> pcs resource debug-start drbd_r0
>
> I get:
>
> Operation start for drbd_r0:0 (ocf:linbit:drbd) returned 0
>   >  stdout:         allow-two-primaries;
>   >  stdout:
>   >  stdout:
>   >  stderr: WARNING: You may be disappointed: This RA is intended for
> pacemaker 1.0 or better!
>   >  stderr: DEBUG: r0: Calling drbdadm -c /etc/drbd.conf adjust r0
>   >  stderr: DEBUG: r0: Exit code 0
>   >  stderr: DEBUG: r0: Command output:
>   >  stderr: DEBUG: r0: Calling /usr/sbin/crm_master -Q -l reboot -v 10000
>   >  stderr: DEBUG: r0: Exit code 0
>   >  stderr: DEBUG: r0: Command output:
>
> This seems to be that it thinks it will work. However, the cluster is
> left at:
>
> Cluster name: an-anvil-04
> Last updated: Wed Jun 18 23:37:43 2014
> Last change: Wed Jun 18 23:19:24 2014 via cibadmin on an-a04n02.alteeve.ca
> Stack: cman
> Current DC: an-a04n01.alteeve.ca - partition with quorum
> Version: 1.1.10-14.el6_5.3-368c726
> 2 Nodes configured
> 4 Resources configured
>
>
> Online: [ an-a04n01.alteeve.ca an-a04n02.alteeve.ca ]
>
> Full list of resources:
>
>   fence_n01_ipmi    (stonith:fence_ipmilan):    Started
> an-a04n01.alteeve.ca
>   fence_n02_ipmi    (stonith:fence_ipmilan):    Started
> an-a04n02.alteeve.ca
>   Master/Slave Set: drbd_r0_Clone [drbd_r0]
>       Masters: [ an-a04n01.alteeve.ca ]
>       Slaves: [ an-a04n02.alteeve.ca ]
>
> Then I check that constraints, I see that this has been created:
>
> Location Constraints:
>    Resource: drbd_r0_Clone
>      Constraint: drbd-fence-by-handler-r0-drbd_r0_Clone
>        Rule: score=-INFINITY role=Master
> (id:drbd-fence-by-handler-r0-rule-drbd_r0_Clone)
>          Expression: #uname ne an-a04n01.alteeve.ca
> (id:drbd-fence-by-handler-r0-expr-drbd_r0_Clone)
> Ordering Constraints:
> Colocation Constraints:
>
> If I delete the constraint, node 2 suddenly promotes properly.
>
> So I have to conclude that, for some reason, the way I am using
> ocf:linbit:drbd is wrong, or I've not configured
> '/etc/drbd.d/global_common.conf' properly.
>
> Speaking of which, that is:
>
> # /etc/drbd.conf
> common {
>      protocol               C;
>      net {
>          allow-two-primaries;
>          after-sb-0pri    discard-zero-changes;
>          after-sb-1pri    discard-secondary;
>          after-sb-2pri    disconnect;
>      }
>      disk {
>          fencing          resource-and-stonith;
>      }
>      syncer {
>          rate             40M;
>      }
>      handlers {
>          fence-peer       /usr/lib/drbd/crm-fence-peer.sh;
>      }
> }
>
> # resource r0 on an-a04n01.alteeve.ca: not ignored, not stacked
> resource r0 {
>      on an-a04n01.alteeve.ca {
>          device           /dev/drbd0 minor 0;
>          disk             /dev/sda5;
>          address          ipv4 10.10.40.1:7788;
>          meta-disk        internal;
>      }
>      on an-a04n02.alteeve.ca {
>          device           /dev/drbd0 minor 0;
>          disk             /dev/sda5;
>          address          ipv4 10.10.40.2:7788;
>          meta-disk        internal;
>      }
> }
>
> # resource r1 on an-a04n01.alteeve.ca: not ignored, not stacked
> resource r1 {
>      on an-a04n01.alteeve.ca {
>          device           /dev/drbd1 minor 1;
>          disk             /dev/sda6;
>          address          ipv4 10.10.40.1:7789;
>          meta-disk        internal;
>      }
>      on an-a04n02.alteeve.ca {
>          device           /dev/drbd1 minor 1;
>          disk             /dev/sda6;
>          address          ipv4 10.10.40.2:7789;
>          meta-disk        internal;
>      }
> }
>
> Note that, for the time being, I've not configured r1 in pacemaker to
> simplify the config while debugging.
>
> Attached is the crm_report, hopefully it might shed some light on what I
> am doing wrong.
>
> Thanks!
>
> digimer

After sending this, I found that adding:

handlers {
	fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
	after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";
}

Allowed the constraint to be removed, so eventually node 2 (an-a04n02) 
eventually promoted, but not before going into the failed state shown above.

Subsequent stop -> start of pacemaker on both nodes started cleanly, not 
fence action reported in /var/log/messages. I notices this time that the 
drbd module was loaded, not sure if that made a difference.

Will keep testing... Any insight is much appreciated.
-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without 
access to education?




More information about the Pacemaker mailing list