[ClusterLabs] ClusterIP won't return to recovered node
Dan Ragle
daniel at Biblestuph.com
Sat Jun 10 11:53:22 EDT 2017
On 5/25/2017 5:33 PM, Ken Gaillot wrote:
> On 05/24/2017 12:27 PM, Dan Ragle wrote:
>> I suspect this has been asked before and apologize if so, a google
>> search didn't seem to find anything that was helpful to me ...
>>
>> I'm setting up an active/active two-node cluster and am having an issue
>> where one of my two defined clusterIPs will not return to the other node
>> after it (the other node) has been recovered.
>>
>> I'm running on CentOS 7.3. My resource setups look like this:
>>
>> # cibadmin -Q|grep dc-version
>> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
>> value="1.1.15-11.el7_3.4-e174ec8"/>
>>
>> # pcs resource show PublicIP-clone
>> Clone: PublicIP-clone
>> Meta Attrs: clone-max=2 clone-node-max=2 globally-unique=true
>> interleave=true
>> Resource: PublicIP (class=ocf provider=heartbeat type=IPaddr2)
>> Attributes: ip=75.144.71.38 cidr_netmask=24 nic=bond0
>> Meta Attrs: resource-stickiness=0
>> Operations: start interval=0s timeout=20s (PublicIP-start-interval-0s)
>> stop interval=0s timeout=20s (PublicIP-stop-interval-0s)
>> monitor interval=30s (PublicIP-monitor-interval-30s)
>>
>> # pcs resource show PrivateIP-clone
>> Clone: PrivateIP-clone
>> Meta Attrs: clone-max=2 clone-node-max=2 globally-unique=true
>> interleave=true
>> Resource: PrivateIP (class=ocf provider=heartbeat type=IPaddr2)
>> Attributes: ip=192.168.1.3 nic=bond1 cidr_netmask=24
>> Meta Attrs: resource-stickiness=0
>> Operations: start interval=0s timeout=20s (PrivateIP-start-interval-0s)
>> stop interval=0s timeout=20s (PrivateIP-stop-interval-0s)
>> monitor interval=10s timeout=20s
>> (PrivateIP-monitor-interval-10s)
>>
>> # pcs constraint --full | grep -i publicip
>> start WEB-clone then start PublicIP-clone (kind:Mandatory)
>> (id:order-WEB-clone-PublicIP-clone-mandatory)
>> # pcs constraint --full | grep -i privateip
>> start WEB-clone then start PrivateIP-clone (kind:Mandatory)
>> (id:order-WEB-clone-PrivateIP-clone-mandatory)
>
> FYI These constraints cover ordering only. If you also want to be sure
> that the IPs only start on a node where the web service is functional,
> then you also need colocation constraints.
>
>>
>> When I first create the resources, they split across the two nodes as
>> expected/desired:
>>
>> Clone Set: PublicIP-clone [PublicIP] (unique)
>> PublicIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
>> PublicIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
>> Clone Set: PrivateIP-clone [PrivateIP] (unique)
>> PrivateIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
>> PrivateIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
>> Clone Set: WEB-clone [WEB]
>> Started: [ node1-pcs node2-pcs ]
>>
>> I then put the second node in standby:
>>
>> # pcs node standby node2-pcs
>>
>> And the IPs both jump to node1 as expected:
>>
>> Clone Set: PublicIP-clone [PublicIP] (unique)
>> PublicIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
>> PublicIP:1 (ocf::heartbeat:IPaddr2): Started node1-pcs
>> Clone Set: WEB-clone [WEB]
>> Started: [ node1-pcs ]
>> Stopped: [ node2-pcs ]
>> Clone Set: PrivateIP-clone [PrivateIP] (unique)
>> PrivateIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
>> PrivateIP:1 (ocf::heartbeat:IPaddr2): Started node1-pcs
>>
>> Then unstandby the second node:
>>
>> # pcs node unstandby node2-pcs
>>
>> The publicIP goes back, but the private does not:
>>
>> Clone Set: PublicIP-clone [PublicIP] (unique)
>> PublicIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
>> PublicIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
>> Clone Set: WEB-clone [WEB]
>> Started: [ node1-pcs node2-pcs ]
>> Clone Set: PrivateIP-clone [PrivateIP] (unique)
>> PrivateIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
>> PrivateIP:1 (ocf::heartbeat:IPaddr2): Started node1-pcs
>>
>> Anybody see what I'm doing wrong? I'm not seeing anything in the logs to
>> indicate that it tries node2 and then fails; but I'm fairly new to the
>> software so it's possible I'm not looking in the right place.
>
> The pcs status would show any failed actions, and anything important in
> the logs would start with "error:" or "warning:".
>
> At any given time, one of the nodes is the DC, meaning it schedules
> actions for the whole cluster. That node will have more "pengine:"
> messages in its logs at the time. You can check those logs to see what
> decisions were made, as well as a "saving inputs" message to get the
> cluster state that was used to make those decisions. There is a
> crm_simulate tool that you can run on that file to get more information.
>
> By default, pacemaker will try to balance the number of resources
> running on each node, so I'm not sure why in this case node1 has four
> resources and node2 has two. crm_simulate might help explain it.
>
> However, there's nothing here telling pacemaker that the instances of
> PrivateIP should run on different nodes when possible. With your
> existing constraints, pacemaker would be equally happy to run both
> PublicIP instances on one node and both PrivateIP instances on the other
> node.
Thanks for your reply. Finally getting back to this.
Looking back at my config and my notes I realized I'm guilty of not
giving you enough information. There was indeed an additional pair of
resources that I didn't list in my original output that I didn't think
were relevant to the issue--my bad. Reading what you wrote made me
realize that it does appear as though pacemaker is simply trying to
balance the overall load of *all* the available resources.
But I'm still confused as to how one would definitively correct the
issue. I tried this full reduction this morning. Starting from an empty
two-node cluster (no resources, no constraints):
[root at node1 clustertest]# pcs status
Cluster name: MyCluster
Stack: corosync
Current DC: NONE
Last updated: Sat Jun 10 10:58:46 2017 Last change: Sat Jun 10
10:40:23 2017 by root via cibadmin on node1-pcs
2 nodes and 0 resources configured
OFFLINE: [ node1-pcs node2-pcs ]
No resources
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
[root at node1 clustertest]# pcs resource create ClusterIP
ocf:heartbeat:IPaddr2 ip=1.2.3.4 nic=bond0 cidr_netmask=24
[root at node1 clustertest]# pcs resource meta ClusterIP
resource-stickiness=0
[root at node1 clustertest]# pcs resource clone ClusterIP clone-max=2
clone-node-max=2 globally-unique=true interleave=true
[root at node1 clustertest]# pcs resource create Test1 systemd:vtest1
[root at node1 clustertest]# pcs resource create Test2 systemd:vtest2
[root at node1 clustertest]# pcs constraint location Test1 prefers
node1-pcs=INFINITY
[root at node1 clustertest]# pcs constraint location Test2 prefers
node1-pcs=INFINITY
[root at node1 clustertest]# pcs node standby node1-pcs
[root at node1 clustertest]# pcs status
Cluster name: MyCluster
Stack: corosync
Current DC: node1-pcs (version 1.1.15-11.el7_3.4-e174ec8) - partition
with quorum
Last updated: Sat Jun 10 11:01:07 2017 Last change: Sat Jun 10
11:00:59 2017 by root via crm_attribute on node1-pcs
2 nodes and 4 resources configured
Node node1-pcs: standby
Online: [ node2-pcs ]
Full list of resources:
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node2-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
Test1 (systemd:vtest1): Started node2-pcs
Test2 (systemd:vtest2): Started node2-pcs
Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
[root at node1 clustertest]# pcs node unstandby node1-pcs
[root at node1 clustertest]# pcs status resources
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node2-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node1-pcs
Test1 (systemd:vtest1): Started node1-pcs
Test2 (systemd:vtest2): Started node1-pcs
[root at node1 clustertest]# pcs node standby node2-pcs
[root at node1 clustertest]# pcs status resources
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node1-pcs
Test1 (systemd:vtest1): Started node1-pcs
Test2 (systemd:vtest2): Started node1-pcs
[root at node1 clustertest]# pcs node unstandby node2-pcs
[root at node1 clustertest]# pcs status resources
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
Test1 (systemd:vtest1): Started node1-pcs
Test2 (systemd:vtest2): Started node1-pcs
[root at node1 clustertest]# pcs resource delete ClusterIP
Attempting to stop: ClusterIP...Stopped
[root at node1 clustertest]# pcs resource create ClusterIP
ocf:heartbeat:IPaddr2 ip=1.2.3.4 nic=bond0 cidr_netmask=24
[root at node1 clustertest]# pcs resource meta ClusterIP
resource-stickiness=0
[root at node1 clustertest]# pcs resource clone ClusterIP clone-max=2
clone-node-max=2 globally-unique=true interleave=true
[root at node1 clustertest]# pcs status resources
Test1 (systemd:vtest1): Started node1-pcs
Test2 (systemd:vtest2): Started node1-pcs
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node2-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
[root at node1 clustertest]# pcs node standby node1-pcs
[root at node1 clustertest]# pcs status resources
Test1 (systemd:vtest1): Started node2-pcs
Test2 (systemd:vtest2): Started node2-pcs
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node2-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
[root at node1 clustertest]# pcs node unstandby node1-pcs
[root at node1 clustertest]# pcs status resources
Test1 (systemd:vtest1): Started node1-pcs
Test2 (systemd:vtest2): Started node1-pcs
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node2-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
[root at node1 clustertest]# pcs node standby node2-pcs
[root at node1 clustertest]# pcs status resources
Test1 (systemd:vtest1): Started node1-pcs
Test2 (systemd:vtest2): Started node1-pcs
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node1-pcs
[root at node1 clustertest]# pcs node unstandby node2-pcs
[root at node1 clustertest]# pcs status resources
Test1 (systemd:vtest1): Started node1-pcs
Test2 (systemd:vtest2): Started node1-pcs
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node1-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
[root at node1 clustertest]# pcs node standby node1-pcs
[root at node1 clustertest]# pcs status resources
Test1 (systemd:vtest1): Started node2-pcs
Test2 (systemd:vtest2): Started node2-pcs
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node2-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
[root at node1 clustertest]# pcs node unstandby node1-pcs
[root at node1 clustertest]# pcs status resources
Test1 (systemd:vtest1): Started node1-pcs
Test2 (systemd:vtest2): Started node1-pcs
Clone Set: ClusterIP-clone [ClusterIP] (unique)
ClusterIP:0 (ocf::heartbeat:IPaddr2): Started node2-pcs
ClusterIP:1 (ocf::heartbeat:IPaddr2): Started node2-pcs
So in the initial configuration, it works as expected; putting the nodes
in standby one at a time (I waited at least 5 seconds between each
standby/unstandby operation) and then restoring the nodes shows the
ClusterIP bouncing back and forth as expected. But then after deleting
the ClusterIP resource and recreating it exactly as it originally was
the clones initially both stay on one node (the one the test resources
are not on). Putting the node the extra resources are on in standby and
then restoring it the IPs stay on the other node. Putting the node the
extra resources are *not* on in standby and then restoring that node
allows the IPs to split once again.
I also did the test above with full pcs status displays after each
standby/unstandby; there were no errors displayed at each step.
So I guess my bottom line question is: How does one tell Pacemaker that
the individual legs of globally unique clones should *always* be spread
across the available nodes whenever possible, regardless of the number
of processes on any one of the nodes? For kicks I did try:
pcs constraint location ClusterIP:0 prefers node1-pcs=INFINITY
but it responded with an error about an invalid character (:).
Thanks,
Dan
>
> I think you could probably get what you want by putting an optional
> (<INFINITY) colocation preference between PrivateIP and PublicIP. The
> only way pacemaker could satisfy that would be to run one of each on
> each node.
>
>> Also, I noticed when putting a node in standby the main NIC appears to
>> be interrupted momentarily (long enough for my SSH session, which is
>> connected via the permanent IP on the NIC and not the clusterIP, to be
>> dropped). Is there any way to avoid this? I was thinking that the
>> cluster operations would only affect the ClusteIP and not the other IPs
>> being served on that NIC.
>
> Nothing in the cluster should cause that behavior. Check all the system
> logs around the time to see if anything unusual is reported.
>
>>
>> Thanks!
>>
>> Dan
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
More information about the Users
mailing list