<div dir="ltr">Thanks Ken for the detailed response.<div>I suppose I could even use some of the pcs/crm CLI commands then.</div><div>Cheers.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 16, 2016 at 8:27 PM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 03/16/2016 05:22 AM, Nikhil Utane wrote:<br>
> I see following info gets updated in CIB. Can I use this or there is better<br>
> way?<br>
><br>
</span>> <node_state id="*node1*" uname="node1" in_ccm="false" crmd="offline"<br>
> crm-debug-origin="peer_update_callback" join="*down*" expected="member"><br>
<br>
in_ccm/crmd/join reflect the current state of the node (as known by the<br>
partition that you're looking at the CIB on), so if the node went down<br>
and came back up, it won't tell you anything about being down.<br>
<br>
- in_ccm indicates that the node is part of the underlying cluster layer<br>
(heartbeat/cman/corosync)<br>
<br>
- crmd indicates that the node is communicating at the pacemaker layer<br>
<br>
- join indicates what phase of the join process the node is at<br>
<br>
There's not a direct way to see what node went down after the fact.<br>
There are ways however:<br>
<br>
- if the node was running resources, those will be failed, and those<br>
failures (including node) will be shown in the cluster status<br>
<br>
- the logs show all node membership events; you can search for logs such<br>
as "state is now lost" and "left us"<br>
<br>
- "stonith -H $NODE_NAME" will show the fence history for a given node,<br>
so if the node went down due to fencing, it will show up there<br>
<br>
- you can configure an ocf:pacemaker:ClusterMon resource to run crm_mon<br>
periodically and run a script for node events, and you can write the<br>
script to do whatever you want (email you, etc.) (in the upcoming 1.1.15<br>
release, built-in notifications will make this more reliable and easier,<br>
but any script you use with ClusterMon will still be usable with the new<br>
method)<br>
<div class="HOEnZb"><div class="h5"><br>
> On Wed, Mar 16, 2016 at 12:40 PM, Nikhil Utane <<a href="mailto:nikhil.subscribed@gmail.com">nikhil.subscribed@gmail.com</a>><br>
> wrote:<br>
><br>
>> Hi Ken,<br>
>><br>
>> Sorry about the long delay. This activity was de-focussed but now it's<br>
>> back on track.<br>
>><br>
>> One part of question that is still not answered is on the newly active<br>
>> node, how to find out which was the node that went down?<br>
>> Anything that gets updated in the status section that can be read and<br>
>> figured out?<br>
>><br>
>> Thanks.<br>
>> Nikhil<br>
>><br>
>> On Sat, Jan 9, 2016 at 3:31 AM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>> wrote:<br>
>><br>
>>> On 01/08/2016 11:13 AM, Nikhil Utane wrote:<br>
>>>>> I think stickiness will do what you want here. Set a stickiness higher<br>
>>>>> than the original node's preference, and the resource will want to stay<br>
>>>>> where it is.<br>
>>>><br>
>>>> Not sure I understand this. Stickiness will ensure that resources don't<br>
>>>> move back when original node comes back up, isn't it?<br>
>>>> But in my case, I want the newly standby node to become the backup node<br>
>>> for<br>
>>>> all other nodes. i.e. it should now be able to run all my resource<br>
>>> groups<br>
>>>> albeit with a lower score. How do I achieve that?<br>
>>><br>
>>> Oh right. I forgot to ask whether you had an opt-out<br>
>>> (symmetric-cluster=true, the default) or opt-in<br>
>>> (symmetric-cluster=false) cluster. If you're opt-out, every node can run<br>
>>> every resource unless you give it a negative preference.<br>
>>><br>
>>> Partly it depends on whether there is a good reason to give each<br>
>>> instance a "home" node. Often, there's not. If you just want to balance<br>
>>> resources across nodes, the cluster will do that by default.<br>
>>><br>
>>> If you prefer to put certain resources on certain nodes because the<br>
>>> resources require more physical resources (RAM/CPU/whatever), you can<br>
>>> set node attributes for that and use rules to set node preferences.<br>
>>><br>
>>> Either way, you can decide whether you want stickiness with it.<br>
>>><br>
>>>> Also can you answer, how to get the values of node that goes active and<br>
>>> the<br>
>>>> node that goes down inside the OCF agent? Do I need to use<br>
>>> notification or<br>
>>>> some simpler alternative is available?<br>
>>>> Thanks.<br>
>>>><br>
>>>><br>
>>>> On Fri, Jan 8, 2016 at 9:30 PM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
>>> wrote:<br>
>>>><br>
>>>>> On 01/08/2016 06:55 AM, Nikhil Utane wrote:<br>
>>>>>> Would like to validate my final config.<br>
>>>>>><br>
>>>>>> As I mentioned earlier, I will be having (upto) 5 active servers and 1<br>
>>>>>> standby server.<br>
>>>>>> The standby server should take up the role of active that went down.<br>
>>> Each<br>
>>>>>> active has some unique configuration that needs to be preserved.<br>
>>>>>><br>
>>>>>> 1) So I will create total 5 groups. Each group has a<br>
>>> "heartbeat::IPaddr2<br>
>>>>>> resource (for virtual IP) and my custom resource.<br>
>>>>>> 2) The virtual IP needs to be read inside my custom OCF agent, so I<br>
>>> will<br>
>>>>>> make use of attribute reference and point to the value of IPaddr2<br>
>>> inside<br>
>>>>> my<br>
>>>>>> custom resource to avoid duplication.<br>
>>>>>> 3) I will then configure location constraint to run the group resource<br>
>>>>> on 5<br>
>>>>>> active nodes with higher score and lesser score on standby.<br>
>>>>>> For e.g.<br>
>>>>>> Group Node Score<br>
>>>>>> ---------------------------------------------<br>
>>>>>> MyGroup1 node1 500<br>
>>>>>> MyGroup1 node6 0<br>
>>>>>><br>
>>>>>> MyGroup2 node2 500<br>
>>>>>> MyGroup2 node6 0<br>
>>>>>> ..<br>
>>>>>> MyGroup5 node5 500<br>
>>>>>> MyGroup5 node6 0<br>
>>>>>><br>
>>>>>> 4) Now if say node1 were to go down, then stop action on node1 will<br>
>>> first<br>
>>>>>> get called. Haven't decided if I need to do anything specific here.<br>
>>>>><br>
>>>>> To clarify, if node1 goes down intentionally (e.g. standby or stop),<br>
>>>>> then all resources on it will be stopped first. But if node1 becomes<br>
>>>>> unavailable (e.g. crash or communication outage), it will get fenced.<br>
>>>>><br>
>>>>>> 5) But when the start action of node 6 gets called, then using crm<br>
>>>>> command<br>
>>>>>> line interface, I will modify the above config to swap node 1 and<br>
>>> node 6.<br>
>>>>>> i.e.<br>
>>>>>> MyGroup1 node6 500<br>
>>>>>> MyGroup1 node1 0<br>
>>>>>><br>
>>>>>> MyGroup2 node2 500<br>
>>>>>> MyGroup2 node1 0<br>
>>>>>><br>
>>>>>> 6) To do the above, I need the newly active and newly standby node<br>
>>> names<br>
>>>>> to<br>
>>>>>> be passed to my start action. What's the best way to get this<br>
>>> information<br>
>>>>>> inside my OCF agent?<br>
>>>>><br>
>>>>> Modifying the configuration from within an agent is dangerous -- too<br>
>>>>> much potential for feedback loops between pacemaker and the agent.<br>
>>>>><br>
>>>>> I think stickiness will do what you want here. Set a stickiness higher<br>
>>>>> than the original node's preference, and the resource will want to stay<br>
>>>>> where it is.<br>
>>>>><br>
>>>>>> 7) Apart from node name, there will be other information which I plan<br>
>>> to<br>
>>>>>> pass by making use of node attributes. What's the best way to get this<br>
>>>>>> information inside my OCF agent? Use crm command to query?<br>
>>>>><br>
>>>>> Any of the command-line interfaces for doing so should be fine, but I'd<br>
>>>>> recommend using one of the lower-level tools (crm_attribute or<br>
>>>>> attrd_updater) so you don't have a dependency on a higher-level tool<br>
>>>>> that may not always be installed.<br>
>>>>><br>
>>>>>> Thank You.<br>
>>>>>><br>
>>>>>> On Tue, Dec 22, 2015 at 9:44 PM, Nikhil Utane <<br>
>>>>> <a href="mailto:nikhil.subscribed@gmail.com">nikhil.subscribed@gmail.com</a>><br>
>>>>>> wrote:<br>
>>>>>><br>
>>>>>>> Thanks to you Ken for giving all the pointers.<br>
>>>>>>> Yes, I can use service start/stop which should be a lot simpler.<br>
>>> Thanks<br>
>>>>>>> again. :)<br>
>>>>>>><br>
>>>>>>> On Tue, Dec 22, 2015 at 9:29 PM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
>>>>> wrote:<br>
>>>>>>><br>
>>>>>>>> On 12/22/2015 12:17 AM, Nikhil Utane wrote:<br>
>>>>>>>>> I have prepared a write-up explaining my requirements and current<br>
>>>>>>>> solution<br>
>>>>>>>>> that I am proposing based on my understanding so far.<br>
>>>>>>>>> Kindly let me know if what I am proposing is good or there is a<br>
>>> better<br>
>>>>>>>> way<br>
>>>>>>>>> to achieve the same.<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>><br>
>>>>><br>
>>> <a href="https://drive.google.com/file/d/0B0zPvL-Tp-JSTEJpcUFTanhsNzQ/view?usp=sharing" rel="noreferrer" target="_blank">https://drive.google.com/file/d/0B0zPvL-Tp-JSTEJpcUFTanhsNzQ/view?usp=sharing</a><br>
>>>>>>>>><br>
>>>>>>>>> Let me know if you face any issue in accessing the above link.<br>
>>> Thanks.<br>
>>>>>>>><br>
>>>>>>>> This looks great. Very well thought-out.<br>
>>>>>>>><br>
>>>>>>>> One comment:<br>
>>>>>>>><br>
>>>>>>>> "8. In the event of any failover, the standby node will get notified<br>
>>>>>>>> through an event and it will execute a script that will read the<br>
>>>>>>>> configuration specific to the node that went down (again using<br>
>>>>>>>> crm_attribute) and become active."<br>
>>>>>>>><br>
>>>>>>>> It may not be necessary to use the notifications for this. Pacemaker<br>
>>>>>>>> will call your resource agent with the "start" action on the standby<br>
>>>>>>>> node, after ensuring it is stopped on the previous node. Hopefully<br>
>>> the<br>
>>>>>>>> resource agent's start action has (or can have, with configuration<br>
>>>>>>>> options) all the information you need.<br>
>>>>>>>><br>
>>>>>>>> If you do end up needing notifications, be aware that the feature<br>
>>> will<br>
>>>>>>>> be disabled by default in the 1.1.14 release, because changes in<br>
>>> syntax<br>
>>>>>>>> are expected in further development. You can define a compile-time<br>
>>>>>>>> constant to enable them.<br>
<br>
</div></div></blockquote></div><br></div>