<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 1, 2020 at 5:55 PM Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">There's no harm on the Pacemaker side in doing so.<br>
<br>
A resource that's running but removed from the configuration is what<br>
Pacemaker calls an "orphan". By default (the stop-orphan-resources<br>
cluster property) it will try to stop these. Pacemaker keeps the set of<br>
parameters that a resource was started with in memory, so it doesn't<br>
need the now-removed configuration to perform the stop. So, the<br>
"ORPHANED" part of this is normal and appropriate.<br>
<br>
The problem in this particular case is the "FAILED ... (blocked)".<br>
Removing the configuration shouldn't cause the resource to fail, and<br>
something is blocking the stop. You should be able to see in the failed<br>
action section of the status, or in the logs, what failed and why it's<br>
blocked. My guess is the stop itself failed, in which case you'd need<br>
to investigate why that happened.<br></blockquote><div><br></div><div>Hi Ken,</div><div><br></div><div>As always, thanks a lot for pointing me to the right direction!</div><div>I have digged logs, but something not logical happens. Maybe you can shed light a bit?</div><div><br></div><div>Just in case, I have a pretty old pacemaker (<span style="color:rgb(0,0,0);font-family:monospace">Pacemaker 1.1.15-11.el7</span>) freezing of the version was conducted by</div><div>changes in stikiness=-1 attribute handling logic. I consider update to a newer stable version later on, but</div><div>at the moment I have to deal with this version.</div><div><br></div><div>First of all "stop-orphan-resources" was not set and thus by default should be true, but still I see a strange message</div><div>> Cluster configured not to stop active orphans. vip-10.0.0.115 must be stopped manually on tt738741-ip2</div><div><br></div><div>I even explicitly set "stop-orphan-resources" to true</div><div>> sudo crm_attribute --type crm_config --name stop-orphan-resources --query<br>scope=crm_config name=stop-orphan-resources value=true<br></div><div><br></div><div>For the sake of justice pacemaker still tries to remove ORPHANED resources, i.e</div><div>> Oct 02 09:20:56 [21765] tt738741-ip2.ops pengine: info: native_color: Stopping orphan resource vip-10.0.0.115</div><div><br></div><div>To my mind the issue is the following:</div><div>> Clearing failcount for monitor on vip-10.0.0.115, tt738741-ip2 failed and now resource parameters have changed.<br></div><div><br></div><div>I suppose that this operation clears parameters of the resource which were used at start</div><div><br></div><div>The error is pretty straight:</div><div>> IPaddr2(vip-10.0.0.115)[21351]: 2020/10/02_09:20:56 ERROR: IP address (the ip parameter) is mandatory<br></div><div><br></div><div>As I understand it means the ip parameter has already vanished at the moment of "stop" action.</div><div><br></div><div>It looks like a bug, but who knows.</div><div><br></div><div>Trimmed logs, if required I can provide full log, cib.xml, etc</div><div>```</div><div>Oct 02 09:20:56 [21765] tt738741-ip2.ops pengine: warning: process_rsc_state: Cluster configured not to stop active orphans. vip-10.0.0.115 must be stopped manually on tt738741-ip2<br>Oct 02 09:20:56 [21765] tt738741-ip2.ops pengine: info: native_add_running: resource vip-10.0.0.115 isn't managed<br>Oct 02 09:20:56 [21765] tt738741-ip2.ops pengine: info: native_add_running: resource haproxy-10.0.0.115 isn't managed<br>Oct 02 09:20:56 [21765] tt738741-ip2.ops pengine: info: determine_op_status: Operation monitor found resource vip-10.0.0.115 active on tt738741-ip2<br>Oct 02 09:20:56 [21765] tt738741-ip2.ops pengine: info: check_operation_expiry: Clearing failcount for monitor on vip-10.0.0.115, tt738741-ip2 failed and now resource parameters have changed.<br>...<br>Oct 02 09:20:56 [21765] tt738741-ip2.ops pengine: warning: process_rsc_state: Detected active orphan vip-10.0.0.115 running on tt738741-ip2<br>...<br>Oct 02 09:20:56 [21765] tt738741-ip2.ops pengine: info: native_print: vip-10.0.0.115 (ocf::heartbeat:IPaddr2): ORPHANED Started tt738741-ip2<br>...<br>Oct 02 09:20:56 [21765] tt738741-ip2.ops pengine: info: native_color: Stopping orphan resource vip-10.0.0.115<br>...<br>Oct 02 09:20:56 [21763] tt738741-ip2.ops lrmd: info: log_execute: executing - rsc:vip-10.0.0.115 action:stop call_id:4358<br>Oct 02 09:20:56 [21766] tt738741-ip2.ops crmd: info: te_crm_command: Executing crm-event (1): clear_failcount on tt738741-ip2<br>Oct 02 09:20:56 [21766] tt738741-ip2.ops crmd: info: process_lrm_event: Result of monitor operation for vip-10.0.0.115 on tt738741-ip2: Cancelled | call=4323 key=vip-10.0.0.115_monitor_10000 confirmed=true<br>Oct 02 09:20:56 [21766] tt738741-ip2.ops crmd: info: handle_failcount_op: Removing failcount for vip-10.0.0.115<br>...<br>IPaddr2(vip-10.0.0.115)[21351]: 2020/10/02_09:20:56 ERROR: IP address (the ip parameter) is mandatory<br>Oct 02 09:20:56 [21763] tt738741-ip2.ops lrmd: notice: operation_finished: vip-10.0.0.115_stop_0:21351:stderr [ ocf-exit-reason:IP address (the ip parameter) is mandatory ]<br>Oct 02 09:20:56 [21763] tt738741-ip2.ops lrmd: info: log_finished: finished - rsc:vip-10.0.0.115 action:stop call_id:4358 pid:21351 exit-code:6 exec-time:124ms queue-time:0ms<br>Oct 02 09:20:56 [21766] tt738741-ip2.ops crmd: notice: process_lrm_event: Result of stop operation for vip-10.0.0.115 on tt738741-ip2: 6 (not configured) | call=4358 key=vip-10.0.0.115_stop_0 confirmed=true cib-update=20532<br>Oct 02 09:20:56 [21766] tt738741-ip2.ops crmd: notice: process_lrm_event: tt738741-ip2-vip-10.0.0.115_stop_0:4358 [ ocf-exit-reason:IP address (the ip parameter) is mandatory\n ]<br><br></div><div>```</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
To address the replacing:<br>
<br>
Never replace the status section; it's continually updated by the live<br>
cluster, so you could be re-uploading old information that leads to<br>
incorrect actions. Replacing just the configuration section is the way<br>
to go.<br></blockquote><div>I'm just trying all possible options. But in general I use old status section only for crm_similate -LS</div><div>in order to predict actions</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
cibadmin --replace or crm_shadow should work fine for replacing the<br>
configuration, but both crm and pcs allow batching commands in a file<br>
and then applying them all at once, so it may not be necessary.<br></blockquote><div>Yes, thanks, I have tried this approach and it is way better than one by one command execution, but still</div><div>it requires additional logic to form those batch commands, i.e. find out what resources should be</div><div>added, removed, updated, etc</div><div>Also it does not exclude possibility of stucking on --wait (it happens when I try to execute several commands in series)</div><div>I haven't found the root, but it is reproduced from time to time and further transactions are not possible</div><div><br></div><div>With XML I don't have to think about this, all I have to do is prepare the correct XML and verify it on syntax errors.</div><div>Well in my case it is just the ideal way to configure a cluster. I lived for ages with crm configure and now trying to</div><div>optimize cluster configuration. moreover it is just much faster then doing this by commands and waiting to</div><div>take effect for each.</div><div> </div></div></div>