Hello <br><br>Maybe my question is stupid, but are you root when you try to killing the procs?<br><br>Thanks<br><br><div class="gmail_quote">2013/1/9 Kazunori INOUE <span dir="ltr"><<a href="mailto:inouekazu@intellilink.co.jp" target="_blank">inouekazu@intellilink.co.jp</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Andrew,<br>
<br>
I have another question about this subject.<br>
Even if pengine, stonithd, and attrd crash after pacemakerd is killed<br>
(for example, killed by OOM_Killer), node status does not change.<br>
<br>
* pseudo testcase<br>
<br>
[dev1 ~]$ crm configure show<br>
node $id="2472913088" dev2<br>
node $id="2506467520" dev1<br>
primitive prmDummy ocf:pacemaker:Dummy \<br>
op monitor on-fail="restart" interval="10s"<br>
property $id="cib-bootstrap-options" \<br>
dc-version="1.1.8-d20d06f" \<br>
cluster-infrastructure="<u></u>corosync" \<br>
no-quorum-policy="ignore" \<br>
stonith-enabled="false" \<br>
startup-fencing="false"<br>
rsc_defaults $id="rsc-options" \<br>
resource-stickiness="INFINITY" \<br>
migration-threshold="1"<br>
<br>
[dev1 ~]$ pkill -9 pacemakerd<br>
[dev1 ~]$ pkill -9 pengine<br>
[dev1 ~]$ pkill -9 stonithd<br>
[dev1 ~]$ pkill -9 attrd<br>
<br>
[dev1 ~]$ ps -ef|egrep 'corosync|pacemaker'<br>
root 19124 1 0 14:27 ? 00:00:01 corosync<br>
496 19144 1 0 14:27 ? 00:00:00 /usr/libexec/pacemaker/cib<br>
root 19146 1 0 14:27 ? 00:00:00 /usr/libexec/pacemaker/lrmd<br>
496 19149 1 0 14:27 ? 00:00:00 /usr/libexec/pacemaker/crmd<br>
<br>
[dev1 ~]$ crm_mon -1<br>
:<br>
Stack: corosync<br>
Current DC: dev2 (2472913088) - partition with quorum<br>
Version: 1.1.8-d20d06f<br>
2 Nodes configured, unknown expected votes<br>
1 Resources configured.<br>
<br>
<br>
Online: [ dev1 dev2 ]<br>
<br>
prmDummy (ocf::pacemaker:Dummy): Started dev1<br>
<br>
Node (dev1) remains Online.<br>
When other processes such as lrmd crash, it becomes "UNCLEAN (offline)".<br>
Is this a bug? Or specifications?<br>
<br>
Best Regards,<br>
Kazunori INOUE<br>
<br>
<br>
(13.01.08 09:16), Andrew Beekhof wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Wed, Dec 19, 2012 at 8:15 PM, Kazunori INOUE<br>
<<a href="mailto:inouekazu@intellilink.co.jp" target="_blank">inouekazu@intellilink.co.jp</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
(12.12.13 08:26), Andrew Beekhof wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
On Wed, Dec 12, 2012 at 8:02 PM, Kazunori INOUE<br>
<<a href="mailto:inouekazu@intellilink.co.jp" target="_blank">inouekazu@intellilink.co.jp</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Hi,<br>
<br>
I recognize that pacemakerd is much less likely to crash.<br>
However, a possibility of being killed by OOM_Killer etc. is not 0%.<br>
</blockquote>
<br>
<br>
True. Although we just established in another thread that we don't<br>
have any leaks :)<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
So I think that a user gets confused. since behavior at the time of<br>
process<br>
death differs even if pacemakerd is running.<br>
<br>
case A)<br>
When pacemakerd and other processes (crmd etc.) are the parent-child<br>
relation.<br>
<br>
</blockquote>
<br>
[snip]<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
For example, crmd died.<br>
However, since it is relaunched, the state of the cluster is not<br>
affected.<br>
</blockquote>
<br>
<br>
Right.<br>
<br>
[snip]<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
case B)<br>
When pacemakerd and other processes are NOT the parent-child relation.<br>
Although pacemakerd was killed, it assumed the state where it was<br>
respawned<br>
by Upstart.<br>
<br>
$ service corosync start ; service pacemaker start<br>
$ pkill -9 pacemakerd<br>
$ ps -ef|egrep 'corosync|pacemaker|UID'<br>
UID PID PPID C STIME TTY TIME CMD<br>
root 21091 1 1 14:52 ? 00:00:00 corosync<br>
496 21099 1 0 14:52 ? 00:00:00 /usr/libexec/pacemaker/cib<br>
root 21100 1 0 14:52 ? 00:00:00<br>
/usr/libexec/pacemaker/<u></u>stonithd<br>
root 21101 1 0 14:52 ? 00:00:00 /usr/libexec/pacemaker/lrmd<br>
496 21102 1 0 14:52 ? 00:00:00<br>
/usr/libexec/pacemaker/attrd<br>
496 21103 1 0 14:52 ? 00:00:00<br>
/usr/libexec/pacemaker/pengine<br>
496 21104 1 0 14:52 ? 00:00:00 /usr/libexec/pacemaker/crmd<br>
root 21128 1 1 14:53 ? 00:00:00 /usr/sbin/pacemakerd<br>
</blockquote>
<br>
<br>
Yep, looks right.<br>
<br>
</blockquote>
<br>
Hi Andrew,<br>
<br>
We discussed this behavior.<br>
Behavior when pacemakerd and other processes are not parent-child<br>
relation (case B) reached the conclusion that there is room for<br>
improvement.<br>
<br>
Since not all users are experts, they may kill pacemakerd accidentally.<br>
Such a user will get confused if the behavior after crmd death changes<br>
with the following conditions.<br>
case A: pacemakerd and others (crmd etc.) are the parent-child relation.<br>
case B: pacemakerd and others are not the parent-child relation.<br>
<br>
So, we want to *always* obtain the same behavior as the case where<br>
there is parent-child relation.<br>
That is, when crmd etc. die, we want pacemaker to always relaunch<br>
the process always immediately.<br>
</blockquote>
<br>
No. Sorry.<br>
Writing features to satisfy an artificial test case is not a good practice.<br>
<br>
We can speed up the failure detection for case B (I'll agree that 60s<br>
is way too long, 5s or 2s might be better depending on the load is<br>
creates), but causing downtime now to _maybe_ avoid downtime in the<br>
future makes no sense.<br>
Especially when you consider that the node will likely be fenced if<br>
the crmd fails anyway.<br>
<br>
Take a look at the logs from a some ComponentFail test runs and you'll<br>
see that the parent-child relationship regularly _fails_ to prevent<br>
downtime.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Regards,<br>
Kazunori INOUE<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In this case, the node will be set to UNCLEAN if crmd dies.<br>
That is, the node will be fenced if there is stonith resource.<br>
</blockquote>
<br>
<br>
Which is exactly what happens if only pacemakerd is killed with your<br>
proposal.<br>
Except now you have time to do a graceful pacemaker restart to<br>
re-establish the parent-child relationship.<br>
<br>
If you want to compare B with something, it needs to be with the old<br>
"children terminate if pacemakerd dies" strategy.<br>
Which is:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
$ service corosync start ; service pacemaker start<br>
$ pkill -9 pacemakerd<br>
... the node will be set to UNCLEAN<br>
</blockquote>
<br>
<br>
Old way: always downtime because children terminate which triggers fencing<br>
Our way: no downtime unless there is an additional failure (to the cib or<br>
crmd)<br>
<br>
Given that we're trying for HA, the second seems preferable.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
$ pkill -9 crmd<br>
$ crm_mon -1<br>
Last updated: Wed Dec 12 14:53:48 2012<br>
Last change: Wed Dec 12 14:53:10 2012 via crmd on dev2<br>
<br>
Stack: corosync<br>
Current DC: dev2 (2472913088) - partition with quorum<br>
Version: 1.1.8-3035414<br>
<br>
2 Nodes configured, unknown expected votes<br>
0 Resources configured.<br>
<br>
Node dev1 (2506467520): UNCLEAN (online)<br>
Online: [ dev2 ]<br>
<br>
<br>
How about making behavior selectable with an option?<br>
</blockquote>
<br>
<br>
MORE_DOWNTIME_PLEASE=(true|<u></u>false) ?<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
When pacemakerd dies,<br>
mode A) which behaves in an existing way. (default)<br>
mode B) which makes the node UNCLEAN.<br>
<br>
Best Regards,<br>
Kazunori INOUE<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Making stop work when there is no pacemakerd process is a different<br>
matter. We can make that work.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Though the best solution is to relaunch pacemakerd, if it is difficult,<br>
I think that a shortcut method is to make a node unclean.<br>
<br>
<br>
And now, I tried Upstart a little bit.<br>
<br>
1) started the corosync and pacemaker.<br>
<br>
$ cat /etc/init/pacemaker.conf<br>
respawn<br>
script<br>
[ -f /etc/sysconfig/pacemaker ] && {<br>
. /etc/sysconfig/pacemaker<br>
}<br>
exec /usr/sbin/pacemakerd<br>
end script<br>
<br>
$ service co start<br>
Starting Corosync Cluster Engine (corosync): [ OK ]<br>
$ initctl start pacemaker<br>
pacemaker start/running, process 4702<br>
<br>
<br>
$ ps -ef|egrep 'corosync|pacemaker'<br>
root 4695 1 0 17:21 ? 00:00:00 corosync<br>
root 4702 1 0 17:21 ? 00:00:00 /usr/sbin/pacemakerd<br>
<a href="tel:496%20%20%20%204703%20%204702%20%200%2017" value="+49647034702017" target="_blank">496 4703 4702 0 17</a>:21 ? 00:00:00 /usr/libexec/pacemaker/cib<br>
root 4704 4702 0 17:21 ? 00:00:00<br>
/usr/libexec/pacemaker/<u></u>stonithd<br>
root 4705 4702 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/lrmd<br>
<a href="tel:496%20%20%20%204706%20%204702%20%200%2017" value="+49647064702017" target="_blank">496 4706 4702 0 17</a>:21 ? 00:00:00<br>
/usr/libexec/pacemaker/attrd<br>
<a href="tel:496%20%20%20%204707%20%204702%20%200%2017" value="+49647074702017" target="_blank">496 4707 4702 0 17</a>:21 ? 00:00:00<br>
/usr/libexec/pacemaker/pengine<br>
<a href="tel:496%20%20%20%204708%20%204702%20%200%2017" value="+49647084702017" target="_blank">496 4708 4702 0 17</a>:21 ? 00:00:00 /usr/libexec/pacemaker/crmd<br>
<br>
2) killed pacemakerd.<br>
<br>
$ pkill -9 pacemakerd<br>
<br>
$ ps -ef|egrep 'corosync|pacemaker'<br>
root 4695 1 0 17:21 ? 00:00:01 corosync<br>
496 4703 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/cib<br>
root 4704 1 0 17:21 ? 00:00:00<br>
/usr/libexec/pacemaker/<u></u>stonithd<br>
root 4705 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/lrmd<br>
496 4706 1 0 17:21 ? 00:00:00<br>
/usr/libexec/pacemaker/attrd<br>
496 4707 1 0 17:21 ? 00:00:00<br>
/usr/libexec/pacemaker/pengine<br>
496 4708 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/crmd<br>
root 4760 1 1 17:24 ? 00:00:00 /usr/sbin/pacemakerd<br>
<br>
3) then I stopped pacemakerd. however, some processes did not stop.<br>
<br>
$ initctl stop pacemaker<br>
pacemaker stop/waiting<br>
<br>
<br>
$ ps -ef|egrep 'corosync|pacemaker'<br>
root 4695 1 0 17:21 ? 00:00:01 corosync<br>
496 4703 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/cib<br>
root 4704 1 0 17:21 ? 00:00:00<br>
/usr/libexec/pacemaker/<u></u>stonithd<br>
root 4705 1 0 17:21 ? 00:00:00 /usr/libexec/pacemaker/lrmd<br>
496 4706 1 0 17:21 ? 00:00:00<br>
/usr/libexec/pacemaker/attrd<br>
496 4707 1 0 17:21 ? 00:00:00<br>
/usr/libexec/pacemaker/pengine<br>
<br>
Best Regards,<br>
Kazunori INOUE<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This isnt the case when the plugin is in use though, but then I'd<br>
also<br>
have expected most of the processes to die also.<br>
<br>
</blockquote>
Since node status will also change if such a result is brought,<br>
we desire to become so.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
----<br>
$ cat /etc/redhat-release<br>
Red Hat Enterprise Linux Server release 6.3 (Santiago)<br>
<br>
$ ./configure --sysconfdir=/etc --localstatedir=/var<br>
--without-cman<br>
--without-heartbeat<br>
-snip-<br>
pacemaker configuration:<br>
Version = 1.1.8 (Build: 9c13d14)<br>
Features = generated-manpages agent-manpages<br>
ascii-docs<br>
publican-docs ncurses libqb-logging libqb-ipc lha-fencing<br>
corosync-native<br>
snmp<br>
<br>
<br>
$ cat config.log<br>
-snip-<br>
6000 | #define BUILD_VERSION "9c13d14"<br>
6001 | /* end confdefs.h. */<br>
6002 | #include <gio/gio.h><br>
6003 |<br>
6004 | int<br>
6005 | main ()<br>
6006 | {<br>
6007 | if (sizeof (GDBusProxy))<br>
6008 | return 0;<br>
6009 | ;<br>
6010 | return 0;<br>
6011 | }<br>
6012 configure:32411: result: no<br>
6013 configure:32417: WARNING: Unable to support systemd/upstart.<br>
You need<br>
to use glib >= 2.26<br>
-snip-<br>
6286 | #define BUILD_VERSION "9c13d14"<br>
6287 | #define SUPPORT_UPSTART 0<br>
6288 | #define SUPPORT_SYSTEMD 0<br>
<br>
<br>
Best Regards,<br>
Kazunori INOUE<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
related bugzilla:<br>
<a href="http://bugs.clusterlabs.org/show_bug.cgi?id=5064" target="_blank">http://bugs.clusterlabs.org/<u></u>show_bug.cgi?id=5064</a><br>
<br>
<br>
Best Regards,<br>
Kazunori INOUE<br>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div><br><br clear="all"><br>-- <br>esta es mi vida e me la vivo hasta que dios quiera