<div dir="ltr"><br><br><div class="gmail_quote">On Thu, Nov 19, 2009 at 2:39 AM, Luke Bigum <span dir="ltr"><<a href="mailto:lbigum@iseek.com.au">lbigum@iseek.com.au</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div link="blue" vlink="purple" lang="EN-AU">
<div>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Angie,</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">I can't tell exactly what's you've provided, can you post your
CRM configuration (the output of 'crm configure show')? While you're at it,
also provide '</span> <span style="font-size: 11pt; color: rgb(31, 73, 125);">crm_verify -LV' and 'crm_mon -fo1'.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> <span style="color: rgb(0, 102, 0);">Here are the outputs:</span><br style="color: rgb(0, 102, 0);"></span></p></div></div></blockquote><div><span style="color: rgb(0, 102, 0);"><span style="color: rgb(255, 102, 0);"># crm configure show</span><br>
node test1.localdomain<br>node test2.localdomain<br>primitive ClusterIP ocf:heartbeat:IPaddr2 \<br> params ip="10.0.0.102" cidr_netmask="255.255.255.0" \<br> op monitor interval="10s"<br>
primitive LoadBalancer lsb:haproxy \<br> op monitor interval="10s"<br>primitive WebSite ocf:heartbeat:apache \<br> params configfile="/etc/httpd/conf/httpd.conf" \<br> op monitor interval="1min"<br>
colocation LoadBalancer-with-ClusterIP inf: LoadBalancer ClusterIP<br>order LoadBalancer-after-ClusterIP inf: ClusterIP LoadBalancer<br>property $id="cib-bootstrap-options" \<br> stonith-enabled="false" \<br>
expected-quorum-votes="2" \<br> dc-version="1.0.5-462f1569a43740667daf7b0f6b521742e9eb8fa7" \<br> cluster-infrastructure="openais" \<br> no-quorum-policy="ignore"<br>
<br><span style="color: rgb(255, 153, 0);"># crm_verify -VL</span></span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);">crm_verify[14263]: 2009/11/19_12:22:57 WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on test1.localdomain: unknown error</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">crm_verify[14263]: 2009/11/19_12:22:57 WARN: unpack_rsc_op: Processing failed op WebSite_start_0 on test2.localdomain: unknown error</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);">crm_verify[14263]: 2009/11/19_12:22:57 WARN: common_apply_stickiness: Forcing WebSite away from test1.localdomain after 1000000 failures (max=1000000)</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">crm_verify[14263]: 2009/11/19_12:22:57 WARN: common_apply_stickiness: Forcing WebSite away from test2.localdomain after 1000000 failures (max=1000000)</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">crm_verify[14263]: 2009/11/19_12:22:57 WARN: native_color: Resource WebSite cannot run anywhere</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);">Warnings found during check: config may not be valid</span><br style="color: rgb(102, 0, 0);">
<br><span style="color: rgb(255, 153, 0);"># crm_mon -fo1</span><br style="color: rgb(255, 153, 0);"><span style="color: rgb(0, 102, 0);">============</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);">Last updated: Thu Nov 19 12:29:41 2009</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">Stack: openais</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);">Current DC: test1.localdomain - partition with quorum</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">Version: 1.0.5-462f1569a43740667daf7b0f6b521742e9eb8fa7</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);">2 Nodes configured, 2 expected votes</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">3 Resources configured.</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);">============</span><br style="color: rgb(0, 102, 0);"><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">Online: [ test1.localdomain test2.localdomain ]</span><br style="color: rgb(0, 102, 0);"><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);">ClusterIP (ocf::heartbeat:IPaddr2): Started test1.localdomain</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">LoadBalancer (lsb:haproxy): Started test1.localdomain</span><br style="color: rgb(0, 102, 0);"><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);">Operations:</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">* Node test1.localdomain: </span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);"> ClusterIP: migration-threshold=1000000</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);"> + (4) start: rc=0 (ok)</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);"> + (5) monitor: interval=10000ms rc=0 (ok)</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);"> LoadBalancer: migration-threshold=1000000</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);"> + (6) start: rc=0 (ok)</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);"> + (7) monitor: interval=10000ms rc=0 (ok)</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);"> WebSite: migration-threshold=1000000 fail-count=1000000</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);"> + (9) start: rc=1 (unknown error)</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);"> + (10) stop: rc=0 (ok)</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">* Node test2.localdomain: </span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);"> WebSite: migration-threshold=1000000 fail-count=1000000</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);"> + (5) start: rc=1 (unknown error)</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);"> + (6) stop: rc=0 (ok)</span><br style="color: rgb(0, 102, 0);"><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);">Failed actions:</span><br style="color: rgb(0, 102, 0);"><span style="color: rgb(0, 102, 0);"> WebSite_start_0 (node=test1.localdomain, call=9, rc=1, status=complete): unknown error</span><br style="color: rgb(0, 102, 0);">
<span style="color: rgb(0, 102, 0);"> WebSite_start_0 (node=test2.localdomain, call=5, rc=1, status=complete): unknown error</span><br style="color: rgb(0, 102, 0);"><br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div link="blue" vlink="purple" lang="EN-AU"><div><p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">This looks suspicious though:</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Nov 19 01:25:08 test2 crmd: [24251]: info: process_lrm_event:
LRM operation WebServer_monitor_60000 (call=483, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Personally I'd start with the OCF RA and leave LSB:httpd alone. From
the above error message, something inside lssb:httpd is returning -2, which is
not a supported return code.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Depending on how confident you are with shell scripts, you might
find it helpful to eliminate Pacemaker from the equation and call the Resource
Agent script yourself to debug problems manually, like so...</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"><span style="color: rgb(0, 153, 0);"> I'll be doing this and reporting you back. </span><br></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Disable your resource so Pacemaker doesn't interfere:</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">crm_resource -r WebSite -m -p target-role -v stopped</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Then move into the RA directory and set a necessary environment
variable:</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">cd =/usr/lib/ocf/resource.d/heartbeat</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">export OCF_ROOT=/usr/lib/ocf</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Start testing the apache RA, setting the only mandatory
environment variable for ocf:heartbeat:apache :</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">export OCF_RESKEY_configfile=/path/to/your/main/apache/config</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">./apache start</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">echo $?</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">That should echo "0" for success. Judging by your logs,
you can start Apache but the monitor is failing:</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">./apache monitor</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">echo $?</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">If that doesn't echo "0", you might get a helpful
error message explaining what's wrong. You might have to read through the
apache script itself to figure out why it's failing. Finally test the 'stop'
operation:</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">./apache stop</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">echo $?</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Should echo "0" as well. If this all works for you,
but the resource in Pacemaker is still not working, then it's probably something
in your CIB (like a bad attribute), as you've just done pretty much exactly
what Pacemaker will do.</span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Let us know how you go.</span></p></div></div></blockquote><div><span style="font-size: 11pt; color: rgb(31, 73, 125);"><span style="color: rgb(0, 153, 0);">Sure, I will. Thank you so much.</span></span> <br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><div link="blue" vlink="purple" lang="EN-AU"><div>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);">Tod <br></span></p>
<p class="MsoNormal"><b><span style="font-size: 14pt; color: rgb(11, 83, 182);" lang="EN-US">Luke Bigum</span></b></p>
<p class="MsoNormal"><b><span style="font-size: 9pt; color: rgb(57, 137, 243);" lang="EN-US">Systems Administrator</span></b></p>
<p class="MsoNormal"><span style="font-size: 8pt; color: rgb(92, 92, 92);" lang="EN-US"> (p) 1300 661 668</span></p>
<p class="MsoNormal"><span style="font-size: 8pt; color: rgb(92, 92, 92);" lang="EN-US"> (f) 1300 661 540</span></p>
<p class="MsoNormal"><span style="font-size: 8pt; color: rgb(92, 92, 92);" lang="EN-US">(e) </span><span style="font-size: 11pt; color: rgb(31, 73, 125);"><a href="mailto:lbigum@iseek.com.au" target="_blank"><span style="font-size: 8pt; color: rgb(57, 137, 243);" lang="EN-US">lbigum@iseek.com.au</span></a></span><span style="font-size: 8pt; color: rgb(92, 92, 92);" lang="EN-US"></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"><a href="http://www.iseek.com.au/" target="_blank"><span style="font-size: 8pt; color: rgb(57, 137, 243);" lang="EN-US">http://www.iseek.com.au</span></a></span><span style="font-size: 8pt; color: rgb(57, 137, 243);" lang="EN-US"></span></p>
<p class="MsoNormal"><span style="font-size: 8pt; color: rgb(92, 92, 92);" lang="EN-US">Level 1, 100 Ipswich Road Woolloongabba QLD 4102</span></p>
<p class="MsoNormal"><span style="font-size: 8pt; color: rgb(92, 92, 92);" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="font-size: 8pt; color: rgb(92, 92, 92);"><img src="cid:image001.jpg@01CA6901.D25D3CD0" alt="iseekbar.jpg" border="0" height="29" width="251"></span></p>
<p class="MsoNormal"><span style="font-size: 8pt; color: rgb(92, 92, 92);"> </span></p>
<p class="MsoNormal"><span style="font-size: 8pt; color: rgb(31, 73, 125);">This e-mail and any files transmitted with it may contain
confidential and privileged material for the sole use of the intended
recipient. Any review, use, distribution or disclosure by others is strictly
prohibited. If you are not the intended recipient (or authorised to receive for
the recipient), please contact the sender by reply e-mail and delete all copies
of this message.</span></p>
<p class="MsoNormal"><span style="font-size: 8pt; color: rgb(92, 92, 92);" lang="EN-US"> </span></p>
<p class="MsoNormal"><span style="font-size: 11pt; color: rgb(31, 73, 125);"> </span></p>
<div style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span style="font-size: 10pt;" lang="EN-US">From:</span></b><span style="font-size: 10pt;" lang="EN-US"> Angie T. Muhammad
[mailto:<a href="mailto:angie.tawfik@gmail.com" target="_blank">angie.tawfik@gmail.com</a>] <br>
<b>Sent:</b> Thursday 19 November 2009 9:57 AM<br>
<b>To:</b> <a href="mailto:pacemaker@oss.clusterlabs.org" target="_blank">pacemaker@oss.clusterlabs.org</a><br>
<b>Subject:</b> [Pacemaker] Error starting Apache on 2 nodes cluster</span></p>
</div><div><div></div><div class="h5">
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">Hello <br>
I'm a pacemaker and openais beginner.<br>
I followed the document 'cluster from scratch' and I successfully managed to
create and monitor a 'ClusterIP' and 'LoadBalancer' resources.<br>
<br>
But, Whenever I try to start Apache: <br>
# crm configure primitive WebSite ocf:heartbeat:apache params
configfile=/etc/httpd/conf/httpd.conf op monitor interval=1min<br>
<br>
whether using (ocf:heartbeat:apache) or (lsb::httpd) I get the following errors
when watching crm_mon:<br>
<br>
============<br>
Last updated: Thu Nov 19 01:38:33 2009<br>
Stack: openais<br>
Current DC: test1.localdomain - partition with quorum<br>
Version: 1.0.5-462f1569a43740667daf7b0f6b521742e9eb8fa7<br>
2 Nodes configured, 2 expected votes<br>
3 Resources configured.<br>
============<br>
<br>
Online: [ test1.localdomain test2.localdomain ]<br>
<br>
ClusterIP
(ocf::heartbeat:IPaddr2): Started
test1.localdomain<br>
LoadBalancer (lsb:haproxy): Started test1.localdomain<br>
<br>
<span style="color: red;">Failed actions:<br>
WebSite_start_0 (node=test1.localdomain, call=9, rc=1,
status=complete): unknown error<br>
WebSite_start_0 (node=test2.localdomain, call=5, rc=1,
status=complete): unknown error<br>
</span>/************************************************************************************************************/<br>
<br>
Knowing that I am using:<br>
CentOS 5.4.. <br>
openais-0.80.5-15.1<br>
pacemaker-1.0.5-4.1<br>
# chkconfig httpd off <br>
server-status is not enabled in my httpd.conf ... <br>
<br>
I always check apache processes before configuring my crm using:<br>
<br>
# ps aux | grep httpd <br>
/* to make sure there are no zombie processes */<br>
<br>
# /etc/init.d/httpd status<br>
/* to gurantee it's stopped and nothing is locked */<br clear="all">
<br>
Last but not least I am ataching the <b>last 100 lines of my /var/log/messages</b>
of the 2nd node to help you help me. <br>
I have been on this loop for four days now and I have no idea why the crm can't
start apache though when manually starting it, everything runs smoothly!!! <br>
<br>
Thank you in advance<br>
-- <br>
All the best,<br>
Angie</p>
</div>
</div></div></div>
</div>
<br>_______________________________________________<br>
Pacemaker mailing list<br>
<a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br>All the best,<br>Angie<br>
</div>