Here's one more scenario with a high load test, and my active content check fails. Instead of a recursive fork bomb, I slowly added a CPU intensive process to the system until I arrived at around 260 load on a single CPU box.<br>
<br>My memcached status check was returning: "INFO: memcached is down"<br><br>But, this is what the cluster status looked like:<br><br><span style="font-family:courier new,monospace">Online: [ mem1 mem2 mem3 ]</span><br style="font-family:courier new,monospace">
<br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> cluster-ip-mem2 (ocf::heartbeat:IPaddr2): Started mem2</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> cluster-ip-mem1 (ocf::heartbeat:IPaddr2): Started mem1 (unmanaged) FAILED</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> Clone Set: memcached_clone [memcached]</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> Started: [ mem3 mem2 ]</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> Stopped: [ memcached:2 ]</span><br style="font-family:courier new,monospace"><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">Failed actions:</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace"> memcached:2_start_0 (node=mem1, call=21, rc=-2, status=Timed Out): unknown exec error</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"> cluster-ip-mem1_stop_0 (node=mem1, call=22, rc=-2, status=Timed Out): unknown exec error</span><br style="font-family:courier new,monospace">
<br>Why wouldn't my mem3 failover happen if it timed out stopping the cluster IP?<br><br>Thank you,<br><br>--Cal<br style="font-family:courier new,monospace"><br><div class="gmail_quote">On Thu, Jul 26, 2012 at 4:09 PM, Cal Heldenbrand <span dir="ltr"><<a href="mailto:cal@fbsdata.com" target="_blank">cal@fbsdata.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">A few more questions, as I test various outage scenarios:<br><br>My memcached OCF script appears to give a false positive occasionally, and pacemaker restarts the service. Under the hood, it uses netcat to localhost with a 3 second connection timeout. I've run my script manually in a loop and it never seems to time out.<br>
<br>My primitive looks like this:<br><br>primitive memcached ocf:fbs:memcached \<div class="im"><br> meta is-managed="true" target-role="Started" \<br></div> op monitor interval="1s" timeout="5s"<br>
<br>I've played around with the primitive's interval and timeout. All that seems to do is decrease the frequency that the false positive happens. Is there any way to add logic to the monitor to say "restart the service only if 3 failures in a row happen?"<br>
<br>Also, I've tried to create a massive load failure by using a fork bomb. A few of the outages we've had on our memcache servers appear to be heavy loads -- the machine response to ICMP on the ethernet card, but doesn't respond on ssh. A fork bomb pretty much recreates the same problem. When I fire off a fork bomb on my test machine, it seems to take 5 minutes or more to actually trigger the failover event. It's difficult for me to make sense of all the logging going on, but these two timeout values seem to be interesting:<br>
<br>crmd: error: crm_timer_popped: Election Timeout (I_ELECTION_DC) just popped in state S_ELECTION! (120000ms)<br>crmd: error: crm_timer_popped: Integration Timer (I_INTEGRATED) just popped in state S_INTEGRATION! (180000ms)<br>
<br>Can those values be adjusted? Or is there a common configuration change to be more responsive to an active content check like I'm doing?<br><br>For reference, please see my attached script <b>memcached</b>.<br><br>
Thanks!<span class="HOEnZb"><font color="#888888"><br><br>--Cal</font></span><div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Thu, Jul 26, 2012 at 1:35 PM, Phil Frost <span dir="ltr"><<a href="mailto:phil@macprofessionals.com" target="_blank">phil@macprofessionals.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On 07/26/2012 02:16 PM, Cal Heldenbrand wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
That seems very handy -- and I don't need to specify 3 clones? Once my memcached OCF script reports a downed service, one of them will automatically transition to the current failover node?<br>
</blockquote>
<br></div>
There are options for the clone on how many instances of the cloned resource to create, but they default to the number of nodes in the cluster. See: <a href="http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/ch10s02s02.html" target="_blank">http://www.clusterlabs.org/<u></u>doc/en-US/Pacemaker/1.1/html/<u></u>Pacemaker_Explained/<u></u>ch10s02s02.html</a><div>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Is there any reason you specified just a single memcache_clone, instead of both the memcache primitive and memcached_clone? I might not be understanding exactly how a clone works. Is it like... maybe a "symbolic link" to a primitive, with the ability to specify different metadata and parameters?<br>
</blockquote>
<br></div>
Once you make a clone, the underlying primitive isn't referenced anywhere else (that I can think of). If you want to stop memcache, you don't stop the primitive; you add a location constraint forbidding the clone from running on the node where you want to stop memcache ("crm resource migrate" is easiest). I can't find the relevant documentation, but this is just how they work. The same is true for groups -- the member primitives are never referenced except by the group. I believe in most cases if you try to reference the primitive, you will get an error.<div>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Despite the advertisement of consistent hashing with memcache clients, I've found that they still have long timeouts waiting on connecting to an IP. So, keeping the clustered IPs up at all times is more important than having a seasoned cache behind them.<br>
</blockquote>
<br></div>
I don't know a whole lot about memcache, but it sounds like you might even want to reduce the colocation score for the ips on memcache to be a large number, but not infinity. This way in the case that memcache is broken everywhere, the ips are still permitted to run. This might also cover you in the case that a bug in your resource agent thinks memcache has failed everywhere, but actually it's still running fine. The decision depends which failure the memcache clients handle better: the IP being down, or the IP being up but not having a working memcache server behind it.<br>
<br>
</blockquote></div><br>
</div></div></blockquote></div><br>