David and Jake,<br><br>Thanks for the replies, that's exactly what I needed.� I think where I was going wrong was playing with the role="Master" and role="Slave" parameters, but always having the same timeout for both.� (crm_verify -L confused me even more on what I was doing wrong)<br>
<br>Speaking of timeouts, why should I stay above 20 seconds for a monitoring interval?� If the service crashes for whatever reason, I'd like to avoid downtime and keep the failover window as quick as possible.� If this makes any difference, Redis and Memcache are different beasts compared to databases.� The clients are usually done with their requests after 20 milliseconds.� It's very fast paced, and 20 seconds of downtime is pretty significant in this application.<br>
<br>Also David, after increasing my interval to 20 seconds and doing a separate monitor definition for Master and Slave, both checks still run duplicate monitor ops.� (See below)� Probably not a big deal, but I was just curious on why Pacemaker would do that.<br>
<br>Thanks!<br><br>--Cal<br><br><br>------------------------------------------------------------------------------------------------------------------------------------------<br>redis(redis:0)[3600]:�� 2012/11/13_08:40:39 INFO: redis-server is running (PID 15221)<br>
redis(redis:0)[3600]:�� 2012/11/13_08:40:39 INFO: redis-server is running (PID 15221)<br>redis(redis:0)[3600]:�� 2012/11/13_08:40:39 INFO: redis-server is running (as master)<br>redis(redis:0)[3600]:�� 2012/11/13_08:40:39 INFO: redis-server is running (as master)<br>
redis(redis:0)[4175]:�� 2012/11/13_08:40:59 INFO: redis-server is running (PID 15221)<br>redis(redis:0)[4175]:�� 2012/11/13_08:40:59 INFO: redis-server is running (PID 15221)<br>redis(redis:0)[4175]:�� 2012/11/13_08:40:59 INFO: redis-server is running (as master)<br>
redis(redis:0)[4175]:�� 2012/11/13_08:40:59 INFO: redis-server is running (as master)<br>------------------------------------------------------------------------------------------------------------------------------------------<br>
redis(redis:1)[7690]:�� 2012/11/13_08:31:06 INFO: redis-server is running (PID 7330)<br>redis(redis:1)[7690]:�� 2012/11/13_08:31:06 INFO: redis-server is running (PID 7330)<br>redis(redis:1)[7690]:�� 2012/11/13_08:31:06 INFO: redis-server is running (as slave)<br>
redis(redis:1)[7690]:�� 2012/11/13_08:31:06 INFO: redis-server is running (as slave)<br>redis(redis:1)[7729]:�� 2012/11/13_08:31:28 INFO: redis-server is running (PID 7330)<br>redis(redis:1)[7729]:�� 2012/11/13_08:31:28 INFO: redis-server is running (PID 7330)<br>
redis(redis:1)[7729]:�� 2012/11/13_08:31:28 INFO: redis-server is running (as slave)<br>redis(redis:1)[7729]:�� 2012/11/13_08:31:28 INFO: redis-server is running (as slave)<br>------------------------------------------------------------------------------------------------------------------------------------------<br>
<br><br><div class="gmail_quote">On Mon, Nov 12, 2012 at 3:37 PM, David Vossel <span dir="ltr"><<a href="mailto:dvossel@redhat.com" target="_blank">dvossel@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">----- Original Message -----<br>
> From: "Cal Heldenbrand" <<a href="mailto:cal@fbsdata.com">cal@fbsdata.com</a>><br>
> To: "The Pacemaker cluster resource manager" <<a href="mailto:pacemaker@oss.clusterlabs.org">pacemaker@oss.clusterlabs.org</a>><br>
</div><div class="im">> Sent: Monday, November 12, 2012 3:27:02 PM<br>
> Subject: Re: [Pacemaker] Custom event script during master/slave failover?<br>
><br>
><br>
> I've managed to get a test Redis configuration up and running...<br>
> however this is the first time I've played with a master/slave<br>
> resource, and I'm noticing some odd behavior in the monitoring<br>
> actions. Setting in a super quick monitoring interval of 1 second, I<br>
> noticed at first only the slave was being monitored, 2x in each<br>
> second. Then after changing a bunch of config options and several<br>
> cluster restarts, I noticed the monitoring action had moved to the<br>
> master, again 2x events in each second.<br>
<br>
</div>Use a longer timeout than 1s. Don't go under 20s... that might be why you see 2x events.<br>
<br>
The monitor action didn't move to the master or move to the slave. �There is a separate monitor operation used for resources in the master and slave role. �Both monitor operations have to be defined, looking at your config it appears only the Master monitor op is defined.<br>
<br>
<a href="http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_monitoring_multi_state_resources.html" target="_blank">http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_monitoring_multi_state_resources.html</a><br>
<br>
If you are new to multi-state resources. I'd recommend playing around with the ocf:pacemaker:Stateful resource some. �Stateful is a fake resource that does nothing except allow you to start/stop/promote/demote/monitor it. �It is useful for testing out configurations and understanding expected behavior.<br>
<span class="HOEnZb"><font color="#888888"><br>
-- Vossel<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
><br>
> redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is<br>
> running (PID 30759)<br>
> redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is<br>
> running (PID 30759)<br>
> redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is<br>
> running (as master)<br>
> redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is<br>
> running (as master)<br>
><br>
> Here is my primitive and master/slave definition:<br>
><br>
> primitive redis ocf:heartbeat:redis \<br>
> meta target-role="Master" is-managed="true" \<br>
> op monitor interval="1s" role="Master" timeout="5s" on-fail="restart"<br>
> ms redis_clone redis \<br>
> meta notify="true" is-managed="true" ordered="false"<br>
> interleave="false" globally-unique="false" target-role="Master"<br>
> migration-threshold="1"<br>
><br>
> I'd like to figure out how to make sure both master and slave are<br>
> actively monitored. Let me know if you need any more info on my<br>
> config!<br>
><br>
> Thank you,<br>
><br>
> --Cal<br>
><br>
><br>
><br>
> On Fri, Nov 9, 2012 at 4:52 PM, David Vossel < <a href="mailto:dvossel@redhat.com">dvossel@redhat.com</a> ><br>
> wrote:<br>
><br>
><br>
><br>
><br>
><br>
> ----- Original Message -----<br>
> > From: "Cal Heldenbrand" < <a href="mailto:cal@fbsdata.com">cal@fbsdata.com</a> ><br>
> > To: "The Pacemaker cluster resource manager" <<br>
> > <a href="mailto:pacemaker@oss.clusterlabs.org">pacemaker@oss.clusterlabs.org</a> ><br>
> > Sent: Friday, November 9, 2012 4:29:05 PM<br>
> > Subject: [Pacemaker] Custom event script during master/slave<br>
> > failover?<br>
> ><br>
> ><br>
> > Hi everyone,<br>
> ><br>
> > I'm playing around with the possibility of using Pacemaker in a<br>
> > Redis<br>
> > master/slave cluster. The difficult part of this, is that Redis<br>
> > will<br>
> > not automatically flip itself from read-only slave mode into master<br>
> > mode. A client needs to connect to the slave server and run this<br>
> > SLAVEOF NO ONE command to turn it into a master.<br>
><br>
> The resource agent should be able to do this. Pacemaker will tell the<br>
> resource agent to promote/demote a resource. Your Redis agent can<br>
> then go off and do whatever logic is necessary to actually<br>
> transition the process a master/slave process.<br>
><br>
><br>
> ><br>
> > If I set up a master/slave primitive, is it possible to fire an<br>
> > event<br>
> > script during a failover event that would allow me to do this?<br>
> ><br>
> > After the stonith event fires and the broken master reboots,<br>
> > I would<br>
> > write in something special in the Redis init.d script to detect if<br>
> > my peer is already a master, and start up in slave mode.<br>
><br>
> If you are using Pacemaker, it should be controlling what node has<br>
> the master resource and what node has the slave resource. Take a<br>
> look at what OCF scripts are and how pacemaker uses them.<br>
><br>
><br>
> ><br>
> > Has anyone else worked up a cluster config with Redis?<br>
><br>
> Using google, found some work someone did on a redis resource agent.<br>
> <a href="https://github.com/martinwalter/resource-agents/blob/master/heartbeat/redis" target="_blank">https://github.com/martinwalter/resource-agents/blob/master/heartbeat/redis</a><br>
><br>
> -- Vossel<br>
> > Thank you,<br>
> ><br>
> > --Cal<br>
> ><br>
> > _______________________________________________<br>
> > Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> > <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
> ><br>
> > Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> > Getting started:<br>
> > <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> > Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
> ><br>
><br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started:<br>
> <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
><br>
><br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started:<br>
> <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
><br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br>