David and Jake,<br><br>Thanks for the replies, that's exactly what I needed. I think where I was going wrong was playing with the role="Master" and role="Slave" parameters, but always having the same timeout for both. (crm_verify -L confused me even more on what I was doing wrong)<br>
<br>Speaking of timeouts, why should I stay above 20 seconds for a monitoring interval? If the service crashes for whatever reason, I'd like to avoid downtime and keep the failover window as quick as possible. If this makes any difference, Redis and Memcache are different beasts compared to databases. The clients are usually done with their requests after 20 milliseconds. It's very fast paced, and 20 seconds of downtime is pretty significant in this application.<br>
<br>Also David, after increasing my interval to 20 seconds and doing a separate monitor definition for Master and Slave, both checks still run duplicate monitor ops. (See below) Probably not a big deal, but I was just curious on why Pacemaker would do that.<br>
<br>Thanks!<br><br>--Cal<br><br><br>------------------------------------------------------------------------------------------------------------------------------------------<br>redis(redis:0)[3600]: 2012/11/13_08:40:39 INFO: redis-server is running (PID 15221)<br>
redis(redis:0)[3600]: 2012/11/13_08:40:39 INFO: redis-server is running (PID 15221)<br>redis(redis:0)[3600]: 2012/11/13_08:40:39 INFO: redis-server is running (as master)<br>redis(redis:0)[3600]: 2012/11/13_08:40:39 INFO: redis-server is running (as master)<br>
redis(redis:0)[4175]: 2012/11/13_08:40:59 INFO: redis-server is running (PID 15221)<br>redis(redis:0)[4175]: 2012/11/13_08:40:59 INFO: redis-server is running (PID 15221)<br>redis(redis:0)[4175]: 2012/11/13_08:40:59 INFO: redis-server is running (as master)<br>
redis(redis:0)[4175]: 2012/11/13_08:40:59 INFO: redis-server is running (as master)<br>------------------------------------------------------------------------------------------------------------------------------------------<br>
redis(redis:1)[7690]: 2012/11/13_08:31:06 INFO: redis-server is running (PID 7330)<br>redis(redis:1)[7690]: 2012/11/13_08:31:06 INFO: redis-server is running (PID 7330)<br>redis(redis:1)[7690]: 2012/11/13_08:31:06 INFO: redis-server is running (as slave)<br>
redis(redis:1)[7690]: 2012/11/13_08:31:06 INFO: redis-server is running (as slave)<br>redis(redis:1)[7729]: 2012/11/13_08:31:28 INFO: redis-server is running (PID 7330)<br>redis(redis:1)[7729]: 2012/11/13_08:31:28 INFO: redis-server is running (PID 7330)<br>
redis(redis:1)[7729]: 2012/11/13_08:31:28 INFO: redis-server is running (as slave)<br>redis(redis:1)[7729]: 2012/11/13_08:31:28 INFO: redis-server is running (as slave)<br>------------------------------------------------------------------------------------------------------------------------------------------<br>
<br><br><div class="gmail_quote">On Mon, Nov 12, 2012 at 3:37 PM, David Vossel <span dir="ltr"><<a href="mailto:dvossel@redhat.com" target="_blank">dvossel@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">----- Original Message -----<br>
> From: "Cal Heldenbrand" <<a href="mailto:cal@fbsdata.com">cal@fbsdata.com</a>><br>
> To: "The Pacemaker cluster resource manager" <<a href="mailto:pacemaker@oss.clusterlabs.org">pacemaker@oss.clusterlabs.org</a>><br>
</div><div class="im">> Sent: Monday, November 12, 2012 3:27:02 PM<br>
> Subject: Re: [Pacemaker] Custom event script during master/slave failover?<br>
><br>
><br>
> I've managed to get a test Redis configuration up and running...<br>
> however this is the first time I've played with a master/slave<br>
> resource, and I'm noticing some odd behavior in the monitoring<br>
> actions. Setting in a super quick monitoring interval of 1 second, I<br>
> noticed at first only the slave was being monitored, 2x in each<br>
> second. Then after changing a bunch of config options and several<br>
> cluster restarts, I noticed the monitoring action had moved to the<br>
> master, again 2x events in each second.<br>
<br>
</div>Use a longer timeout than 1s. Don't go under 20s... that might be why you see 2x events.<br>
<br>
The monitor action didn't move to the master or move to the slave. There is a separate monitor operation used for resources in the master and slave role. Both monitor operations have to be defined, looking at your config it appears only the Master monitor op is defined.<br>
<br>
<a href="http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_monitoring_multi_state_resources.html" target="_blank">http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/_monitoring_multi_state_resources.html</a><br>
<br>
If you are new to multi-state resources. I'd recommend playing around with the ocf:pacemaker:Stateful resource some. Stateful is a fake resource that does nothing except allow you to start/stop/promote/demote/monitor it. It is useful for testing out configurations and understanding expected behavior.<br>
<span class="HOEnZb"><font color="#888888"><br>
-- Vossel<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
><br>
> redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is<br>
> running (PID 30759)<br>
> redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is<br>
> running (PID 30759)<br>
> redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is<br>
> running (as master)<br>
> redis(redis:1)[2203]: 2012/11/12_15:17:52 INFO: redis-server is<br>
> running (as master)<br>
><br>
> Here is my primitive and master/slave definition:<br>
><br>
> primitive redis ocf:heartbeat:redis \<br>
> meta target-role="Master" is-managed="true" \<br>
> op monitor interval="1s" role="Master" timeout="5s" on-fail="restart"<br>
> ms redis_clone redis \<br>
> meta notify="true" is-managed="true" ordered="false"<br>
> interleave="false" globally-unique="false" target-role="Master"<br>
> migration-threshold="1"<br>
><br>
> I'd like to figure out how to make sure both master and slave are<br>
> actively monitored. Let me know if you need any more info on my<br>
> config!<br>
><br>
> Thank you,<br>
><br>
> --Cal<br>
><br>
><br>
><br>
> On Fri, Nov 9, 2012 at 4:52 PM, David Vossel < <a href="mailto:dvossel@redhat.com">dvossel@redhat.com</a> ><br>
> wrote:<br>
><br>
><br>
><br>
><br>
><br>
> ----- Original Message -----<br>
> > From: "Cal Heldenbrand" < <a href="mailto:cal@fbsdata.com">cal@fbsdata.com</a> ><br>
> > To: "The Pacemaker cluster resource manager" <<br>
> > <a href="mailto:pacemaker@oss.clusterlabs.org">pacemaker@oss.clusterlabs.org</a> ><br>
> > Sent: Friday, November 9, 2012 4:29:05 PM<br>
> > Subject: [Pacemaker] Custom event script during master/slave<br>
> > failover?<br>
> ><br>
> ><br>
> > Hi everyone,<br>
> ><br>
> > I'm playing around with the possibility of using Pacemaker in a<br>
> > Redis<br>
> > master/slave cluster. The difficult part of this, is that Redis<br>
> > will<br>
> > not automatically flip itself from read-only slave mode into master<br>
> > mode. A client needs to connect to the slave server and run this<br>
> > SLAVEOF NO ONE command to turn it into a master.<br>
><br>
> The resource agent should be able to do this. Pacemaker will tell the<br>
> resource agent to promote/demote a resource. Your Redis agent can<br>
> then go off and do whatever logic is necessary to actually<br>
> transition the process a master/slave process.<br>
><br>
><br>
> ><br>
> > If I set up a master/slave primitive, is it possible to fire an<br>
> > event<br>
> > script during a failover event that would allow me to do this?<br>
> ><br>
> > After the stonith event fires and the broken master reboots,<br>
> > I would<br>
> > write in something special in the Redis init.d script to detect if<br>
> > my peer is already a master, and start up in slave mode.<br>
><br>
> If you are using Pacemaker, it should be controlling what node has<br>
> the master resource and what node has the slave resource. Take a<br>
> look at what OCF scripts are and how pacemaker uses them.<br>
><br>
><br>
> ><br>
> > Has anyone else worked up a cluster config with Redis?<br>
><br>
> Using google, found some work someone did on a redis resource agent.<br>
> <a href="https://github.com/martinwalter/resource-agents/blob/master/heartbeat/redis" target="_blank">https://github.com/martinwalter/resource-agents/blob/master/heartbeat/redis</a><br>
><br>
> -- Vossel<br>
> > Thank you,<br>
> ><br>
> > --Cal<br>
> ><br>
> > _______________________________________________<br>
> > Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> > <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
> ><br>
> > Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> > Getting started:<br>
> > <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> > Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
> ><br>
><br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started:<br>
> <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
><br>
><br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started:<br>
> <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
><br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br>