<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:'times new roman', 'new york', times, serif;font-size:12pt"><div>Andrew,</div><div><br></div><div>Thanks for the help</div><div><br></div><div>Comments inline with <BS></div><div style="font-family:times new roman, new york, times, serif;font-size:12pt"><br><div style="font-family:arial, helvetica, sans-serif;font-size:13px"><font size="2" face="Tahoma"><hr size="1"><b><span style="font-weight: bold;">From:</span></b> Andrew Beekhof <andrew@beekhof.net><br><b><span style="font-weight: bold;">To:</span></b> Bob Schatz <bschatz@yahoo.com><br><b><span style="font-weight: bold;">Cc:</span></b> The Pacemaker cluster resource manager <pacemaker@oss.clusterlabs.org><br><b><span style="font-weight: bold;">Sent:</span></b> Thu, April 14, 2011 2:14:40 AM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [Pacemaker] Question
regarding starting of master/slave resources and ELECTIONs<br></font><br>
On Thu, Apr 14, 2011 at 10:49 AM, Andrew Beekhof <<a ymailto="mailto:andrew@beekhof.net" href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>> wrote:<br><br>>>> I noticed that 4 of the master/slave resources will start right away but<br>>>> the<br>>>> 5 master/slave resource seems to take a minute or so and I am only running<br>>>> with one node.<br>>>> Is this expected?<br>>><br>>> Probably, if the other 4 take around a minute each to start.<br>>> There is an lrmd config variable that controls how much parallelism it<br>>> allows (but i forget the name).<br>>> <Bob> It's max-children and I set it to 40 for this test to see if it would<br>>> change the behavior. (/sbin/lrmadmin -p max-children 40)<br>><br>> Thats surprising. I'll have a look at the logs.<br><br>Looking at the logs, I see a couple of things:<br><br><br>This is very bad:<br>Apr
12 19:33:42 mgraid-S000030311-1 crmd: [17529]: WARN: get_uuid:<br>Could not calculate UUID for mgraid-s000030311-0<br>Apr 12 19:33:42 mgraid-S000030311-1 crmd: [17529]: WARN:<br>populate_cib_nodes_ha: Node mgraid-s000030311-0: no uuid found<br><br>For some reason pacemaker cant get the node's uuid from heartbeat.<br><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:13px"><BS> I create the uuid when the node comes up.<br><br>So we start a few things:<br><br>Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>do_lrm_rsc_op: Performing<br>key=23:3:0:48aac631-8177-4cda-94ea-48dfa9b1a90f<br>op=SSS000030311:0_start_0 )<br>Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>do_lrm_rsc_op: Performing<br>key=49:3:0:48aac631-8177-4cda-94ea-48dfa9b1a90f<br>op=SSJ000030312:0_start_0 )<br>Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>do_lrm_rsc_op:
Performing<br>key=75:3:0:48aac631-8177-4cda-94ea-48dfa9b1a90f<br>op=SSJ000030313:0_start_0 )<br>Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>do_lrm_rsc_op: Performing<br>key=101:3:0:48aac631-8177-4cda-94ea-48dfa9b1a90f<br>op=SSJ000030314:0_start_0 )<br><br>But then another change comes in:<br><br>Apr 12 19:33:41 mgraid-S000030311-1 crmd: [17529]: info:<br>abort_transition_graph: need_abort:59 - Triggered transition abort<br>(complete=0) : Non-status change<br><br>Normally we'd recompute and keep going, but it was a(nother) replace<br>operation, so:<br><br>Apr 12 19:33:42 mgraid-S000030311-1 crmd: [17529]: info:<br>do_state_transition: State transition S_TRANSITION_ENGINE -><br>S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL<br>origin=do_cib_replaced ]<br><br>All the time goes here:<br><br>Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>action_timer_callback: Timer popped (timeout=20000,<br>abort_level=1000000,
complete=true)<br>Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>action_timer_callback: Ignoring timeout while not in transition<br>Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>action_timer_callback: Timer popped (timeout=20000,<br>abort_level=1000000, complete=true)<br>Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>action_timer_callback: Ignoring timeout while not in transition<br>Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>action_timer_callback: Timer popped (timeout=20000,<br>abort_level=1000000, complete=true)<br>Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>action_timer_callback: Ignoring timeout while not in transition<br>Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>action_timer_callback: Timer popped (timeout=20000,<br>abort_level=1000000, complete=true)<br>Apr 12 19:35:31 mgraid-S000030311-1 crmd: [17529]: WARN:<br>action_timer_callback: Ignoring timeout while
not in transition<br>Apr 12 19:37:00 mgraid-S000030311-1 crmd: [17529]: ERROR:<br>crm_timer_popped: Integration Timer (I_INTEGRATED) just popped!<br><br>but its not at all clear to me why - although certainly avoiding the<br>election would help.<br>Is there any chance to load all the changes at once?<br><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:13px"><BS> Yes. That worked. I created the configuration in a file and then did a "crm configure load update <filename>" to avoid the election</div><div style="font-family:arial, helvetica, sans-serif;font-size:13px"><br>Possibly the delay related to the UUID issue above, possibly it might<br>be related to one of these two patches that went in after 1.0.9<br><br>andrew (stable-1.0) High: crmd: Make sure we always poke the FSA after<br>a transition to clear any TE_HALT actions CS: 9187c0506fd3 On:<br>2010-07-07<br>andrew
(stable-1.0) High: crmd: Reschedule the PE_START action if its<br>not already running when we try to use it CS: e44dfe49e448 On:<br>2010-11-11<br><br>Could you try turning on debug and/or a more recent version?<br></div><div style="font-family:arial, helvetica, sans-serif;font-size:13px"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:13px"><BS> I turned on debug and grabbed the logs, configuration and and /var/lib/pengine directory. They are attached.</div><div style="font-family:arial, helvetica, sans-serif;font-size:13px"> Unfortunately I cannot try a new version with this hardware at this time. :(</div><div style="font-family:arial, helvetica, sans-serif;font-size:13px"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:13px"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:13px">Thanks,</div><div style="font-family:arial,
helvetica, sans-serif;font-size:13px"><br></div><div style="font-family:arial, helvetica, sans-serif;font-size:13px">Bob</div></div><div style="position:fixed"></div>
</div></body></html>