<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Hi,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>We have a Mysql cluster which works fine when I have a single master (“A”) and slave (“B”). Failover is almost immediate and I am happy with this approach.<o:p></o:p></p><p class=MsoNormal>When we configured two additional slaves, strange things start to happen. From time to time I am noticing that all slaves mysql instances are restarted and I cannot figure out why.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I tried to find out what is happening, and this is how far I got:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There is a repeating sequence in the DC, which looks like this when everything is fine:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Sep 10 01:45:42 oamgr crmd: [3385]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Sep 10 01:45:42 oamgr crmd: [3385]: info: do_te_invoke: Processing graph 71358 (ref=pe_calc-dc-1378777542-165977) derived from /var/lib/pengine/pe-input-3179.bz2<o:p></o:p></p><p class=MsoNormal>Sep 10 01:45:42 oamgr crmd: [3385]: notice: run_graph: ==== Transition 71358 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-3179.bz2): Complete<o:p></o:p></p><p class=MsoNormal>Sep 10 01:45:42 oamgr crmd: [3385]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<o:p></o:p></p><p class=MsoNormal>Sep 10 01:47:42 oamgr crmd: [3385]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (120000ms)<o:p></o:p></p><p class=MsoNormal>Sep 10 01:47:42 oamgr crmd: [3385]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]<o:p></o:p></p><p class=MsoNormal>Sep 10 01:47:42 oamgr crmd: [3385]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED<o:p></o:p></p><p class=MsoNormal>….<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>But <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>It looks somewhat different when I see the restarts:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>….<o:p></o:p></p><p class=MsoNormal>Sep 10 01:51:42 oamgr crmd: [3385]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Sep 10 01:51:42 oamgr crmd: [3385]: info: do_te_invoke: Processing graph 71361 (ref=pe_calc-dc-1378777902-165980) derived from /var/lib/pengine/pe-input-3179.bz2<o:p></o:p></p><p class=MsoNormal>Sep 10 01:51:42 oamgr crmd: [3385]: notice: run_graph: ==== Transition 71361 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-3179.bz2): Complete<o:p></o:p></p><p class=MsoNormal>Sep 10 01:51:42 oamgr crmd: [3385]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-oadb2-master-db-mysql.1, name=master-db-mysql:1, value=0, magic=NA, cib=0.4829.3480) : Transient attribute: update<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-oadb2-readable, name=readable, value=0, magic=NA, cib=0.4829.3481) : Transient attribute: update<o:p></o:p></p><p class=MsoNormal>…..<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>There is a transaction abort, and shortly after this, the slaves are restarted:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>….<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr pengine: [3384]: notice: LogActions: Move db-mysql:1 (Slave oadb2 -> huoadb1)<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr pengine: [3384]: notice: LogActions: Move db-mysql:2 (Slave huoadb1 -> oadb2)<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: info: do_te_invoke: Processing graph 71362 (ref=pe_calc-dc-1378777965-165981) derived from /var/lib/pengine/pe-input-3180.bz2<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: info: te_rsc_command: Initiating action 148: notify db-mysql:0_pre_notify_stop_0 on oadb1<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: info: te_rsc_command: Initiating action 150: notify db-mysql:1_pre_notify_stop_0 on oadb2<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: info: te_rsc_command: Initiating action 151: notify db-mysql:2_pre_notify_stop_0 on huoadb1<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: info: te_rsc_command: Initiating action 152: notify db-mysql:3_pre_notify_stop_0 on huoadb2<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr pengine: [3384]: notice: process_pe_message: Transition 71362: PEngine Input stored in: /var/lib/pengine/pe-input-3180.bz2<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: info: te_rsc_command: Initiating action 39: stop db-mysql:1_stop_0 on oadb2<o:p></o:p></p><p class=MsoNormal>Sep 10 01:52:45 oamgr crmd: [3385]: info: te_rsc_command: Initiating action 43: stop db-mysql:2_stop_0 on huoadb1<o:p></o:p></p><p class=MsoNormal>….<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>It appears that oadb2 and huoadb1 are replaced with each other (in terms of db-mysql:1 and db-mysql:2 )? Does that make any sense?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>It happens only when I have all 4 mysql nodes online. (oadb1, oadb2, huoadb1, huoadb2). When I moved oadb2 to standby for a day, I did not see restarts.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Could someone help me troubleshoot this?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Mysql version is 5.1.66<o:p></o:p></p><p class=MsoNormal>Pacemaker 1.1.7<o:p></o:p></p><p class=MsoNormal>Corosync 1.4.2<o:p></o:p></p><p class=MsoNormal>Mysql RA is the latest from github<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks in advance,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Attila<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>