<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">We found one of our cluster nodes down this morning. The server was up but cluster services were not running. Upon examination of the logs, we found that the cluster just stopped around 9:40:31 and then I started it up manually (pcs cluster
start) at 11:49:48. I can’t imagine that Pacemaker just randomly terminates. Any thoughts why it would behave this way?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:25:31 [92170] 001store01a pengine: notice: process_pe_message: Calculated transition 91482, saving inputs in /var/lib/pacemaker/pengine/pe-input-756.bz2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:25:31 [92171] 001store01a crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:25:31 [92171] 001store01a crmd: info: do_te_invoke: Processing graph 91482 (ref=pe_calc-dc-1622121931-124396) derived from /var/lib/pacemaker/pengine/pe-input-756.bz2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:25:31 [92171] 001store01a crmd: notice: run_graph: Transition 91482 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-756.bz2):
Complete<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:25:31 [92171] 001store01a crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:25:31 [92171] 001store01a crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92171] 001store01a crmd: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (900000ms)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92171] 001store01a crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92171] 001store01a crmd: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: process_pe_message: Input has not changed since last time, not saving to disk<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: determine_online_status: Node 001store01a is online<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: determine_op_status: Operation monitor found resource p_pure-ftpd-itls active on 001store01a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: warning: unpack_rsc_op_failure: Processing failed op monitor for p_vip_ftpclust01 on 001store01a: unknown error (1)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: determine_op_status: Operation monitor found resource p_pure-ftpd-etls active on 001store01a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: unpack_node_loop: Node 1 is already processed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: unpack_node_loop: Node 1 is already processed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: common_print: p_vip_ftpclust01 (ocf::heartbeat:IPaddr2): Started 001store01a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: common_print: p_replicator (systemd:pure-replicator): Started 001store01a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: common_print: p_pure-ftpd-etls (systemd:pure-ftpd-etls): Started 001store01a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: common_print: p_pure-ftpd-itls (systemd:pure-ftpd-itls): Started 001store01a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: LogActions: Leave p_vip_ftpclust01 (Started 001store01a)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: LogActions: Leave p_replicator (Started 001store01a)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: LogActions: Leave p_pure-ftpd-etls (Started 001store01a)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: info: LogActions: Leave p_pure-ftpd-itls (Started 001store01a)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92170] 001store01a pengine: notice: process_pe_message: Calculated transition 91483, saving inputs in /var/lib/pacemaker/pengine/pe-input-756.bz2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92171] 001store01a crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92171] 001store01a crmd: info: do_te_invoke: Processing graph 91483 (ref=pe_calc-dc-1622122831-124397) derived from /var/lib/pacemaker/pengine/pe-input-756.bz2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92171] 001store01a crmd: notice: run_graph: Transition 91483 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-756.bz2):
Complete<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92171] 001store01a crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 09:40:31 [92171] 001store01a crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [MAIN ] Corosync Cluster Engine ('2.4.3'): started and ready to provide service.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncinfo [MAIN ] Corosync built-in features: dbus systemd xmlconf qdevices qnetd snmp libcgroup pie relro bindnow<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [TOTEM ] Initializing transport (UDP/IP Unicast).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [TOTEM ] Initializing transmit/receive security (NSS) crypto: none hash: none<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [TOTEM ] The network interface [10.51.14.40] is now up.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [SERV ] Service engine loaded: corosync configuration map access [0]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncinfo [QB ] server name: cmap<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [SERV ] Service engine loaded: corosync configuration service [1]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncinfo [QB ] server name: cfg<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncinfo [QB ] server name: cpg<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [SERV ] Service engine loaded: corosync profile loading service [4]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [QUORUM] Using quorum provider corosync_votequorum<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncinfo [QB ] server name: votequorum<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncinfo [QB ] server name: quorum<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [TOTEM ] adding new UDPU member {10.51.14.40}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [TOTEM ] adding new UDPU member {10.51.14.41}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [TOTEM ] A new membership (10.51.14.40:6412) was formed. Members joined: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [VOTEQ ] Waiting for all cluster members. Current votes: 1 expected_votes: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [QUORUM] Members[1]: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">[10667] 001store01a.ccnva.local corosyncnotice [MAIN ] Completed service synchronization, ready to provide service.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: notice: main: Starting Pacemaker 1.1.18-11.el7_5.3 | build=2b07d5c5a9 features: generated-manpages agent-manpages ncurses
libqb-logging libqb-ipc systemd nagios corosync-native atomic-attrd acls<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: main: Maximum core file size is: 18446744073709551615<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: qb_ipcs_us_publish: server name: pacemakerd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: crm_get_peer: Created entry 05ad8b08-25a3-4a2d-84cb-1fc355fb697c/0x55d844a446b0 for node 001store01a/1 (1 total)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: crm_get_peer: Node 1 is now known as 001store01a<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: crm_get_peer: Node 1 has uuid 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: crm_update_peer_proc: cluster_connect_cpg: Node 001store01a[1] - corosync-cpg is now online<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: warning: cluster_connect_quorum: Quorum lost<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: crm_get_peer: Created entry 2f1f038e-9cc1-4a43-bab9-e7c91ca0bf3f/0x55d844a45ee0 for node 001store01b/2 (2 total)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: crm_get_peer: Node 2 is now known as 001store01b<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: crm_get_peer: Node 2 has uuid 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: start_child: Using uid=189 and group=189 for process cib<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: start_child: Forked child 10682 for process cib<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: start_child: Forked child 10683 for process stonith-ng<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: start_child: Forked child 10684 for process lrmd<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:Consolas">May 27 11:49:48 [10681] 001store01a.ccnva.local pacemakerd: info: start_child: Using uid=189 and group=189 for process attrd<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><img width="500" height="96" style="width:5.2083in;height:1.0in" id="Picture_x0020_1" src="cid:image001.png@01D752F6.E50B3B70"><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this
email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility
for any loss or damage arising from the use of this email or attachments.
</body>
</html>