<div dir="ltr">Thanks Ken,<div><br></div><div>We're currently using Pacemaker 1.1.11 and at the moment its not an option to upgrade.<br></div><div>I've spun up and down these boxes on AWS and even tried different sizes. I think a recent upgrade broke this deploy.</div><div><br></div><div>This is the output from dmesg:</div><div><br></div><div><div>cib[16656] general protection ip:7f45391e9545 sp:7ffddf16c8b8 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7f45390be000+18a000]</div><div>cib[16659] general protection ip:7fa36fa89545 sp:7ffe28416288 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7fa36f95e000+18a000]</div><div>cib[16663] general protection ip:7fa3defce545 sp:7ffeb5b29c58 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7fa3deea3000+18a000]</div><div>cib[16666] general protection ip:7fa1cefe4545 sp:7ffcc4b9c778 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7fa1ceeb9000+18a000]</div><div>cib[16669] general protection ip:7f4b3900f545 sp:7ffdcd65aaf8 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7f4b38ee4000+18a000]</div><div>cib[16672] general protection ip:7fc38be2b545 sp:7fffbc7e1598 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7fc38bd00000+18a000]</div><div>cib[16675] general protection ip:7f9c6890c545 sp:7ffca09539f8 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7f9c687e1000+18a000]</div><div>cib[16678] general protection ip:7f1c636ad545 sp:7ffc677d2008 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7f1c63582000+18a000]</div><div>cib[16681] general protection ip:7fed0b47e545 sp:7ffd051f0618 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7fed0b353000+18a000]</div><div>cib[16684] general protection ip:7f2ee87cd545 sp:7fff8d9ae288 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7f2ee86a2000+18a000]</div><div>cib[16687] general protection ip:7f41c3789545 sp:7fff9f005848 error:0 in <a href="http://libc-2.12.so">libc-2.12.so</a>[7f41c365e000+18a000]</div></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 10, 2015 at 9:54 AM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 08/09/2015 02:27 PM, David Neudorfer wrote:<br>
> Where can I dig deeper to figure out why cib keeps terminating? selinux and<br>
> iptables are both disabled and I've have debug enabled. Google hasn't been<br>
> able to help me thus far.<br>
><br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: debug:<br>
> get_local_nodeid: Local nodeid is 84939948<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> plugin_get_details: Server details: id=84939948 uname=ip-172-20-16-5<br>
> cname=pcmk<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> crm_get_peer: Created entry<br>
> c1f204b2-c994-48d9-81b6-87e1a7fc1ee7/0xa2c460 for node<br>
> ip-172-20-16-5/84939948 (1 total)<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> crm_get_peer: Node 84939948 is now known as ip-172-20-16-5<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> crm_get_peer: Node 84939948 has uuid ip-172-20-16-5<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> crm_update_peer_proc: init_cs_connection_classic: Node<br>
> ip-172-20-16-5[84939948] - unknown is now online<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> init_cs_connection_once: Connection to 'classic openais (with<br>
> plugin)': established<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: notice:<br>
> get_node_name: Defaulting to uname -n for the local classic openais<br>
> (with plugin) node name<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> qb_ipcs_us_publish: server name: cib_ro<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> qb_ipcs_us_publish: server name: cib_rw<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> qb_ipcs_us_publish: server name: cib_shm<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info: cib_init:<br>
> Starting cib mainloop<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: notice:<br>
> plugin_handle_membership: Membership 104: quorum acquired<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> crm_update_peer_proc: plugin_handle_membership: Node<br>
> ip-172-20-16-5[84939948] - unknown is now member<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: notice:<br>
> crm_update_peer_state: cib_peer_update_callback: Node<br>
> ip-172-20-16-5[84939948] - state is now lost (was (null))<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: notice:<br>
> crm_reap_dead_member: Removing ip-172-20-16-5/84939948 from the<br>
> membership list<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: notice:<br>
> reap_crm_member: Purged 1 peers with id=84939948 and/or uname=(null)<br>
> from the membership cache<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: notice:<br>
> crm_update_peer_state: plugin_handle_membership: Node ��[2077843320]<br>
> - state is now member (was member)<br>
> Aug 09 18:54:29 [12526] ip-172-20-16-5 cib: info:<br>
> crm_update_peer: plugin_handle_membership: Node ��: id=2077843320<br>
> state=r(0) ip(172.20.16.5) addr=r(0) ip(172.20.16.5) (new) votes=1<br>
> (new) born=104 seen=104 proc=00000000000000000000000000111312<br>
<br>
</div></div>The unprintable characters strongly implies memory corruption. There are<br>
known issues with that when using the legacy plugin with some versions<br>
of pacemaker. What version are you using? If you are compiling yourself,<br>
I would recommend using the current upstream master branch (not 1.1.13,<br>
which has the issue).<br>
<br>
An even better solution would be to switch to corosync 2 instead of the<br>
plugin, as corosync 2 gets more development and testing these days.<br>
<br>
><br>
> <a href="https://gist.github.com/davidneudorfer/bc97082a9d9dfb12985b" rel="noreferrer" target="_blank">https://gist.github.com/davidneudorfer/bc97082a9d9dfb12985b</a><br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><p style="margin:0in 0in 0.0001pt"><font face="Arial, sans-serif"><span style="font-size:15px">David Neudorfer</span></font></p><p style="margin:0in 0in 0.0001pt"><font face="Arial, sans-serif"><span style="font-size:15px"></span></font><span style="font-size:13.3333px;border-collapse:collapse">Automation Engineer<br></span></p><p style="margin:0in 0in 0.0001pt"><span style="font-size:13.3333px;border-collapse:collapse"><a href="http://www.google.com/url?q=http%3A%2F%2Fwww.warbyparker.com%2F&sa=D&sntz=1&usg=AFrqEzfkTF4rhdwjlARqomkRV4TIc34NxA" style="color:blue;font-size:9pt;font-family:Arial,sans-serif" target="_blank">WARBY PARKER</a></span></p></div></div></div></div></div></div></div></div>
</div>