<div dir="ltr">Hi,<div><br></div><div>I have setup a 2 node cluster, using the following packages:</div><div><br></div><div>pacemaker 1.1.10+git20130802-1ubuntu2<br></div><div>corosync 2.3.3-1ubuntu1<br>
</div><div><br></div><div>My cluster config is as so:</div><div><br></div><div><div>node $id="12303" ldb03</div><div>node $id="12304" ldb04</div><div>primitive p_fence_ldb03 stonith:external/vcenter \</div>
<div> params VI_SERVER="10.17.248.10" VI_CREDSTORE="/root/.vmware/credstore/vicredentials.xml" HOSTLIST="ldb03=ldb03" RESETPOWERON="0" pcmk_host_check="static-list" pcmk_host_list="ldb03" \</div>
<div> op start interval="0" timeout="500s"</div><div>primitive p_fence_ldb04 stonith:external/vcenter \</div><div> params VI_SERVER="10.17.248.10" VI_CREDSTORE="/root/.vmware/credstore/vicredentials.xml" HOSTLIST="ldb04=ldb04" RESETPOWERON="0" pcmk_host_check="static-list" pcmk_host_list="ldb04" \</div>
<div> op start interval="0" timeout="500s"</div><div>primitive p_fs_mysql ocf:heartbeat:Filesystem \</div><div> params device="nfsserver:/LDB_Cluster1" directory="/var/lib/mysql" fstype="nfs" options="relatime,rw,hard,nointr,rsize=32768,wsize=32768,bg,vers=3,proto=tcp" \</div>
<div> op start interval="0" timeout="60s" \</div><div> op stop interval="0" timeout="120s" \</div><div> op monitor interval="60s" timeout="60s" \</div>
<div> meta is-managed="true"</div><div>primitive p_ip_1 ocf:heartbeat:IPaddr2 \</div><div> params ip="10.10.10.11" cidr_netmask="25" \</div><div> op monitor interval="30s" \</div>
<div> meta target-role="Started" is-managed="true"</div><div>primitive p_ip_2 ocf:heartbeat:IPaddr2 \</div><div> params ip="10.10.10.12" cidr_netmask="25" \</div><div>
op monitor interval="30s" \</div><div> meta target-role="Started" is-managed="true"</div><div>primitive p_ip_3 ocf:heartbeat:IPaddr2 \</div><div> params ip="10.10.10.13" cidr_netmask="25" \</div>
<div> op monitor interval="30s" \</div><div> meta target-role="Started" is-managed="true"</div><div>primitive p_mysql ocf:heartbeat:mysql \</div><div> params datadir="/var/lib/mysql" binary="/usr/bin/mysqld_safe" socket="/var/run/mysqld/mysqld.sock" \</div>
<div> op start interval="0" timeout="120" \</div><div> op stop interval="0" timeout="120" \</div><div> op monitor interval="20" timeout="30" \</div>
<div> meta target-role="Started" is-managed="true"</div><div>group g_mysql p_fs_mysql p_mysql p_ip_1 p_ip_2 p_ip_3 \</div></div><div><div>location l_fence_ldb03 p_fence_ldb03 -inf: ldb03<br></div>
<div>location l_fence_ldb04 p_fence_ldb04 -inf: ldb04</div><div>property $id="cib-bootstrap-options" \</div><div> dc-version="1.1.10-42f2063" \</div><div> cluster-infrastructure="corosync" \</div>
<div> no-quorum-policy="ignore" \</div><div> stonith-enabled="true" \</div><div> stop-all-resources="false" \</div><div> expected-quorum-votes="2" \</div>
<div> last-lrm-refresh="1407325251"</div></div><div><br></div><div><br></div><div>This exact configuration has worked during the setup, but I have encountered a problem with my inactive node ldb03. Corosync shows this node as up:</div>
<div><br></div><div><div>root@ldb03:~# corosync-cmapctl | grep members</div><div>runtime.totem.pg.mrp.srp.members.12303.config_version (u64) = 0</div><div>runtime.totem.pg.mrp.srp.members.12303.ip (str) = r(0) ip(10.10.10.8)</div>
<div>runtime.totem.pg.mrp.srp.members.12303.join_count (u32) = 1</div><div>runtime.totem.pg.mrp.srp.members.12303.status (str) = joined</div><div>runtime.totem.pg.mrp.srp.members.12304.config_version (u64) = 0</div><div>runtime.totem.pg.mrp.srp.members.12304.ip (str) = r(0) ip(10.10.10.9)</div>
<div>runtime.totem.pg.mrp.srp.members.12304.join_count (u32) = 1</div><div>runtime.totem.pg.mrp.srp.members.12304.status (str) = joined</div></div><div><br></div><div>and crm status and crm node status show it as online:</div>
<div><br></div><div><div>Last updated: Wed Aug 6 14:16:24 2014<br></div><div>Last change: Wed Aug 6 14:02:00 2014 via crm_resource on ldb04</div><div>Stack: corosync</div><div>Current DC: ldb04 (12304) - partition with quorum</div>
<div>Version: 1.1.10-42f2063</div><div>2 Nodes configured</div><div>7 Resources configured</div><div>Online: [ ldb03 ldb04 ]<br></div></div><div><br></div><div><div>root@ldb03:~# crm node status</div><div><nodes></div>
<div> <node id="12304" uname="ldb04"/></div><div> <node id="12303" uname="ldb03"/></div><div></nodes></div></div><div><br></div><div><br></div><div>but....after seeing this entry in my logs:</div>
<div>Aug 6 13:26:23 ldb03 cibadmin[2140]: notice: crm_log_args: Invoked: cibadmin -M -c -o status --xml-text <node_state id="ldb03" uname="ldb03" ha="active" in_ccm="false" crmd="offline" join="member" expected="down" crm-debug-origin="manual_clear" shutdown="0"/><br>
</div><div><br></div><div>I noticed that cibadmin shows it as normal(offline)</div><div><div>root@ldb03:~# crm node show</div><div>ldb04(12304): normal</div><div>ldb03(12303): normal(offline)</div></div><div><br></div><div>
The offline state is not present in anything but cibadmin. Not the cib.xml, not corosync-quorumtool and a tcpdump shows multicast traffic from both hosts.</div><div><br></div><div>I tried (hesitantly) to delete the line using cibadmin, but I couldn't quite get the syntax right. Any tips on how to get this node to show as online and subsequently be able to run resources? Currently, when I run crm resource move, this has no effect, no errors and nothing noticeable in the logfiles either.</div>
<div><br></div><div>Sorry for long thread....I can attach more logs/config if necessary.</div><div><br></div><div>Thanks,</div><div><br></div><div>Jamie.<br></div></div>