<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>I'm trying to setup a 2-node HA iSCSI cluster but I am having some problems with the failover. I want each node to be active with half of the storage available in each node. So each node will have half the storage and it will be served by different IP/target/lun. I have it working until I have an initiator setup and test failover. If I have an initiator (proxmox) that is using the target/lun, then when I migrate lun/target/drbd to the other node, I get inconsistent results.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Sometimes it works fine, but it usually fails. It seems to be that it fails in stopping the target after stopping the LUN. At that point the node is fenced (rebooted). After fencing, sometimes the resources will migrate to the surviving node successfully, sometimes it will wait until the fenced node comes back online and then move the resources, and sometimes it will just remain in the failed state. Sometimes, after the fenced node boots back up, it will try and move the resources back to itself and this will fail and the two nodes will continue to fence each other until there is some manual intervention.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Any assistance if getting this setup correctly would be appreciated. Relevant details below.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Setup would look like:<o:p></o:p></p><p class=MsoNormal>On Storage1001A<o:p></o:p></p><p class=MsoNormal>Lun:vol01 - Target:data01 - IP1/IP11 - LV:lv_vol01 - VG:vg_data01 - DRBD:data01<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>On Storage1001B<o:p></o:p></p><p class=MsoNormal>Lun:vol01 - Target:data02 - IP2/IP22 - LV:lv_vol01 - VG:vg_data02 - DRBD:data02<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Versions:<o:p></o:p></p><p class=MsoNormal>Linux Storage1001A.xdomain.com 3.2.0-4-amd64 #1 SMP Debian 3.2.51-1 x86_64 GNU/Linux<o:p></o:p></p><p class=MsoNormal>targetcli 2.0rc1-2 <o:p></o:p></p><p class=MsoNormal>lio-utils 3.1+git2.fd0b34fd-2 <o:p></o:p></p><p class=MsoNormal>drbd8-utils 2:8.3.13-2 <o:p></o:p></p><p class=MsoNormal>corosync 1.4.2-3<o:p></o:p></p><p class=MsoNormal>pacemaker 1.1.7-1 <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Network is setup with 3 bonded pairs<o:p></o:p></p><p class=MsoNormal>- Bond0: Client interface<o:p></o:p></p><p class=MsoNormal>- Bond1: Crossover<o:p></o:p></p><p class=MsoNormal>- Bond2: Client interface (for future multicast)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Log file shows<o:p></o:p></p><p class=MsoNormal>- Working fine<o:p></o:p></p><p class=MsoNormal>- Migrate from Storage1001A to Storage1001B<o:p></o:p></p><p class=MsoNormal>- Storage1001A hangs/crashes after stoping LUN before stoppin Target<o:p></o:p></p><p class=MsoNormal>- Storage1001A is fenced<o:p></o:p></p><p class=MsoNormal>- Resources migrate to Storage1001B (not 100% sure on this)<o:p></o:p></p><p class=MsoNormal>- Storage1001A boots back up and tries to take back resources<o:p></o:p></p><p class=MsoNormal>- Storage1001B hangs after stoping LUN before stoppin Target<o:p></o:p></p><p class=MsoNormal>- Manually cycle power on Storage1001B<o:p></o:p></p><p class=MsoNormal>- Resources remain "stuck" on Storage1001B until Storage1001B is back online (stonith currently disabled for Storage1001B, but sometimes it remains "stuck" when enabled).<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal>*** Normal Status<o:p></o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal># crm status<o:p></o:p></p><p class=MsoNormal>============<o:p></o:p></p><p class=MsoNormal>Last updated: Sat Dec 14 14:58:04 2013<o:p></o:p></p><p class=MsoNormal>Last change: Sat Dec 14 14:45:43 2013 via crm_resource on Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Stack: openais<o:p></o:p></p><p class=MsoNormal>Current DC: Storage1001A.xdomain.com - partition with quorum<o:p></o:p></p><p class=MsoNormal>Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff<o:p></o:p></p><p class=MsoNormal>2 Nodes configured, 2 expected votes<o:p></o:p></p><p class=MsoNormal>22 Resources configured.<o:p></o:p></p><p class=MsoNormal>============<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Online: [ Storage1001B.xdomain.com Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>st_node_a (stonith:external/synaccess): Started Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Resource Group: rg_data01<o:p></o:p></p><p class=MsoNormal> p_lvm01 (ocf::heartbeat:LVM): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_ip1 (ocf::heartbeat:IPaddr): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_ip11 (ocf::heartbeat:IPaddr): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_target_data01 (ocf::heartbeat:iSCSITarget): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_lu_data01_vol01 (ocf::heartbeat:iSCSILogicalUnit): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_email_admin1 (ocf::heartbeat:MailTo): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Master/Slave Set: ms_drbd1 [p_drbd1]<o:p></o:p></p><p class=MsoNormal> Masters: [ Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal> Slaves: [ Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal>Master/Slave Set: ms_drbd2 [p_drbd2]<o:p></o:p></p><p class=MsoNormal> Masters: [ Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal> Slaves: [ Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal>Clone Set: c_lsb_target [p_target]<o:p></o:p></p><p class=MsoNormal> Started: [ Storage1001A.xdomain.com Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal>Clone Set: c_ping [p_ping]<o:p></o:p></p><p class=MsoNormal> Started: [ Storage1001B.xdomain.com Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal>*** Status after migrate<o:p></o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal># crm resource migrate rg_data01 Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal># crm status<o:p></o:p></p><p class=MsoNormal>============<o:p></o:p></p><p class=MsoNormal>Last updated: Sat Dec 14 16:40:55 2013<o:p></o:p></p><p class=MsoNormal>Last change: Sat Dec 14 16:40:48 2013 via crm_resource on Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Stack: openais<o:p></o:p></p><p class=MsoNormal>Current DC: Storage1001A.xdomain.com - partition with quorum<o:p></o:p></p><p class=MsoNormal>Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff<o:p></o:p></p><p class=MsoNormal>2 Nodes configured, 2 expected votes<o:p></o:p></p><p class=MsoNormal>22 Resources configured.<o:p></o:p></p><p class=MsoNormal>============<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Online: [ Storage1001B.xdomain.com Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>st_node_a (stonith:external/synaccess): Started Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Resource Group: rg_data01<o:p></o:p></p><p class=MsoNormal> p_lvm01 (ocf::heartbeat:LVM): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_ip1 (ocf::heartbeat:IPaddr): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_ip11 (ocf::heartbeat:IPaddr): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_target_data01 (ocf::heartbeat:iSCSITarget): Started Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_lu_data01_vol01 (ocf::heartbeat:iSCSILogicalUnit): Stopped<o:p></o:p></p><p class=MsoNormal> p_email_admin1 (ocf::heartbeat:MailTo): Stopped<o:p></o:p></p><p class=MsoNormal>Master/Slave Set: ms_drbd1 [p_drbd1]<o:p></o:p></p><p class=MsoNormal> Masters: [ Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal> Slaves: [ Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal>Master/Slave Set: ms_drbd2 [p_drbd2]<o:p></o:p></p><p class=MsoNormal> Masters: [ Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal> Slaves: [ Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal>Clone Set: c_lsb_target [p_target]<o:p></o:p></p><p class=MsoNormal> Started: [ Storage1001A.xdomain.com Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal>Clone Set: c_ping [p_ping]<o:p></o:p></p><p class=MsoNormal> Started: [ Storage1001B.xdomain.com Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal>*** Status after first node boots back up and tries to take back resource<o:p></o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal># crm status<o:p></o:p></p><p class=MsoNormal>============<o:p></o:p></p><p class=MsoNormal>Last updated: Tue Dec 17 14:06:15 2013<o:p></o:p></p><p class=MsoNormal>Last change: Tue Dec 17 14:05:44 2013 via cibadmin on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Stack: openais<o:p></o:p></p><p class=MsoNormal>Current DC: Storage1001B.xdomain.com - partition with quorum<o:p></o:p></p><p class=MsoNormal>Version: 1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff<o:p></o:p></p><p class=MsoNormal>2 Nodes configured, 2 expected votes<o:p></o:p></p><p class=MsoNormal>22 Resources configured.<o:p></o:p></p><p class=MsoNormal>============<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Node Storage1001B.xdomain.com: UNCLEAN (online)<o:p></o:p></p><p class=MsoNormal>Online: [ Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>st_node_a (stonith:external/synaccess): Started Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Resource Group: rg_data01<o:p></o:p></p><p class=MsoNormal> p_lvm01 (ocf::heartbeat:LVM): Started Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_ip1 (ocf::heartbeat:IPaddr): Started Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_ip11 (ocf::heartbeat:IPaddr): Started Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal> p_target_data01 (ocf::heartbeat:iSCSITarget): Started Storage1001B.xdomain.com FAILED<o:p></o:p></p><p class=MsoNormal> p_lu_data01_vol01 (ocf::heartbeat:iSCSILogicalUnit): Stopped<o:p></o:p></p><p class=MsoNormal> p_email_admin1 (ocf::heartbeat:MailTo): Stopped<o:p></o:p></p><p class=MsoNormal>Master/Slave Set: ms_drbd1 [p_drbd1]<o:p></o:p></p><p class=MsoNormal> Masters: [ Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal> Slaves: [ Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal>Master/Slave Set: ms_drbd2 [p_drbd2]<o:p></o:p></p><p class=MsoNormal> Masters: [ Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal> Slaves: [ Storage1001A.xdomain.com ]<o:p></o:p></p><p class=MsoNormal>Clone Set: c_lsb_target [p_target]<o:p></o:p></p><p class=MsoNormal> Started: [ Storage1001A.xdomain.com Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal>Clone Set: c_ping [p_ping]<o:p></o:p></p><p class=MsoNormal> Started: [ Storage1001A.xdomain.com Storage1001B.xdomain.com ]<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Failed actions:<o:p></o:p></p><p class=MsoNormal> p_target_data01_stop_0 (node=Storage1001B.xdomain.com, call=111, rc=-2, status=Timed Out): unknown exec error<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal>*** Configuration<o:p></o:p></p><p class=MsoNormal>*** Note: Some resources are stopped to try and get one<o:p></o:p></p><p class=MsoNormal>*** resource group working properly<o:p></o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal>node Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>node Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>primitive p_drbd1 ocf:linbit:drbd \<o:p></o:p></p><p class=MsoNormal> params drbd_resource="Data01" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="3" role="Master" timeout="9" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="4" role="Slave" timeout="12"<o:p></o:p></p><p class=MsoNormal>primitive p_drbd2 ocf:linbit:drbd \<o:p></o:p></p><p class=MsoNormal> params drbd_resource="Data02" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="3" role="Master" timeout="9" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="4" role="Slave" timeout="12"<o:p></o:p></p><p class=MsoNormal>primitive p_email_admin1 ocf:heartbeat:MailTo \<o:p></o:p></p><p class=MsoNormal> params email="<a href="mailto:admin@xdomain.com">admin@xdomain.com</a>" subject="Cluster Failover"<o:p></o:p></p><p class=MsoNormal>primitive p_email_admin2 ocf:heartbeat:MailTo \<o:p></o:p></p><p class=MsoNormal> params email="<a href="mailto:admin@xdomain.com">admin@xdomain.com</a>" subject="Cluster Failover"<o:p></o:p></p><p class=MsoNormal>primitive p_ip1 ocf:heartbeat:IPaddr \<o:p></o:p></p><p class=MsoNormal> params ip="10.11.2.13" nic="bond0" cidr_netmask="21" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="5s"<o:p></o:p></p><p class=MsoNormal>primitive p_ip11 ocf:heartbeat:IPaddr \<o:p></o:p></p><p class=MsoNormal> params ip="10.11.10.13" nic="bond2" cidr_netmask="21" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="5s"<o:p></o:p></p><p class=MsoNormal>primitive p_ip2 ocf:heartbeat:IPaddr \<o:p></o:p></p><p class=MsoNormal> params ip="10.11.2.14" nic="bond0" cidr_netmask="21" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="5s"<o:p></o:p></p><p class=MsoNormal>primitive p_ip22 ocf:heartbeat:IPaddr \<o:p></o:p></p><p class=MsoNormal> params ip="10.11.10.14" nic="bond2" cidr_netmask="21" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="5s"<o:p></o:p></p><p class=MsoNormal>primitive p_lu_data01_vol01 ocf:heartbeat:iSCSILogicalUnit \<o:p></o:p></p><p class=MsoNormal> params target_iqn="iqn.2013-10.com.xdomain.storage1001.data01" lun="1" path="/dev/vg_data01/lv_vol01" implementation="lio" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="10"<o:p></o:p></p><p class=MsoNormal>primitive p_lu_data02_vol01 ocf:heartbeat:iSCSILogicalUnit \<o:p></o:p></p><p class=MsoNormal> params target_iqn="iqn.2013-10.com.xdomain.storage1001.data02" lun="1" path="/dev/vg_data02/lv_vol01" implementation="lio" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="10"<o:p></o:p></p><p class=MsoNormal>primitive p_lvm01 ocf:heartbeat:LVM \<o:p></o:p></p><p class=MsoNormal> params volgrpname="vg_data01" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="4" timeout="8"<o:p></o:p></p><p class=MsoNormal>primitive p_lvm02 ocf:heartbeat:LVM \<o:p></o:p></p><p class=MsoNormal> params volgrpname="vg_data02" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="4" timeout="8"<o:p></o:p></p><p class=MsoNormal>primitive p_ping ocf:pacemaker:ping \<o:p></o:p></p><p class=MsoNormal> op monitor interval="5s" timeout="15s" \<o:p></o:p></p><p class=MsoNormal> params host_list="10.11.10.1" multiplier="200" name="p_ping"<o:p></o:p></p><p class=MsoNormal>primitive p_target lsb:target \<o:p></o:p></p><p class=MsoNormal> op monitor interval="30" timeout="30"<o:p></o:p></p><p class=MsoNormal>primitive p_target_data01 ocf:heartbeat:iSCSITarget \<o:p></o:p></p><p class=MsoNormal> params iqn="iqn.2013-10.com.xdomain.storage1001.data01" implementation="lio" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="10s" timeout="20s"<o:p></o:p></p><p class=MsoNormal>primitive p_target_data02 ocf:heartbeat:iSCSITarget \<o:p></o:p></p><p class=MsoNormal> params iqn="iqn.2013-10.com.xdomain.storage1001.data02" implementation="lio" \<o:p></o:p></p><p class=MsoNormal> op monitor interval="10s" timeout="20s"<o:p></o:p></p><p class=MsoNormal>primitive st_node_a stonith:external/synaccess \<o:p></o:p></p><p class=MsoNormal> params synaccessip="reboot11.xdomain.com" community="*******" port="Storage1001A" pcmk_host_list="Storage1001A.xdomain.com" \<o:p></o:p></p><p class=MsoNormal> meta target-role="Started"<o:p></o:p></p><p class=MsoNormal>primitive st_node_b stonith:external/synaccess \<o:p></o:p></p><p class=MsoNormal> params synaccessip="reboot10.xdomain.com" community="******" port="Storage1001B" pcmk_host_list="Storage1001B.xdomain.com" \<o:p></o:p></p><p class=MsoNormal> meta target-role="Stopped"<o:p></o:p></p><p class=MsoNormal>group rg_data01 p_lvm01 p_ip1 p_ip11 p_target_data01 p_lu_data01_vol01 p_email_admin1 \<o:p></o:p></p><p class=MsoNormal> meta target-role="Started"<o:p></o:p></p><p class=MsoNormal>group rg_data02 p_lvm02 p_ip2 p_ip22 p_target_data02 p_lu_data02_vol01 p_email_admin2 \<o:p></o:p></p><p class=MsoNormal> meta target-role="Stopped"<o:p></o:p></p><p class=MsoNormal>ms ms_drbd1 p_drbd1 \<o:p></o:p></p><p class=MsoNormal> meta notify="true" master-max="1" clone-max="2" clone-node-max="1" target-role="Started" \<o:p></o:p></p><p class=MsoNormal> meta resource-stickiness="101"<o:p></o:p></p><p class=MsoNormal>ms ms_drbd2 p_drbd2 \<o:p></o:p></p><p class=MsoNormal> meta notify="true" master-max="1" clone-max="2" clone-node-max="1" target-role="Started" \<o:p></o:p></p><p class=MsoNormal> meta resource-stickiness="101"<o:p></o:p></p><p class=MsoNormal>clone c_lsb_target p_target \<o:p></o:p></p><p class=MsoNormal> meta target-role="Started"<o:p></o:p></p><p class=MsoNormal>clone c_ping p_ping \<o:p></o:p></p><p class=MsoNormal> meta globally-unique="false" target-role="Started"<o:p></o:p></p><p class=MsoNormal>location data01_prefer_a ms_drbd1 \<o:p></o:p></p><p class=MsoNormal> rule $id="data01_prefer_a_rule" $role="Master" 100: #uname eq Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>location data02_prefer_b ms_drbd2 \<o:p></o:p></p><p class=MsoNormal> rule $id="data02_prefer_b_rule" $role="Master" 100: #uname eq Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>location st_node_a-loc st_node_a \<o:p></o:p></p><p class=MsoNormal> rule $id="st_node_a-loc-id" -inf: #uname eq Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>location st_node_b-loc st_node_b \<o:p></o:p></p><p class=MsoNormal> rule $id="st_node_b-loc-id" -inf: #uname eq Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>colocation c_drbd1 inf: rg_data01 ms_drbd1:Master<o:p></o:p></p><p class=MsoNormal>colocation c_drbd2 inf: rg_data02 ms_drbd2:Master<o:p></o:p></p><p class=MsoNormal>order o_data01_start inf: ms_drbd1:promote rg_data01:start<o:p></o:p></p><p class=MsoNormal>order o_data01_stop inf: rg_data01:stop ms_drbd1:demote<o:p></o:p></p><p class=MsoNormal>order o_data02_start inf: ms_drbd2:promote rg_data02:start<o:p></o:p></p><p class=MsoNormal>order o_data02_stop inf: rg_data02:stop ms_drbd2:demote<o:p></o:p></p><p class=MsoNormal>order o_target_before_data01 inf: c_lsb_target:start rg_data01<o:p></o:p></p><p class=MsoNormal>order o_target_before_data02 inf: c_lsb_target:start rg_data02<o:p></o:p></p><p class=MsoNormal>property $id="cib-bootstrap-options" \<o:p></o:p></p><p class=MsoNormal> dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \<o:p></o:p></p><p class=MsoNormal> cluster-infrastructure="openais" \<o:p></o:p></p><p class=MsoNormal> expected-quorum-votes="2" \<o:p></o:p></p><p class=MsoNormal> stonith-enabled="true" \<o:p></o:p></p><p class=MsoNormal> no-quorum-policy="ignore" \<o:p></o:p></p><p class=MsoNormal> default-resource-stickiness="1" \<o:p></o:p></p><p class=MsoNormal> last-lrm-refresh="1387064240"<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal>*** Syslog Storage1001A<o:p></o:p></p><p class=MsoNormal>***********************************************************<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib_process_request: Operation complete: op cib_delete for section constraints (origin=local/crm_resource/3, version=0.260.23): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <cib admin_epoch="0" epoch="260" num_updates="23" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <configuration ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <constraints ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <rsc_location id="cli-prefer-rg_data01" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <rule id="cli-prefer-rule-rg_data01" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - <expression value="Storage1001A.xdomain.com" id="cli-prefer-expr-rg_data01" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - </rule><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - </rsc_location><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - </constraints><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - </configuration><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: - </cib><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: + <cib epoch="261" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="Storage1001A.xdomain.com" update-client="crm_resource" cib-last-written="Tue Dec 17 13:55:53 2013" have-quorum="1" dc-uuid="Storage1001A.xdomain.com" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001A cib: [3057]: info: cib:diff: + <configuration ><o:p></o:p></p><p class=MsoNormal>Dec 17 14Dec 17 14:05:31 Storage1001A kernel: imklog 5.8.11, log source = /proc/kmsg started.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A rsyslogd: [origin software="rsyslogd" swVersion="5.8.11" x-pid="2318" x-info="<a href="http://www.rsyslog.com">http://www.rsyslog.com</a>"] start<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 0.000000] Initializing cgroup subsys cpuset<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 0.000000] Initializing cgroup subsys cpu<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 0.000000] Linux version 3.2.0-4-amd64 (<a href="mailto:debian-kernel@lists.debian.org">debian-kernel@lists.debian.org</a>) (gcc version 4.6.3 (Debian 4.6.3-14) ) #1 SMP Debian 3.2.51-1<o:p></o:p></p><p class=MsoNormal>*** Boot Messages Removed ***<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 10.404711] fbcon: radeondrmfb (fb0) is primary device<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 10.566191] Console: switching to colour frame buffer device 128x48<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 10.573706] fb0: radeondrmfb frame buffer device<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 10.573707] drm: registered panic notifier<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 10.573713] [drm] Initialized radeon 2.16.0 20080528 for 0000:0c:01.0 on minor 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 11.841717] EXT4-fs (dm-0): re-mounted. Opts: (null)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 11.980402] EXT4-fs (dm-0): re-mounted. Opts: errors=remount-ro<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 12.023447] loop: module loaded<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 12.324689] Adding 667644k swap on /dev/mapper/Storage1001A-swap_1. Priority:-1 extents:1 across:667644k<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.407466] bonding: Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.407472] bonding: Warning: miimon must be specified, otherwise bonding will not detect link failure and link speed which are essential for TLB/ALB load balancing<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.407475] bonding: Forcing miimon to 100msec<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.407477] bonding: In ALB mode you might experience client disconnections upon reconnection of a link if the bonding module updelay parameter (0 msec) is incompatible with the forwarding delay time of the switch<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.407480] bonding: MII link monitoring set to 100 ms<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.410588] bonding: bond0: Setting MII monitoring interval to 100.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.411752] bonding: bond0: setting mode to balance-alb (6).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.413901] bonding: bond0: Adding slave eth0.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.560030] e1000e 0000:05:00.0: irq 67 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.617297] e1000e 0000:05:00.0: irq 67 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.620289] bonding: bond0: enslaving eth0 as an active interface with a down link.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.623045] bonding: bond0: Adding slave eth4.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.649821] e1000e 0000:07:00.0: irq 71 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.705298] e1000e 0000:07:00.0: irq 71 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.708389] bonding: bond0: enslaving eth4 as an active interface with a down link.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.712623] e1000e 0000:05:00.0: eth0: changing MTU from 1500 to 8192<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.903885] e1000e 0000:07:00.0: eth4: changing MTU from 1500 to 8192<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.980307] ADDRCONF(NETDEV_UP): bond0: link is not ready<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.997627] bonding: bond1 is being created...<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 14.998786] bonding: bond1: Setting MII monitoring interval to 100.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.001494] bonding: bond1: setting mode to balance-rr (0).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.003576] bonding: bond1: Adding slave eth1.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.151980] e1000e 0000:05:00.1: irq 68 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.209292] e1000e 0000:05:00.1: irq 68 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.211322] bonding: bond1: enslaving eth1 as an active interface with a down link.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.214034] bonding: bond1: Adding slave eth5.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.249840] e1000e 0000:07:00.1: irq 72 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.305290] e1000e 0000:07:00.1: irq 72 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.307254] bonding: bond1: enslaving eth5 as an active interface with a down link.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.311595] e1000e 0000:05:00.1: eth1: changing MTU from 1500 to 9000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.499927] e1000e 0000:07:00.1: eth5: changing MTU from 1500 to 9000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.576560] ADDRCONF(NETDEV_UP): bond1: link is not ready<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.592865] bonding: bond2 is being created...<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.593997] bonding: bond2: Setting MII monitoring interval to 100.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.595266] bonding: bond2: setting mode to balance-alb (6).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.597568] bonding: bond2: Adding slave eth2.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.743998] e1000e 0000:06:00.0: irq 69 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.801292] e1000e 0000:06:00.0: irq 69 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.804286] bonding: bond2: enslaving eth2 as an active interface with a down link.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.807068] bonding: bond2: Adding slave eth3.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 15.953819] e1000e 0000:06:00.1: irq 70 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.009294] e1000e 0000:06:00.1: irq 70 for MSI/MSI-X<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.012279] bonding: bond2: enslaving eth3 as an active interface with a down link.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.016311] e1000e 0000:06:00.0: eth2: changing MTU from 1500 to 8192<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.207847] e1000e 0000:06:00.1: eth3: changing MTU from 1500 to 8192<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.396867] ADDRCONF(NETDEV_UP): bond2: link is not ready<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.762069] RPC: Registered named UNIX socket transport module.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.762073] RPC: Registered udp transport module.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.762075] RPC: Registered tcp transport module.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.762076] RPC: Registered tcp NFSv4.1 backchannel transport module.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.782584] FS-Cache: Loaded<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.827809] FS-Cache: Netfs 'nfs' registered for caching<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 16.851676] Installing knfsd (copyright (C) 1996 <a href="mailto:okir@monad.swb.de">okir@monad.swb.de</a>).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 17.244926] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 17.276019] bonding: bond1: link status definitely up for interface eth1, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 17.277008] ADDRCONF(NETDEV_CHANGE): bond1: link becomes ready<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 17.288927] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 17.376024] bonding: bond0: link status definitely up for interface eth0, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 17.376029] bonding: bond0: making interface eth0 the new active one.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 17.378190] bonding: bond0: first active interface up!<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 17.379114] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A acpid: starting up with netlink and the input layer<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A acpid: 1 rule loaded<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A acpid: waiting for events: event logging is off<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001A kernel: [ 18.003326] e1000e: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.076015] bonding: bond0: link status definitely up for interface eth4, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A logd: [2538]: info: logd started with /etc/logd.cf.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A logd: [2538]: WARN: Core dumps could be lost if multiple dumps occur.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A logd: [2538]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A logd: [2538]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.388930] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.392017] bonding: bond2: link status definitely up for interface eth2, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.392036] bonding: bond2: making interface eth2 the new active one.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.394197] bonding: bond2: first active interface up!<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.395186] ADDRCONF(NETDEV_CHANGE): bond2: link becomes ready<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A openhpid: ERROR: (init.c, 76, OpenHPI is not configured. See openhpi.conf file.)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A openhpid: ERROR: (openhpid.cpp, 270, There was an error initializing OpenHPI)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.491379] e1000e: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.568922] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.576018] bonding: bond1: link status definitely up for interface eth5, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001A kernel: [ 18.592012] bonding: bond2: link status definitely up for interface eth3, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A /usr/sbin/cron[2618]: (CRON) INFO (pidfile fd = 3)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A /usr/sbin/cron[2626]: (CRON) STARTUP (fork ok)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2624]: ntpd <a href="mailto:4.2.6p5@1.2349-o">4.2.6p5@1.2349-o</a> Sat May 12 09:54:55 UTC 2012 (1)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: proto: precision = 0.117 usec<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen and drop on 1 v6wildcard :: UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 2 lo 127.0.0.1 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 3 bond0 10.11.2.11 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 4 bond1 10.100.100.11 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 5 bond2 10.11.10.11 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 6 bond1 fe80::226:55ff:fed7:3d90 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: Listen normally on 7 lo ::1 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: peers refreshed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:33 Storage1001A ntpd[2644]: Listening on routing socket on fd #24 for interface updates<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:36 Storage1001A /usr/sbin/cron[2626]: (CRON) INFO (Running @reboot jobs)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:38 Storage1001A ntpd[2644]: Listen normally on 8 bond2 fe80::226:55ff:fed7:3d93 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:38 Storage1001A ntpd[2644]: Listen normally on 9 bond0 fe80::226:55ff:fed7:3d91 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:38 Storage1001A ntpd[2644]: peers refreshed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.264486] Unable to load target_core_stgt<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.562625] drbd: initialized. Version: 8.3.11 (api:88/proto:86-96)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.562628] drbd: srcversion: F937DCB2E5D83C6CCE4A6C9<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.562630] drbd: registered as block device major 147<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.562632] drbd: minor_table @ 0xffff8802249fbdc0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.584528] block drbd1: Starting worker thread (from drbdsetup [2974])<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.584630] block drbd1: disk( Diskless -> Attaching )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.586424] block drbd1: Found 4 transactions (38 active extents) in activity log.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.586427] block drbd1: Method to ensure write ordering: flush<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.586429] block drbd1: max BIO size = 131072<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.586433] block drbd1: drbd_bm_resize called with capacity == 3906129416<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.613463] block drbd1: resync bitmap: bits=488266177 words=7629160 pages=14901<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:39 Storage1001A kernel: [ 25.613469] block drbd1: size = 1863 GB (1953064708 KB)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:40 Storage1001A kernel: [ 26.644370] block drbd1: bitmap READ of 14901 pages took 258 jiffies<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:40 Storage1001A kernel: [ 26.692206] block drbd1: recounting of set bits took additional 12 jiffies<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:40 Storage1001A kernel: [ 26.692211] block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:40 Storage1001A kernel: [ 26.692230] block drbd1: Marked additional 148 MB as out-of-sync based on AL.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:40 Storage1001A kernel: [ 26.692502] block drbd1: bitmap WRITE of 0 pages took 0 jiffies<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:40 Storage1001A kernel: [ 26.693035] block drbd1: 148 MB (37888 bits) marked out-of-sync by on disk bit-map.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:40 Storage1001A kernel: [ 26.693047] block drbd1: disk( Attaching -> Consistent )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:40 Storage1001A kernel: [ 26.693051] block drbd1: attached to UUIDs 82E61159C15A62E0:0000000000000000:711FF2F50EF3A977:711EF2F50EF3A977<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:41 Storage1001A kernel: [ 27.472006] bond1: no IPv6 routers present<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:41 Storage1001A kernel: [ 27.704347] block drbd2: Starting worker thread (from drbdsetup [2990])<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:41 Storage1001A kernel: [ 27.704424] block drbd2: disk( Diskless -> Attaching )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:41 Storage1001A kernel: [ 27.705553] block drbd2: Found 3 transactions (3 active extents) in activity log.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:41 Storage1001A kernel: [ 27.705556] block drbd2: Method to ensure write ordering: flush<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:41 Storage1001A kernel: [ 27.705559] block drbd2: max BIO size = 131072<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:41 Storage1001A kernel: [ 27.705564] block drbd2: drbd_bm_resize called with capacity == 3906084368<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:41 Storage1001A kernel: [ 27.732182] block drbd2: resync bitmap: bits=488260546 words=7629072 pages=14901<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:41 Storage1001A kernel: [ 27.732188] block drbd2: size = 1863 GB (1953042184 KB)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.224005] bond0: no IPv6 routers present<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.480005] bond2: no IPv6 routers present<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.853639] block drbd2: bitmap READ of 14901 pages took 280 jiffies<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.901429] block drbd2: recounting of set bits took additional 12 jiffies<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.901433] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.901444] block drbd2: disk( Attaching -> Consistent )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.901448] block drbd2: attached to UUIDs 76A87E2130121406:0000000000000000:B007DA0B699331F8:B006DA0B699331F9<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.928435] block drbd1: conn( StandAlone -> Unconnected )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.928456] block drbd1: Starting receiver thread (from drbd1_worker [2975])<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.928514] block drbd1: receiver (re)started<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.928526] block drbd1: conn( Unconnected -> WFConnection )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.929753] block drbd2: conn( StandAlone -> Unconnected )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.929762] block drbd2: Starting receiver thread (from drbd2_worker [2991])<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.929813] block drbd2: receiver (re)started<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:42 Storage1001A kernel: [ 28.929820] block drbd2: conn( Unconnected -> WFConnection )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427516] block drbd1: Handshake successful: Agreed network protocol version 96<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427526] block drbd1: conn( WFConnection -> WFReportParams )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427563] block drbd1: Starting asender thread (from drbd1_receiver [3003])<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427567] block drbd2: Handshake successful: Agreed network protocol version 96<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427574] block drbd2: conn( WFConnection -> WFReportParams )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427611] block drbd2: Starting asender thread (from drbd2_receiver [3007])<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427718] block drbd2: data-integrity-alg: <not-used><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427729] block drbd1: data-integrity-alg: <not-used><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427764] block drbd1: drbd_sync_handshake:<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427768] block drbd1: self 82E61159C15A62E0:0000000000000000:711FF2F50EF3A977:711EF2F50EF3A977 bits:37888 flags:0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427771] block drbd1: peer E48A0D04194BDA73:82E61159C15A62E0:711FF2F50EF3A976:711EF2F50EF3A977 bits:0 flags:0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427774] block drbd1: uuid_compare()=-1 by rule 50<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427778] block drbd2: drbd_sync_handshake:<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427782] block drbd2: self 76A87E2130121406:0000000000000000:B007DA0B699331F8:B006DA0B699331F9 bits:0 flags:0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427786] block drbd2: peer 2A0715EFC37BB5BB:76A87E2130121407:B007DA0B699331F9:B006DA0B699331F9 bits:0 flags:0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427792] block drbd1: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent -> Outdated ) pdsk( DUnknown -> UpToDate )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427800] block drbd2: uuid_compare()=-1 by rule 50<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 29.427807] block drbd2: peer( Unknown -> Primary ) conn( WFReportParams -> WFBitMapT ) disk( Consistent -> Outdated ) pdsk( DUnknown -> UpToDate )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A kernel: [ 30.168469] block drbd1: conn( WFBitMapT -> WFSyncUUID )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A corosync[3057]: [MAIN ] Corosync Cluster Engine ('1.4.2'): started and ready to provide service.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A corosync[3057]: [MAIN ] Corosync built-in features: nss<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A corosync[3057]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A corosync[3057]: [TOTEM ] Initializing transport (UDP/IP Unicast).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001A corosync[3057]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.271375] block drbd1: updated sync uuid 82E71159C15A62E0:0000000000000000:711FF2F50EF3A977:711EF2F50EF3A977<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [TOTEM ] The network interface [10.100.100.11] is now up.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.294016] block drbd1: helper command: /sbin/drbdadm before-resync-target minor-1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.296565] block drbd1: helper command: /sbin/drbdadm before-resync-target minor-1 exit code 0 (0x0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.296577] block drbd1: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.296587] block drbd1: Began resync as SyncTarget (will sync 151552 KB [37888 bits set]).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: process_ais_conf: Reading configure<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: config_find_init: Local handle: 2730409743423111171 for logging<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: config_find_next: Processing additional logging options...<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: get_config_opt: Found 'off' for option: debug<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: get_config_opt: Found 'no' for option: to_logfile<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: get_config_opt: Found 'yes' for option: to_syslog<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: get_config_opt: Found 'daemon' for option: syslog_facility<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: config_find_init: Local handle: 5880381755227111428 for quorum<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: config_find_next: Processing additional quorum options...<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: get_config_opt: Found 'corosync_votequorum' for option: provider<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: config_find_init: Local handle: 4835695805891346437 for service<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: config_find_next: Processing additional service options...<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: get_config_opt: Found '0' for option: ver<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_startup: CRM: Initialized<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] Logging: Initialized pcmk_startup<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_startup: Service: 9<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_startup: Local hostname: Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_update_nodeid: Local node id: 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: Creating entry for node 191128586 born on 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: 0x1997aa0 Node 191128586 now known as Storage1001A.xdomain.com (was: (null))<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: Node Storage1001A.xdomain.com now has 1 quorum votes (was 0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: Node 191128586/Storage1001A.xdomain.com is now: member<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child: Forked child 3095 for process cib<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child: Forked child 3096 for process stonith-ng<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child: Forked child 3097 for process lrmd<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child: Forked child 3098 for process attrd<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child: Forked child 3099 for process pengine<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: spawn_child: Forked child 3100 for process crmd<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.1.7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine loaded: corosync extended virtual synchrony service<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine loaded: corosync configuration service<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine loaded: corosync cluster config database access v1.01<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine loaded: corosync profile loading service<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Using quorum provider corosync_votequorum<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [VOTEQ ] Reading configuration<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[0]:<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine loaded: corosync votes quorum service v0.91<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 1888: memb=0, new=0, lost=0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 1888: memb=1, new=1, lost=0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_peer_update: NEW: Storage1001A.xdomain.com 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_peer_update: MEMB: Storage1001A.xdomain.com 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: Node Storage1001A.xdomain.com now has process list: 00000000000000000000000000111312 (1118994)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[1]: 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[1]: 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [CPG ] chosen downlist: sender r(0) ip(10.100.100.11) ; members(old:0 left:0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [MAIN ] Completed service synchronization, ready to provide service.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.403515] block drbd2: conn( WFBitMapT -> WFSyncUUID )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 1892: memb=1, new=0, lost=0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_peer_update: memb: Storage1001A.xdomain.com 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 1892: memb=2, new=1, lost=0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: Creating entry for node 207905802 born on 1892<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: Node 207905802/unknown is now: member<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_peer_update: MEMB: Storage1001A.xdomain.com 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: send_member_notification: Sending membership update 1892 to 0 children<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: 0x1997aa0 Node 191128586 ((null)) born on: 1892<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [VOTEQ ] quorum regained, resuming activity<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] This node is within the primary component and will provide service.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[2]: 191128586 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: 0x199f440 Node 207905802 (Storage1001B.xdomain.com) born on: 1884<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: 0x199f440 Node 207905802 now known as Storage1001B.xdomain.com (was: (null))<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: Node Storage1001B.xdomain.com now has process list: 00000000000000000000000000111312 (1118994)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: update_member: Node Storage1001B.xdomain.com now has 1 quorum votes (was 0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: send_member_notification: Sending membership update 1892 to 0 children<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [QUORUM] Members[2]: 191128586 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.461235] block drbd2: updated sync uuid 76A97E2130121406:0000000000000000:B007DA0B699331F8:B006DA0B699331F9<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [CPG ] chosen downlist: sender r(0) ip(10.100.100.11) ; members(old:1 left:0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [MAIN ] Completed service synchronization, ready to provide service.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.crmd failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.551166] block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.554239] block drbd2: helper command: /sbin/drbdadm before-resync-target minor-2 exit code 0 (0x0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.554250] block drbd2: conn( WFSyncUUID -> SyncTarget ) disk( Outdated -> Inconsistent )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.554259] block drbd2: Began resync as SyncTarget (will sync 0 KB [0 bits set]).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A lrmd: [3097]: info: enabling coredumps<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A lrmd: [3097]: WARN: Core dumps could be lost if multiple dumps occur.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A lrmd: [3097]: WARN: Consider setting non-default value in /proc/sys/kernel/core_pattern (or equivalent) for maximum supportability<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A lrmd: [3097]: WARN: Consider setting /proc/sys/kernel/core_uses_pid (or equivalent) to 1 for maximum supportability<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A lrmd: [3097]: info: Started.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.557373] block drbd2: Resync done (total 1 sec; paused 0 sec; 0 K/sec)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.557379] block drbd2: updated UUIDs 2A0715EFC37BB5BA:0000000000000000:76A97E2130121406:76A87E2130121407<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.557384] block drbd2: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: Invoked: /usr/lib/pacemaker/stonithd<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/root<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: get_cluster_type: Cluster type is: 'openais'<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: init_ais_connection_classic: Creating connection to our Corosync plugin<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: init_ais_connection_classic: AIS connection established<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc: Recorded connection 0x19a3a60 for stonith-ng/3096<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: get_ais_nodeid: Server details: id=191128586 uname=Storage1001A.xdomain.com cname=pcmk<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: crm_new_peer: Node Storage1001A.xdomain.com now has id: 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A stonith-ng: [3096]: info: crm_new_peer: Node 191128586 is now known as Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A pengine: [3099]: info: Invoked: /usr/lib/pacemaker/pengine<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A attrd: [3098]: info: Invoked: /usr/lib/pacemaker/attrd<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A attrd: [3098]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc: Recorded connection 0x19a7dc0 for attrd/3098<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A attrd: [3098]: notice: main: Starting mainloop...<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: validate_with_relaxng: Creating RNG parser context<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: startCib: CIB Initialization completed successfully<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: get_cluster_type: Cluster type is: 'openais'<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: init_ais_connection_classic: Creating connection to our Corosync plugin<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: init_ais_connection_classic: AIS connection established<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc: Recorded connection 0x19ac120 for cib/3095<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc: Sending membership update 1892 to cib<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: get_ais_nodeid: Server details: id=191128586 uname=Storage1001A.xdomain.com cname=pcmk<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_new_peer: Node Storage1001A.xdomain.com now has id: 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_new_peer: Node 191128586 is now known as Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: cib_init: Starting cib mainloop<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: notice: ais_dispatch_message: Membership 1892: quorum acquired<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_update_peer: Node Storage1001A.xdomain.com: id=191128586 state=member (new) addr=r(0) ip(10.100.100.11) (new) votes=1 (new) born=1892 seen=1892 proc=00000000000000000000000000111312 (new)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_new_peer: Node Storage1001B.xdomain.com now has id: 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_new_peer: Node 207905802 is now known as Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: crm_update_peer: Node Storage1001B.xdomain.com: id=207905802 state=member (new) addr=r(0) ip(10.100.100.12) votes=1 born=1884 seen=1892 proc=00000000000000000000000000111312<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: Invoked: /usr/lib/pacemaker/crmd<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: main: CRM Git Version: ee0730e13d124c3d58f00016c3376a1de5323cff<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: do_cib_control: CIB connection established<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: get_cluster_type: Cluster type is: 'openais'<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: init_ais_connection_classic: Creating connection to our Corosync plugin<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: init_ais_connection_classic: AIS connection established<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc: Recorded connection 0x19b06c0 for crmd/3100<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A corosync[3057]: [pcmk ] info: pcmk_ipc: Sending membership update 1892 to crmd<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: get_ais_nodeid: Server details: id=191128586 uname=Storage1001A.xdomain.com cname=pcmk<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_new_peer: Node Storage1001A.xdomain.com now has id: 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_new_peer: Node 191128586 is now known as Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: ais_status_callback: status: Storage1001A.xdomain.com is now unknown<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: do_ha_control: Connected to the cluster<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: do_started: Delaying start, no membership data (0000000000100000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: ais_dispatch_message: Membership 1892: quorum acquired<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: crmd_peer_update: Status update: Client Storage1001A.xdomain.com/crmd now has status [online] (DC=<null>)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: ais_status_callback: status: Storage1001A.xdomain.com is now member (was unknown)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_update_peer: Node Storage1001A.xdomain.com: id=191128586 state=member (new) addr=r(0) ip(10.100.100.11) (new) votes=1 (new) born=1892 seen=1892 proc=00000000000000000000000000111312 (new)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_new_peer: Node Storage1001B.xdomain.com now has id: 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_new_peer: Node 207905802 is now known as Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: ais_status_callback: status: Storage1001B.xdomain.com is now unknown<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: ais_status_callback: status: Storage1001B.xdomain.com is now member (was unknown)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: crm_update_peer: Node Storage1001B.xdomain.com: id=207905802 state=member (new) addr=r(0) ip(10.100.100.12) votes=1 born=1884 seen=1892 proc=00000000000000000000000000111312<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: info: do_started: Delaying start, Config not read (0000000000000040)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: do_started: The local CRM is operational<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crmd: [3100]: notice: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.776140] block drbd2: helper command: /sbin/drbdadm after-resync-target minor-2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A crm-unfence-peer.sh[3113]: invoked for Data02<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: cib_process_diff: Diff 0.262.25 -> 0.263.1 not applied to 0.261.0: current "epoch" is less than required<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: cib_server_process_diff: Requesting re-sync from peer<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 30.860423] block drbd2: helper command: /sbin/drbdadm after-resync-target minor-2 exit code 0 (0x0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A cib: [3095]: info: cib_replace_notify: Replaced: -1.-1.-1 -> 0.263.1 from Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001A kernel: [ 31.174689] block drbd2: bitmap WRITE of 14901 pages took 78 jiffies<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:45 Storage1001A kernel: [ 31.526981] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:45 Storage1001A stonith-ng: [3096]: notice: setup_cib: Watching for stonith topology changes<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:45 Storage1001A stonith-ng: [3096]: info: main: Starting stonith-ng mainloop<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:45 Storage1001A crmd: [3100]: info: ais_dispatch_message: Membership 1892: quorum retained<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A kernel: [ 32.599664] block drbd1: Resync done (total 2 sec; paused 0 sec; 75776 K/sec)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A kernel: [ 32.599671] block drbd1: updated UUIDs E48A0D04194BDA72:0000000000000000:82E71159C15A62E0:82E61159C15A62E0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A kernel: [ 32.599680] block drbd1: conn( SyncTarget -> Connected ) disk( Inconsistent -> UpToDate )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A kernel: [ 32.652349] block drbd1: helper command: /sbin/drbdadm after-resync-target minor-1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A crm-unfence-peer.sh[3132]: invoked for Data01<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A kernel: [ 32.695859] block drbd1: helper command: /sbin/drbdadm after-resync-target minor-1 exit code 0 (0x0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A crmd: [3100]: info: update_dc: Set DC to Storage1001B.xdomain.com (3.0.6)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A cib: [3095]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=Storage1001B.xdomain.com/crmd/130, version=0.264.1): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A crmd: [3100]: info: erase_status_tag: Deleting xpath: //node_state[@uname='Storage1001A.xdomain.com']/transient_attributes<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A crmd: [3100]: info: update_attrd: Connecting to attrd...<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A crmd: [3100]: notice: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A attrd: [3098]: notice: attrd_local_callback: Sending full refresh (origin=crmd)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A kernel: [ 32.787912] block drbd1: bitmap WRITE of 14889 pages took 23 jiffies<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001A kernel: [ 32.986722] block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: notice: lrmd_rsc_new(): No lrm_rprovider field in message<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:st_node_a probe[2] (pid 3148)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: notice: lrmd_rsc_new(): No lrm_rprovider field in message<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:st_node_b probe[3] (pid 3149)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:p_lvm01 probe[4] (pid 3150)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:p_ip1 probe[5] (pid 3151)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:p_ip11 probe[6] (pid 3152)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: rsc:p_target_data01 probe[7] (pid 3153)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: notice: lrmd_rsc_new(): No lrm_rprovider field in message<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A stonith-ng: [3096]: notice: stonith_device_action: Device st_node_b not found<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A stonith-ng: [3096]: info: stonith_command: Processed st_execute from lrmd: rc=-12<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A stonith-ng: [3096]: notice: stonith_device_action: Device st_node_a not found<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A stonith-ng: [3096]: info: stonith_command: Processed st_execute from lrmd: rc=-12<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[3] on st_node_b for client 3100: pid 3149 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[2] on st_node_a for client 3100: pid 3148 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation st_node_b_monitor_0 (call=3, rc=7, cib-update=7, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation st_node_a_monitor_0 (call=2, rc=7, cib-update=8, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[5] on p_ip1 for client 3100: pid 3151 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[6] on p_ip11 for client 3100: pid 3152 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[7] on p_target_data01 for client 3100: pid 3153 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_ip1_monitor_0 (call=5, rc=7, cib-update=9, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_ip11_monitor_0 (call=6, rc=7, cib-update=10, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_target_data01_monitor_0 (call=7, rc=0, cib-update=11, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A LVM[3150]: INFO: LVM Volume vg_data01 is offline<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A lrmd: [3097]: info: operation monitor[4] on p_lvm01 for client 3100: pid 3150 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_lvm01_monitor_0 (call=4, rc=7, cib-update=12, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: rsc:p_lu_data01_vol01 probe[8] (pid 3241)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: rsc:p_email_admin1 probe[9] (pid 3242)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: rsc:p_lvm02 probe[10] (pid 3243)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: rsc:p_ip2 probe[11] (pid 3244)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:48 Storage1001A lrmd: [3097]: info: operation monitor[11] on p_ip2 for client 3100: pid 3244 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:48 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_ip2_monitor_0 (call=11, rc=7, cib-update=13, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[9] on p_email_admin1 for client 3100: pid 3242 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_email_admin1_monitor_0 (call=9, rc=7, cib-update=14, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[8] on p_lu_data01_vol01 for client 3100: pid 3241 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: rsc:p_ip22 probe[12] (pid 3310)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: rsc:p_target_data02 probe[13] (pid 3311)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: rsc:p_lu_data02_vol01 probe[14] (pid 3312)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_lu_data01_vol01_monitor_0 (call=8, rc=7, cib-update=15, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A LVM[3243]: INFO: LVM Volume vg_data02 is offline<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[10] on p_lvm02 for client 3100: pid 3243 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: rsc:p_drbd1:0 probe[16] (pid 3332)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[12] on p_ip22 for client 3100: pid 3310 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_lvm02_monitor_0 (call=10, rc=7, cib-update=16, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_ip22_monitor_0 (call=12, rc=7, cib-update=17, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[13] on p_target_data02 for client 3100: pid 3311 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[14] on p_lu_data02_vol01 for client 3100: pid 3312 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_target_data02_monitor_0 (call=13, rc=7, cib-update=18, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_lu_data02_vol01_monitor_0 (call=14, rc=7, cib-update=19, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A attrd: [3098]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd1:0 (10000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A lrmd: [3097]: info: operation monitor[16] on p_drbd1:0 for client 3100: pid 3332 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_drbd1:0_monitor_0 (call=16, rc=0, cib-update=20, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A attrd: [3098]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd1:0 (10000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001A attrd: [3098]: notice: attrd_perform_update: Sent update 4: master-p_drbd1:0=10000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_email_admin2 probe[15] (pid 3396)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: operation monitor[15] on p_email_admin2 for client 3100: pid 3396 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_email_admin2_monitor_0 (call=15, rc=7, cib-update=21, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_drbd2:0 probe[17] (pid 3400)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_target:0 probe[18] (pid 3401)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_ping:0 probe[19] (pid 3402)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: operation monitor[19] on p_ping:0 for client 3100: pid 3402 exited with return code 7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_ping:0_monitor_0 (call=19, rc=7, cib-update=22, confirmed=true) not running<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A attrd: [3098]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd2:0 (10000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A attrd: [3098]: notice: attrd_perform_update: Sent update 7: master-p_drbd2:0=10000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: operation monitor[17] on p_drbd2:0 for client 3100: pid 3400 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_drbd2:0_monitor_0 (call=17, rc=0, cib-update=23, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: operation monitor[18] on p_target:0 for client 3100: pid 3401 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A crmd: [3100]: info: process_lrm_event: LRM operation p_target:0_monitor_0 (call=18, rc=0, cib-update=24, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A attrd: [3098]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A attrd: [3098]: notice: attrd_perform_update: Sent update 10: probe_complete=true<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_drbd1:0 monitor[20] (pid 3460)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_drbd2:0 monitor[21] (pid 3461)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_target:0 monitor[22] (pid 3462)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001A lrmd: [3097]: info: rsc:p_ping:0 start[23] (pid 3463)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>************************************************************<o:p></o:p></p><p class=MsoNormal>*** Syslog Storage1001B<o:p></o:p></p><p class=MsoNormal>***********************************************************<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[85] (pid 13822)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001B lrmd: [3070]: info: operation notify[85] on p_drbd1:1 for client 3073: pid 13822 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:36 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_notify_0 (call=85, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[86] (pid 14131)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: crm_new_peer: Node Storage1001A.xdomain.com now has id: 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: crm_new_peer: Node 191128586 is now known as Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: can_fence_host_with_device: st_node_a can fence Storage1001A.xdomain.com: static-list<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[87] (pid 14132)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: can_fence_host_with_device: st_node_a can fence Storage1001A.xdomain.com: static-list<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: stonith_fence: Found 1 matching devices for 'Storage1001A.xdomain.com'<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B stonith-ng: [3069]: info: stonith_command: Processed st_fence from Storage1001A.xdomain.com: rc=-1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B lrmd: [3070]: info: operation notify[86] on p_drbd1:1 for client 3073: pid 14131 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_notify_0 (call=86, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B lrmd: [3070]: info: operation notify[87] on p_drbd2:1 for client 3073: pid 14132 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:56 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd2:1_notify_0 (call=87, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:57 Storage1001B kernel: [ 643.872134] e1000e: eth1 NIC Link is Down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:57 Storage1001B kernel: [ 643.872233] e1000e: eth5 NIC Link is Down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:57 Storage1001B kernel: [ 643.936009] bonding: bond1: link status definitely down for interface eth1, disabling it<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:57 Storage1001B kernel: [ 643.936015] bonding: bond1: now running without any active interface !<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:57 Storage1001B kernel: [ 643.936019] bonding: bond1: link status definitely down for interface eth5, disabling it<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:58 Storage1001B cib: [3068]: info: cib_stats: Processed 471 operations (84.00us average, 0% utilization) in the last 10min<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.488010] block drbd2: PingAck did not arrive in time.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.488106] block drbd2: peer( Secondary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.488141] block drbd2: new current UUID 2A0715EFC37BB5BB:76A87E2130121407:B007DA0B699331F9:B006DA0B699331F9<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.512675] block drbd2: asender terminated<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.512680] block drbd2: Terminating drbd2_asender<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.521157] block drbd2: Connection closed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.521203] block drbd2: conn( NetworkFailure -> Unconnected )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.521209] block drbd2: helper command: /sbin/drbdadm fence-peer minor-2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.521216] block drbd2: receiver terminated<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.521226] block drbd2: Restarting drbd2_receiver<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.521236] block drbd2: receiver (re)started<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.521241] block drbd2: conn( Unconnected -> WFConnection )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B crm-fence-peer.sh[14270]: invoked for Data02<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B crm-fence-peer.sh[14270]: INFO suitable constraint already placed: 'drbd-fence-by-handler-Data02-ms_drbd2'<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.555063] block drbd2: helper command: /sbin/drbdadm fence-peer minor-2 exit code 4 (0x400)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.555068] block drbd2: fence-peer helper returned 4 (peer was fenced)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:03:59 Storage1001B kernel: [ 645.555076] block drbd2: pdsk( DUnknown -> Outdated )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="262" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="Storage1001A.xdomain.com" update-client="crm_resource" cib-last-written="Tue Dec 17 14:03:36 2013" have-quorum="0" dc-uuid="Storage1001B.xdomain.com" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:00 Storage1001B corosync[3033]: [TOTEM ] A processor failed, forming new configuration.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.400010] block drbd1: PingAck did not arrive in time.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.400111] block drbd1: peer( Primary -> Unknown ) conn( Connected -> NetworkFailure ) pdsk( UpToDate -> DUnknown )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.416165] block drbd1: asender terminated<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.416170] block drbd1: Terminating drbd1_asender<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.416223] block drbd1: Connection closed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.416229] block drbd1: conn( NetworkFailure -> Unconnected )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.416235] block drbd1: receiver terminated<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.416237] block drbd1: Restarting drbd1_receiver<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.416240] block drbd1: receiver (re)started<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:01 Storage1001B kernel: [ 647.416244] block drbd1: conn( Unconnected -> WFConnection )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: notice: ais_dispatch_message: Membership 1888: quorum lost<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: crm_update_peer: Node Storage1001A.xdomain.com: id=191128586 state=lost (new) addr=r(0) ip(10.100.100.11) votes=1 born=1864 seen=1884 proc=00000000000000000000000000111312<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 1888: memb=1, new=0, lost=1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info: pcmk_peer_update: memb: Storage1001B.xdomain.com 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info: pcmk_peer_update: lost: Storage1001A.xdomain.com 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [VOTEQ ] quorum lost, blocking activity<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [QUORUM] This node is within the non-primary component and will NOT provide any services.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [QUORUM] Members[1]: 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 1888: memb=1, new=0, lost=0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info: pcmk_peer_update: MEMB: Storage1001B.xdomain.com 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info: ais_mark_unseen_peer_dead: Node Storage1001A.xdomain.com was not seen in the previous transition<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info: update_member: Node 191128586/Storage1001A.xdomain.com is now: lost<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: ais_dispatch_message: Membership 1888: quorum lost<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [pcmk ] info: send_member_notification: Sending membership update 1888 to 2 children<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: ais_status_callback: status: Storage1001A.xdomain.com is now lost (was member)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: crm_update_peer: Node Storage1001A.xdomain.com: id=191128586 state=lost (new) addr=r(0) ip(10.100.100.11) votes=1 born=1864 seen=1884 proc=00000000000000000000000000111312<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [CPG ] chosen downlist: sender r(0) ip(10.100.100.12) ; members(old:2 left:1)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: WARN: check_dead_member: Our DC node (Storage1001A.xdomain.com) left the cluster<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B corosync[3033]: [MAIN ] Completed service synchronization, ready to provide service.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=check_dead_member ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_te_control: Registering TE UUID: aad28f35-6727-40aa-9b27-f0f51cc082a6<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: set_graph_functions: Setting custom graph functions<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_dc_takeover: Taking over DC status for this partition<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_readwrite: We are now in R/W mode<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/76, version=0.261.9): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/77, version=0.261.10): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/79, version=0.261.11): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: join_make_offer: Making join offers based on membership 1888<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/81, version=0.261.12): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: ais_dispatch_message: Membership 1888: quorum still lost<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: crmd_ais_dispatch: Setting expected votes to 2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: ais_dispatch_message: Membership 1888: quorum still lost<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/84, version=0.261.13): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: crmd_ais_dispatch: Setting expected votes to 2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: update_dc: Set DC to Storage1001B.xdomain.com (3.0.6)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/86, version=0.261.14): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_dc_join_finalize: join-1: Syncing the CIB from Storage1001B.xdomain.com to the rest of the cluster<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/88, version=0.261.14): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/89, version=0.261.15): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_dc_join_ack: join-1: Updating node state to member for Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting xpath: //node_state[@uname='Storage1001B.xdomain.com']/lrm<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001B.xdomain.com']/lrm (origin=local/crmd/90, version=0.261.16): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: abort_transition_graph: do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_local_callback: Sending full refresh (origin=crmd)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd2:1 (10000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/92, version=0.261.18): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: WARN: match_down_event: No match for shutdown action on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: te_update_diff: Stonith/shutdown of Storage1001A.xdomain.com not matched<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:234 - Triggered transition abort (complete=1, tag=node_state, id=Storage1001A.xdomain.com, magic=NA, cib=0.261.19) : Node failure<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/94, version=0.261.20): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd1:1 (10000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: p_ping (200)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node Storage1001A.xdomain.com will be fenced because it is un-expectedly down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: determine_online_status: Node Storage1001A.xdomain.com is unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: unpack_rsc_op: Processing failed op p_target_data01_last_failure_0 on Storage1001A.xdomain.com: unknown exec error (-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: common_apply_stickiness: Forcing p_target_data01 away from Storage1001A.xdomain.com after 1000000 failures (max=1000000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_lvm01_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_ip1_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_ip11_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_target_data01_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_drbd1:0_demote_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_drbd1:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_drbd1:0_demote_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_drbd1:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_drbd2:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_drbd2:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_target:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Action p_ping:0_stop_0 on Storage1001A.xdomain.com is unrunnable (offline)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: custom_action: Marking node Storage1001A.xdomain.com unclean<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node Storage1001A.xdomain.com for STONITH<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Move p_lvm01#011(Started Storage1001A.xdomain.com -> Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip1#011(Started Storage1001A.xdomain.com -> Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip11#011(Started Storage1001A.xdomain.com -> Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Move p_target_data01#011(Started Storage1001A.xdomain.com -> Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Start p_lu_data01_vol01#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Start p_email_admin1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Demote p_drbd1:0#011(Master -> Stopped Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd1:1#011(Slave -> Master Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Stop p_drbd2:0#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Stop p_target:0#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B pengine: [3072]: notice: LogActions: Stop p_ping:0#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1387317844-35) derived from /var/lib/pengine/pe-warn-179.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 134: notify p_drbd1:1_pre_notify_demote_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[88] (pid 14458)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 138: notify p_drbd2:1_pre_notify_stop_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[89] (pid 14459)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B crmd: [3073]: notice: te_fence_node: Executing reboot fencing operation (108) on Storage1001A.xdomain.com (timeout=60000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info: initiate_remote_stonith_op: Initiating remote operation reboot for Storage1001A.xdomain.com: 70f0c47c-e9be-470d-be4e-fb1da3f0587e<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info: can_fence_host_with_device: st_node_a can fence Storage1001A.xdomain.com: static-list<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info: call_remote_stonith: Requesting that Storage1001B.xdomain.com perform op reboot Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info: can_fence_host_with_device: st_node_a can fence Storage1001A.xdomain.com: static-list<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info: stonith_fence: Found 1 matching devices for 'Storage1001A.xdomain.com'<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:04 Storage1001B stonith-ng: [3069]: info: stonith_command: Processed st_fence from Storage1001B.xdomain.com: rc=-1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B pengine: [3072]: WARN: process_pe_message: Transition 0: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-179.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B pengine: [3072]: notice: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B lrmd: [3070]: info: operation notify[88] on p_drbd1:1 for client 3073: pid 14458 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_notify_0 (call=88, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B lrmd: [3070]: info: operation notify[89] on p_drbd2:1 for client 3073: pid 14459 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd2:1_notify_0 (call=89, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B stonith-ng: [3069]: notice: log_operation: Operation 'reboot' [14133] (call 0 from dd5b1efb-dcb7-4fcf-b41c-df8f0f1f97b0) for host 'Storage1001A.xdomain.com' with device 'st_node_a' returned: 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B stonith-ng: [3069]: info: log_operation: st_node_a: Performing: stonith -t external/synaccess -T reset Storage1001A<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B stonith-ng: [3069]: info: log_operation: st_node_a: success: Storage1001A 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B stonith-ng: [3069]: notice: remote_op_done: Operation reboot of Storage1001A.xdomain.com by Storage1001B.xdomain.com for Storage1001A.xdomain.com[dd5b1efb-dcb7-4fcf-b41c-df8f0f1f97b0]: OK<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:05 Storage1001B crmd: [3073]: notice: tengine_stonith_notify: Peer Storage1001A.xdomain.com was terminated (reboot) by Storage1001B.xdomain.com for Storage1001A.xdomain.com: OK (ref=be5efd32-362d-4c70-bdb8-9e27606cc748)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:06 Storage1001B kernel: [ 652.076389] e1000e: eth5 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:06 Storage1001B kernel: [ 652.076396] e1000e 0000:07:00.1: eth5: 10/100 speed: disabling TSO<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:06 Storage1001B kernel: [ 652.136019] bonding: bond1: link status definitely up for interface eth5, 10 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:06 Storage1001B kernel: [ 652.136030] bonding: bond1: first active interface up!<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:09 Storage1001B kernel: [ 655.420236] e1000e: eth5 NIC Link is Down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:09 Storage1001B kernel: [ 655.436019] bonding: bond1: link status definitely down for interface eth5, disabling it<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:09 Storage1001B kernel: [ 655.436029] bonding: bond1: now running without any active interface !<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B stonith-ng: [3069]: notice: log_operation: Operation 'reboot' [14532] (call 0 from 06b688b5-7fbe-4e14-8f63-6705013bd929) for host 'Storage1001A.xdomain.com' with device 'st_node_a' returned: 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B stonith-ng: [3069]: info: log_operation: st_node_a: Performing: stonith -t external/synaccess -T reset Storage1001A<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B stonith-ng: [3069]: info: log_operation: st_node_a: success: Storage1001A 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B stonith-ng: [3069]: notice: remote_op_done: Operation reboot of Storage1001A.xdomain.com by Storage1001B.xdomain.com for Storage1001B.xdomain.com[06b688b5-7fbe-4e14-8f63-6705013bd929]: OK<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: tengine_stonith_callback: StonithOp <st-reply st_origin="stonith_construct_async_reply" t="stonith-ng" st_op="reboot" st_remote_op="70f0c47c-e9be-470d-be4e-fb1da3f0587e" st_clientid="06b688b5-7fbe-4e14-8f63-6705013bd929" st_target="Storage1001A.xdomain.com" st_device_action="st_fence" st_callid="0" st_callopt="0" st_rc="0" st_output="Performing: stonith -t external/synaccess -T reset Storage1001A#012success: <o:p></o:p></p><p class=MsoNormal>Storage1001A 0#012" src="Storage1001B.xdomain.com" seq="6" state="2" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting xpath: //node_state[@uname='Storage1001A.xdomain.com']/lrm<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting xpath: //node_state[@uname='Storage1001A.xdomain.com']/transient_attributes<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: notice: crmd_peer_update: Status update: Client Storage1001A.xdomain.com/crmd now has status [offline] (DC=true)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: notice: tengine_stonith_notify: Peer Storage1001A.xdomain.com was terminated (reboot) by Storage1001B.xdomain.com for Storage1001B.xdomain.com: OK (ref=70f0c47c-e9be-470d-be4e-fb1da3f0587e)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 114: notify p_drbd2:1_post_notify_stop_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[90] (pid 14805)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: cib_fencing_updated: Fencing update 97 for Storage1001A.xdomain.com: complete<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 135: notify p_drbd1:1_post_notify_demote_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[91] (pid 14806)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001A.xdomain.com']/lrm (origin=local/crmd/98, version=0.261.26): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=st_node_b_last_0, magic=0:7;5:0:7:6c3b329e-4284-439d-9d22-171f31f6deb7, cib=0.261.26) : Resource op removal<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001A.xdomain.com']/transient_attributes (origin=local/crmd/99, version=0.261.27): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:194 - Triggered transition abort (complete=0, tag=transient_attributes, id=Storage1001A.xdomain.com, magic=NA, cib=0.261.27) : Transient attribute: removal<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd1:1 (1000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B attrd: [3071]: notice: attrd_perform_update: Sent update 25: master-p_drbd1:1=1000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: RA output: (p_drbd1:1:notify:stdout)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: operation notify[91] on p_drbd1:1 for client 3073: pid 14806 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair, id=status-Storage1001B.xdomain.com-master-p_drbd1.1, name=master-p_drbd1:1, value=1000, magic=NA, cib=0.261.29) : Transient attribute: update<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_notify_0 (call=91, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: RA output: (p_drbd2:1:notify:stdout)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B lrmd: [3070]: info: operation notify[90] on p_drbd2:1 for client 3073: pid 14805 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd2:1_notify_0 (call=90, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: notice: run_graph: ==== Transition 0 (Complete=36, Pending=0, Fired=0, Skipped=25, Incomplete=9, Source=/var/lib/pengine/pe-warn-179.bz2): Stopped<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start p_lvm01#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start p_ip1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start p_ip11#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start p_target_data01#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start p_lu_data01_vol01#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Start p_email_admin1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd1:1#011(Slave -> Master Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B kernel: [ 661.023431] e1000e: eth5 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:14 Storage1001B kernel: [ 661.023437] e1000e 0000:07:00.1: eth5: 10/100 speed: disabling TSO<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B kernel: [ 661.036013] bonding: bond1: link status definitely up for interface eth5, 10 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B kernel: [ 661.036023] bonding: bond1: first active interface up!<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B crmd: [3073]: info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1387317854-44) derived from /var/lib/pengine/pe-input-602.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 117: notify p_drbd1:1_pre_notify_promote_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[92] (pid 14861)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B pengine: [3072]: notice: process_pe_message: Transition 1: PEngine Input stored in: /var/lib/pengine/pe-input-602.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B lrmd: [3070]: info: operation notify[92] on p_drbd1:1 for client 3073: pid 14861 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_notify_0 (call=92, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 31: promote p_drbd1:1_promote_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 promote[93] (pid 14884)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:15 Storage1001B kernel: [ 661.950685] block drbd1: helper command: /sbin/drbdadm fence-peer minor-1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B crm-fence-peer.sh[14931]: invoked for Data01<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: - <cib admin_epoch="0" epoch="261" num_updates="29" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="262" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="Storage1001A.xdomain.com" update-client="crm_resource" cib-last-written="Tue Dec 17 14:03:36 2013" have-quorum="0" dc-uuid="Storage1001B.xdomain.com" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <configuration ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <constraints ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <rsc_location rsc="ms_drbd1" id="drbd-fence-by-handler-Data01-ms_drbd1" __crm_diff_marker__="added:top" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <rule role="Master" score="-INFINITY" id="drbd-fence-by-handler-Data01-rule-ms_drbd1" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + <expression attribute="#uname" operation="ne" value="Storage1001B.xdomain.com" id="drbd-fence-by-handler-Data01-expr-ms_drbd1" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + </rule><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + </rsc_location><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + </constraints><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + </configuration><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib:diff: + </cib><o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:126 - Triggered transition abort (complete=0, tag=diff, id=(null), magic=NA, cib=0.262.1) : Non-status change<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_create for section constraints (origin=local/cibadmin/2, version=0.262.1): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B crm-fence-peer.sh[14931]: INFO peer is not reachable, my disk is UpToDate: placed constraint 'drbd-fence-by-handler-Data01-ms_drbd1'<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B kernel: [ 663.001061] block drbd1: helper command: /sbin/drbdadm fence-peer minor-1 exit code 5 (0x500)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B kernel: [ 663.001066] block drbd1: fence-peer helper returned 5 (peer is unreachable, assumed to be dead)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B kernel: [ 663.001077] block drbd1: role( Secondary -> Primary ) pdsk( DUnknown -> Outdated )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:16 Storage1001B kernel: [ 663.023104] block drbd1: new current UUID E48A0D04194BDA73:82E61159C15A62E0:711FF2F50EF3A976:711EF2F50EF3A977<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:17 Storage1001B lrmd: [3070]: info: RA output: (p_drbd1:1:promote:stdout)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:17 Storage1001B lrmd: [3070]: info: operation promote[93] on p_drbd1:1 for client 3073: pid 14884 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:17 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_promote_0 (call=93, rc=0, cib-update=102, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 118: notify p_drbd1:1_post_notify_promote_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:17 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[94] (pid 14960)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd1:1 (10000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B attrd: [3071]: notice: attrd_perform_update: Sent update 27: master-p_drbd1:1=10000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation notify[94] on p_drbd1:1 for client 3073: pid 14960 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: RA output: (p_drbd1:1:notify:stdout)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair, id=status-Storage1001B.xdomain.com-master-p_drbd1.1, name=master-p_drbd1:1, value=10000, magic=NA, cib=0.262.3) : Transient attribute: update<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_notify_0 (call=94, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: notice: run_graph: ==== Transition 1 (Complete=9, Pending=0, Fired=0, Skipped=14, Incomplete=0, Source=/var/lib/pengine/pe-input-602.bz2): Stopped<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start p_lvm01#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start p_ip1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start p_ip11#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start p_target_data01#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start p_lu_data01_vol01#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: LogActions: Start p_email_admin1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: do_te_invoke: Processing graph 2 (ref=pe_calc-dc-1387317858-50) derived from /var/lib/pengine/pe-input-603.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 9: start p_lvm01_start_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_lvm01 start[95] (pid 15023)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 32: monitor p_drbd1:1_monitor_3000 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 monitor[96] (pid 15024)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B LVM[15023]: INFO: Activating volume group vg_data01<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B pengine: [3072]: notice: process_pe_message: Transition 2: PEngine Input stored in: /var/lib/pengine/pe-input-603.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[96] on p_drbd1:1 for client 3073: pid 15024 exited with return code 8<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_monitor_3000 (call=96, rc=8, cib-update=104, confirmed=false) master<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B LVM[15023]: INFO: Reading all physical volumes. This may take a while... Found volume group "Storage1001B" using metadata type lvm2 Found volume group "vg_data02" using metadata type lvm2 Found volume group "vg_data01" using metadata type lvm2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B LVM[15023]: INFO: 1 logical volume(s) in volume group "vg_data01" now active<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[95] on p_lvm01 for client 3073: pid 15023 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_lvm01_start_0 (call=95, rc=0, cib-update=105, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 10: monitor p_lvm01_monitor_4000 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_lvm01 monitor[97] (pid 15091)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 11: start p_ip1_start_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_ip1 start[98] (pid 15092)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B IPaddr[15092]: INFO: Using calculated netmask for 10.11.2.13: 255.255.248.0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B IPaddr[15092]: INFO: eval ifconfig bond0:0 10.11.2.13 netmask 255.255.248.0 broadcast 10.11.7.255<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[98] on p_ip1 for client 3073: pid 15092 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_ip1_start_0 (call=98, rc=0, cib-update=106, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 12: monitor p_ip1_monitor_5000 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_ip1 monitor[99] (pid 15182)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 13: start p_ip11_start_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_ip11 start[100] (pid 15183)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[99] on p_ip1 for client 3073: pid 15182 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_ip1_monitor_5000 (call=99, rc=0, cib-update=107, confirmed=false) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B IPaddr[15183]: INFO: Using calculated netmask for 10.11.10.13: 255.255.248.0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[97] on p_lvm01 for client 3073: pid 15091 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_lvm01_monitor_4000 (call=97, rc=0, cib-update=108, confirmed=false) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B IPaddr[15183]: INFO: eval ifconfig bond2:0 10.11.10.13 netmask 255.255.248.0 broadcast 10.11.15.255<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[100] on p_ip11 for client 3073: pid 15183 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_ip11_start_0 (call=100, rc=0, cib-update=109, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 14: monitor p_ip11_monitor_5000 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_ip11 monitor[101] (pid 15268)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 15: start p_target_data01_start_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_target_data01 start[102] (pid 15269)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[101] on p_ip11 for client 3073: pid 15268 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_ip11_monitor_5000 (call=101, rc=0, cib-update=110, confirmed=false) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B iSCSITarget[15269]: INFO: Successfully created network portal: 0.0.0.0:3260 created iqn.2013-10.com.xdomain.storage1001.data01 TPGT: 1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B iSCSITarget[15269]: INFO: Successfully disabled Initiator ACL mode (Enabled DemoMode) on iSCSI Target Portal Group: iqn.2013-10.com.xdomain.storage1001.data01 1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B iSCSITarget[15269]: INFO: Successfully disabled iSCSI Authentication on iSCSI Target Portal Group: iqn.2013-10.com.xdomain.storage1001.data01 1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B iSCSITarget[15269]: INFO: Successfully enabled iSCSI Target Portal Group: iqn.2013-10.com.xdomain.storage1001.data01 1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[102] on p_target_data01 for client 3073: pid 15269 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_target_data01_start_0 (call=102, rc=0, cib-update=111, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 16: monitor p_target_data01_monitor_10000 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_target_data01 monitor[103] (pid 15367)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 17: start p_lu_data01_vol01_start_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_lu_data01_vol01 start[104] (pid 15368)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[103] on p_target_data01 for client 3073: pid 15367 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_target_data01_monitor_10000 (call=103, rc=0, cib-update=112, confirmed=false) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B kernel: [ 664.698063] bio: create slab <bio-1> at 1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B iSCSILogicalUnit[15368]: INFO: Status: DEACTIVATED Execute/Left/Max Queue Depth: 0/128/128 SectorSize: 512 MaxSectors: 256 iBlock device: dm-2 UDEV PATH: /dev/vg_data01/lv_vol01 Major: 254 Minor: 2 CLAIMED: IBLOCK<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B iSCSILogicalUnit[15368]: INFO: Successfully created iSCSI Target Logical Unit<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation start[104] on p_lu_data01_vol01 for client 3073: pid 15368 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_lu_data01_vol01_start_0 (call=104, rc=0, cib-update=113, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 18: monitor p_lu_data01_vol01_monitor_10000 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_lu_data01_vol01 monitor[105] (pid 15486)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 19: start p_email_admin1_start_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: rsc:p_email_admin1 start[106] (pid 15487)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B lrmd: [3070]: info: operation monitor[105] on p_lu_data01_vol01 for client 3073: pid 15486 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:18 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_lu_data01_vol01_monitor_10000 (call=105, rc=0, cib-update=114, confirmed=false) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:19 Storage1001B lrmd: [3070]: info: operation start[106] on p_email_admin1 for client 3073: pid 15487 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:19 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_email_admin1_start_0 (call=106, rc=0, cib-update=115, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:19 Storage1001B crmd: [3073]: notice: run_graph: ==== Transition 2 (Complete=14, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-603.bz2): Complete<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:19 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:19 Storage1001B kernel: [ 665.824233] e1000e: eth5 NIC Link is Down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:19 Storage1001B kernel: [ 665.836015] bonding: bond1: link status definitely down for interface eth5, disabling it<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:19 Storage1001B kernel: [ 665.836024] bonding: bond1: now running without any active interface !<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:21 Storage1001B kernel: [ 667.416923] e1000e: eth1 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:21 Storage1001B kernel: [ 667.416930] e1000e 0000:05:00.1: eth1: 10/100 speed: disabling TSO<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:21 Storage1001B kernel: [ 667.436017] bonding: bond1: link status definitely up for interface eth1, 10 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:21 Storage1001B kernel: [ 667.436030] bonding: bond1: first active interface up!<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:21 Storage1001B kernel: [ 667.671427] e1000e: eth5 NIC Link is Up 10 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:21 Storage1001B kernel: [ 667.671433] e1000e 0000:07:00.1: eth5: 10/100 speed: disabling TSO<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:21 Storage1001B kernel: [ 667.736011] bonding: bond1: link status definitely up for interface eth5, 10 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:22 Storage1001B ntpd[2540]: Listen normally on 14 bond0:0 10.11.2.13 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:22 Storage1001B ntpd[2540]: Listen normally on 15 bond2:0 10.11.10.13 UDP 123<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:22 Storage1001B ntpd[2540]: peers refreshed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:31 Storage1001B kernel: [ 677.592125] e1000e: eth1 NIC Link is Down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:31 Storage1001B kernel: [ 677.636014] bonding: bond1: link status definitely down for interface eth1, disabling it<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:31 Storage1001B kernel: [ 677.780230] e1000e: eth5 NIC Link is Down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:31 Storage1001B kernel: [ 677.836011] bonding: bond1: link status definitely down for interface eth5, disabling it<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:31 Storage1001B kernel: [ 677.836018] bonding: bond1: now running without any active interface !<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:33 Storage1001B kernel: [ 679.808970] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:33 Storage1001B kernel: [ 679.836012] bonding: bond1: link status definitely up for interface eth1, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:33 Storage1001B kernel: [ 679.836023] bonding: bond1: first active interface up!<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:34 Storage1001B kernel: [ 680.543355] e1000e: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:04:34 Storage1001B kernel: [ 680.636010] bonding: bond1: link status definitely up for interface eth5, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="263" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="Storage1001B.xdomain.com" update-client="cibadmin" cib-last-written="Tue Dec 17 14:04:16 2013" have-quorum="1" dc-uuid="Storage1001B.xdomain.com" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:01 Storage1001B /USR/SBIN/CRON[17000]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:16 Storage1001B kernel: [ 722.400128] e1000e: eth1 NIC Link is Down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:16 Storage1001B kernel: [ 722.436017] bonding: bond1: link status definitely down for interface eth1, disabling it<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:16 Storage1001B kernel: [ 722.620238] e1000e: eth5 NIC Link is Down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:16 Storage1001B kernel: [ 722.636008] bonding: bond1: link status definitely down for interface eth5, disabling it<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:16 Storage1001B kernel: [ 722.636013] bonding: bond1: now running without any active interface !<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:18 Storage1001B kernel: [ 725.023327] e1000e: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:19 Storage1001B kernel: [ 725.036023] bonding: bond1: link status definitely up for interface eth5, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:19 Storage1001B kernel: [ 725.036036] bonding: bond1: first active interface up!<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:29 Storage1001B kernel: [ 735.816294] e1000e: eth5 NIC Link is Down<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:29 Storage1001B kernel: [ 735.836012] bonding: bond1: link status definitely down for interface eth5, disabling it<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:29 Storage1001B kernel: [ 735.836021] bonding: bond1: now running without any active interface !<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001B kernel: [ 737.220924] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001B kernel: [ 737.236010] bonding: bond1: link status definitely up for interface eth1, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:31 Storage1001B kernel: [ 737.236021] bonding: bond1: first active interface up!<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001B kernel: [ 738.343456] e1000e: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:32 Storage1001B kernel: [ 738.436013] bonding: bond1: link status definitely up for interface eth5, 1000 Mbps full duplex.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272055] block drbd1: Handshake successful: Agreed network protocol version 96<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272068] block drbd2: Handshake successful: Agreed network protocol version 96<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272074] block drbd1: conn( WFConnection -> WFReportParams )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272088] block drbd2: conn( WFConnection -> WFReportParams )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272120] block drbd2: Starting asender thread (from drbd2_receiver [2983])<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272165] block drbd1: Starting asender thread (from drbd1_receiver [2980])<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272283] block drbd1: data-integrity-alg: <not-used><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272303] block drbd1: drbd_sync_handshake:<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272306] block drbd1: self E48A0D04194BDA73:82E61159C15A62E0:711FF2F50EF3A976:711EF2F50EF3A977 bits:0 flags:0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272309] block drbd1: peer 82E61159C15A62E0:0000000000000000:711FF2F50EF3A977:711EF2F50EF3A977 bits:37888 flags:2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272313] block drbd1: uuid_compare()=1 by rule 70<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272319] block drbd1: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated -> Consistent )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272560] block drbd2: data-integrity-alg: <not-used><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272582] block drbd2: drbd_sync_handshake:<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272585] block drbd2: self 2A0715EFC37BB5BB:76A87E2130121407:B007DA0B699331F9:B006DA0B699331F9 bits:0 flags:0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272588] block drbd2: peer 76A87E2130121406:0000000000000000:B007DA0B699331F8:B006DA0B699331F9 bits:0 flags:0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272592] block drbd2: uuid_compare()=1 by rule 70<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:43 Storage1001B kernel: [ 749.272603] block drbd2: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( Outdated -> Consistent )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.082878] block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.086021] block drbd1: helper command: /sbin/drbdadm before-resync-source minor-1 exit code 0 (0x0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.086033] block drbd1: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.086043] block drbd1: Began resync as SyncSource (will sync 151552 KB [37888 bits set]).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.086065] block drbd1: updated sync UUID E48A0D04194BDA73:82E71159C15A62E0:82E61159C15A62E0:711FF2F50EF3A976<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: notice: ais_dispatch_message: Membership 1892: quorum acquired<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: crm_update_peer: Node Storage1001A.xdomain.com: id=191128586 state=member (new) addr=r(0) ip(10.100.100.11) votes=1 born=1864 seen=1892 proc=00000000000000000000000000111312<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 1892: memb=1, new=0, lost=0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: pcmk_peer_update: memb: Storage1001B.xdomain.com 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 1892: memb=2, new=1, lost=0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: update_member: Node 191128586/Storage1001A.xdomain.com is now: member<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: pcmk_peer_update: NEW: Storage1001A.xdomain.com 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: pcmk_peer_update: MEMB: Storage1001A.xdomain.com 191128586<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: pcmk_peer_update: MEMB: Storage1001B.xdomain.com 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: send_member_notification: Sending membership update 1892 to 2 children<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: notice: ais_dispatch_message: Membership 1892: quorum acquired<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: notice: crmd_peer_update: Status update: Client Storage1001A.xdomain.com/crmd now has status [online] (DC=true)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: ais_status_callback: status: Storage1001A.xdomain.com is now member (was lost)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: WARN: match_down_event: No match for shutdown action on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting xpath: //node_state[@uname='Storage1001A.xdomain.com']/lrm<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting xpath: //node_state[@uname='Storage1001A.xdomain.com']/transient_attributes<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: crm_update_peer: Node Storage1001A.xdomain.com: id=191128586 state=member (new) addr=r(0) ip(10.100.100.11) votes=1 born=1864 seen=1892 proc=00000000000000000000000000111312 (new)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001A.xdomain.com']/lrm (origin=local/crmd/117, version=0.262.17): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [VOTEQ ] quorum regained, resuming activity<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [QUORUM] This node is within the primary component and will provide service.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [QUORUM] Members[2]: 191128586 207905802<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: update_member: 0x2561200 Node 191128586 (Storage1001A.xdomain.com) born on: 1892<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [pcmk ] info: send_member_notification: Sending membership update 1892 to 2 children<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001A.xdomain.com']/transient_attributes (origin=local/crmd/118, version=0.262.18): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/119, version=0.262.19): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: ais_dispatch_message: Membership 1892: quorum retained<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/121, version=0.262.21): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: crmd_ais_dispatch: Setting expected votes to 2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: ais_dispatch_message: Membership 1892: quorum retained<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/123, version=0.262.22): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.300208] block drbd2: helper command: /sbin/drbdadm before-resync-source minor-2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.303982] block drbd2: helper command: /sbin/drbdadm before-resync-source minor-2 exit code 0 (0x0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.303993] block drbd2: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.304018] block drbd2: Began resync as SyncSource (will sync 0 KB [0 bits set]).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.304027] block drbd2: updated sync UUID 2A0715EFC37BB5BB:76A97E2130121407:76A87E2130121407:B007DA0B699331F9<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/124, version=0.262.23): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [CPG ] chosen downlist: sender r(0) ip(10.100.100.11) ; members(old:1 left:0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: crmd_ais_dispatch: Setting expected votes to 2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=crmd_peer_update ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: abort_transition_graph: do_te_invoke:169 - Triggered transition abort (complete=1) : Peer Halt<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: join_make_offer: Making join offers based on membership 1892<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: do_dc_join_offer_all: join-2: Waiting on 2 outstanding join acks<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/127, version=0.262.25): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B corosync[3033]: [MAIN ] Completed service synchronization, ready to provide service.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B crmd: [3073]: info: update_dc: Set DC to Storage1001B.xdomain.com (3.0.6)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <cib admin_epoch="0" epoch="262" num_updates="25" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <configuration ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <constraints ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <rsc_location rsc="ms_drbd2" id="drbd-fence-by-handler-Data02-ms_drbd2" __crm_diff_marker__="removed:top" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <rule role="Master" score="-INFINITY" id="drbd-fence-by-handler-Data02-rule-ms_drbd2" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - <expression attribute="#uname" operation="ne" value="Storage1001B.xdomain.com" id="drbd-fence-by-handler-Data02-expr-ms_drbd2" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - </rule><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - </rsc_location><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - </constraints><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - </configuration><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: - </cib><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="263" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="Storage1001B.xdomain.com" update-client="cibadmin" cib-last-written="Tue Dec 17 14:04:16 2013" have-quorum="1" dc-uuid="Storage1001B.xdomain.com" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section 'all' (origin=Storage1001A.xdomain.com/cibadmin/2, version=0.263.1): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=Storage1001A.xdomain.com/Storage1001A.xdomain.com/(null), version=0.263.1): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.940922] block drbd2: Resync done (total 1 sec; paused 0 sec; 0 K/sec)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.940929] block drbd2: updated UUIDs 2A0715EFC37BB5BB:0000000000000000:76A97E2130121407:76A87E2130121407<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:44 Storage1001B kernel: [ 750.940937] block drbd2: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:45 Storage1001B kernel: [ 751.403368] block drbd2: bitmap WRITE of 14901 pages took 115 jiffies<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:45 Storage1001B crmd: [3073]: info: do_dc_join_offer_all: A new node joined the cluster<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:45 Storage1001B crmd: [3073]: info: do_dc_join_offer_all: join-3: Waiting on 2 outstanding join acks<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:45 Storage1001B crmd: [3073]: info: update_dc: Set DC to Storage1001B.xdomain.com (3.0.6)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:45 Storage1001B kernel: [ 751.572257] block drbd2: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B kernel: [ 752.425119] block drbd1: Resync done (total 2 sec; paused 0 sec; 75776 K/sec)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B kernel: [ 752.425126] block drbd1: updated UUIDs E48A0D04194BDA73:0000000000000000:82E71159C15A62E0:82E61159C15A62E0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B kernel: [ 752.425135] block drbd1: conn( SyncSource -> Connected ) pdsk( Inconsistent -> UpToDate )<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <cib admin_epoch="0" epoch="263" num_updates="1" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <configuration ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <constraints ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <rsc_location rsc="ms_drbd1" id="drbd-fence-by-handler-Data01-ms_drbd1" __crm_diff_marker__="removed:top" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <rule role="Master" score="-INFINITY" id="drbd-fence-by-handler-Data01-rule-ms_drbd1" ><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - <expression attribute="#uname" operation="ne" value="Storage1001B.xdomain.com" id="drbd-fence-by-handler-Data01-expr-ms_drbd1" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - </rule><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - </rsc_location><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - </constraints><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - </configuration><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: - </cib><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib:diff: + <cib epoch="264" num_updates="1" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.6" update-origin="Storage1001A.xdomain.com" update-client="cibadmin" cib-last-written="Tue Dec 17 14:05:44 2013" have-quorum="1" dc-uuid="Storage1001B.xdomain.com" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section 'all' (origin=Storage1001A.xdomain.com/cibadmin/2, version=0.264.1): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B crmd: [3073]: info: do_dc_join_finalize: join-3: Syncing the CIB from Storage1001A.xdomain.com to the rest of the cluster<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=Storage1001A.xdomain.com/Storage1001A.xdomain.com/130, version=0.264.1): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/131, version=0.264.2): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B kernel: [ 752.571688] block drbd1: bitmap WRITE of 14889 pages took 23 jiffies<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/132, version=0.264.3): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001A.xdomain.com']/transient_attributes (origin=Storage1001A.xdomain.com/crmd/6, version=0.264.4): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:46 Storage1001B kernel: [ 752.775379] block drbd1: 0 KB (0 bits) marked out-of-sync by on disk bit-map.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: do_dc_join_ack: join-3: Updating node state to member for Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting xpath: //node_state[@uname='Storage1001A.xdomain.com']/lrm<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: do_dc_join_ack: join-3: Updating node state to member for Storage1001B.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: erase_status_tag: Deleting xpath: //node_state[@uname='Storage1001B.xdomain.com']/lrm<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001A.xdomain.com']/lrm (origin=local/crmd/133, version=0.264.5): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: abort_transition_graph: do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_local_callback: Sending full refresh (origin=crmd)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd2:1 (10000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001B.xdomain.com']/lrm (origin=local/crmd/135, version=0.264.7): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:320 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=st_node_b_last_0, magic=0:7;9:399:7:6c3b329e-4284-439d-9d22-171f31f6deb7, cib=0.264.7) : Resource op removal<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:276 - Triggered transition abort (complete=1, tag=diff, id=(null), magic=NA, cib=0.264.8) : LRM Refresh<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/137, version=0.264.9): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/139, version=0.264.11): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-p_drbd1:1 (10000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: p_ping (200)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: LogActions: Start p_drbd1:0#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: LogActions: Start p_drbd2:0#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: LogActions: Start p_target:0#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: LogActions: Start p_ping:0#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: do_te_invoke: Processing graph 3 (ref=pe_calc-dc-1387317947-74) derived from /var/lib/pengine/pe-input-604.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 14: monitor st_node_a_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 15: monitor st_node_b_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 16: monitor p_lvm01_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 17: monitor p_ip1_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 18: monitor p_ip11_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 19: monitor p_target_data01_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 20: monitor p_lu_data01_vol01_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 21: monitor p_email_admin1_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 22: monitor p_lvm02_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 23: monitor p_ip2_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 24: monitor p_ip22_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 25: monitor p_target_data02_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 26: monitor p_lu_data02_vol01_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 27: monitor p_email_admin2_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 28: monitor p_drbd1:0_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 29: monitor p_drbd2:0_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 30: monitor p_target:0_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 31: monitor p_ping:0_monitor_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 143: notify p_drbd1:1_pre_notify_start_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[107] (pid 18654)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 154: notify p_drbd2:1_pre_notify_start_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[108] (pid 18655)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B pengine: [3072]: notice: process_pe_message: Transition 3: PEngine Input stored in: /var/lib/pengine/pe-input-604.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: RA output: (p_drbd2:1:notify:stdout)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: RA output: (p_drbd1:1:notify:stdout)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: operation notify[108] on p_drbd2:1 for client 3073: pid 18655 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B lrmd: [3070]: info: operation notify[107] on p_drbd1:1 for client 3073: pid 18654 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd2:1_notify_0 (call=108, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_notify_0 (call=107, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 19 (p_target_data01_monitor_0) on Storage1001A.xdomain.com failed (target: 7 vs. rc: 0): Error<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:47 Storage1001B crmd: [3073]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=p_target_data01_last_failure_0, magic=0:0;19:3:7:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.20) : Event failed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 28 (p_drbd1:0_monitor_0) on Storage1001A.xdomain.com failed (target: 7 vs. rc: 0): Error<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001B crmd: [3073]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=p_drbd1:0_last_failure_0, magic=0:0;28:3:7:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.29) : Event failed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:49 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair, id=status-Storage1001A.xdomain.com-master-p_drbd1.0, name=master-p_drbd1:0, value=10000, magic=NA, cib=0.264.30) : Transient attribute: update<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair, id=status-Storage1001A.xdomain.com-master-p_drbd2.0, name=master-p_drbd2:0, value=10000, magic=NA, cib=0.264.33) : Transient attribute: update<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 29 (p_drbd2:0_monitor_0) on Storage1001A.xdomain.com failed (target: 7 vs. rc: 0): Error<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=p_drbd2:0_last_failure_0, magic=0:0;29:3:7:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.34) : Event failed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 30 (p_target:0_monitor_0) on Storage1001A.xdomain.com failed (target: 7 vs. rc: 0): Error<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=p_target:0_last_failure_0, magic=0:0;30:3:7:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.35) : Event failed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 13: probe_complete probe_complete on Storage1001A.xdomain.com - no waiting<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: notice: run_graph: ==== Transition 3 (Complete=29, Pending=0, Fired=0, Skipped=10, Incomplete=12, Source=/var/lib/pengine/pe-input-604.bz2): Stopped<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B pengine: [3072]: ERROR: native_create_actions: Resource p_target_data01 (ocf::iSCSITarget) is active on 2 nodes attempting recovery<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B pengine: [3072]: WARN: native_create_actions: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: LogActions: Move p_target_data01#011(Started Storage1001A.xdomain.com -> Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: LogActions: Restart p_lu_data01_vol01#011(Started Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: LogActions: Restart p_email_admin1#011(Started Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B pengine: [3072]: notice: LogActions: Start p_ping:0#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1387317950-98) derived from /var/lib/pengine/pe-error-0.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 27: stop p_email_admin1_stop_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: rsc:p_email_admin1 stop[109] (pid 18886)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 40: monitor p_drbd1:0_monitor_4000 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 72: monitor p_drbd2:0_monitor_4000 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 103: monitor p_target:0_monitor_30000 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 13: probe_complete probe_complete on Storage1001A.xdomain.com - no waiting<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 110: start p_ping:0_start_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B pengine: [3072]: ERROR: process_pe_message: Transition 4: ERRORs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-error-0.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: operation stop[109] on p_email_admin1 for client 3073: pid 18886 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_email_admin1_stop_0 (call=109, rc=0, cib-update=144, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 25: stop p_lu_data01_vol01_stop_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: cancel_op: operation monitor[105] on p_lu_data01_vol01 for client 3073, its parameters: implementation=[lio] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] target_iqn=[iqn.2013-10.com.xdomain.storage1001.data01] path=[/dev/vg_data01/lv_vol01] crm_feature_set=[3.0.6] CRM_meta_interval=[10000] lun=[1] cancelled<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: rsc:p_lu_data01_vol01 stop[110] (pid 18902)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_lu_data01_vol01_monitor_10000 (call=105, status=1, cib-update=0, confirmed=true) Cancelled<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B iSCSILogicalUnit[18902]: INFO: Successfully deleted iSCSI Target Logical Unit<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: operation stop[110] on p_lu_data01_vol01 for client 3073: pid 18902 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_lu_data01_vol01_stop_0 (call=110, rc=0, cib-update=145, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 23: stop p_target_data01_stop_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: cancel_op: operation monitor[103] on p_target_data01 for client 3073, its parameters: crm_feature_set=[3.0.6] CRM_meta_timeout=[20000] CRM_meta_name=[monitor] CRM_meta_interval=[10000] implementation=[lio] iqn=[iqn.2013-10.com.xdomain.storage1001.data01] cancelled<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B lrmd: [3070]: info: rsc:p_target_data01 stop[111] (pid 18952)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 22: stop p_target_data01_stop_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_target_data01_monitor_10000 (call=103, status=1, cib-update=0, confirmed=true) Cancelled<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.825198] BUG: unable to handle kernel NULL pointer dereference at 0000000000000088<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.825358] IP: [<ffffffffa04bf993>] transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.825515] PGD 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.825561] Oops: 0000 [#1] SMP<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.825641] CPU 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.825680] Modules linked in: crc32c drbd lru_cache tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod configfs nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc bonding ext2 loop radeon ttm drm_kms_helper drm power_supply i2c_algo_bit i5000_edac iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc snd_timer processor edac_core thermal_sys i5k_amb snd soundcore i2c_i801 psmouse joydev pcspkr serio_raw i2c_core coretemp evdev shpchp ioatdma rng_core dca container button ext4 crc16 jbd2 mbcache sr_mod cdrom dm_mod usbhid hid sg sd_mod crc_t10dif usb_storage uhci_hcd ahci libahci libata ehci_hcd usbcore usb_common 3w_9xxx scsi_mod e1000e [last unloaded: scsi_wait_scan]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.827987]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.828025] Pid: 2903, comm: iscsi_ttx Not tainted 3.2.0-4-amd64 #1 Debian 3.2.51-1 Supermicro X7DBN/X7DBN<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.828216] RIP: 0010:[<ffffffffa04bf993>] [<ffffffffa04bf993>] transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.828393] RSP: 0018:ffff8801a7015d90 EFLAGS: 00010246<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.828476] RAX: 0000000000000000 RBX: ffff8801a521e280 RCX: ffff8801a521e468<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.828579] RDX: dead000000200200 RSI: 0000000000000286 RDI: ffff8801a7a9ab80<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.828683] RBP: ffff8801a7a9abb8 R08: ffff8801a7015d90 R09: ffff8801a521e4a0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.828787] R10: ffff8801a4f39c00 R11: ffff8801a4f39c00 R12: ffff8801a7015d90<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.828891] R13: ffff8801a7a9ab80 R14: 0000000000000008 R15: ffff8801a4f39d48<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.828995] FS: 0000000000000000(0000) GS:ffff8801afc00000(0000) knlGS:0000000000000000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829118] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] CR2: 0000000000000088 CR3: 00000001a5620000 CR4: 00000000000006f0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] Process iscsi_ttx (pid: 2903, threadinfo ffff8801a7014000, task ffff8801a5e93590)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] Stack:<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] ffff8801a7015d90 ffff8801a7015d90 ffff880199232668 ffff8801a521e280<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] ffff8801a561a400 ffff8801a4f39ee8 ffff8801a4f39d48 ffffffffa04c0d22<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] ffff8801a4f39c00 ffffffffa04eef03 ffff8801a7015fd8 ffffffff81070fc1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] Call Trace:<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffffa04c0d22>] ? transport_generic_free_cmd+0x5c/0x65 [target_core_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffffa04eef03>] ? iscsit_close_connection+0x120/0x4bb [iscsi_target_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff81070fc1>] ? arch_local_irq_save+0x11/0x17<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffffa04f038a>] ? iscsi_target_tx_thread+0x10ec/0x112b [iscsi_target_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff81039ac2>] ? finish_task_switch+0x88/0xb9<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff8105fc83>] ? add_wait_queue+0x3c/0x3c<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffffa04ef29e>] ? iscsit_close_connection+0x4bb/0x4bb [iscsi_target_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff8105f631>] ? kthread+0x76/0x7e<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff81356374>] ? kernel_thread_helper+0x4/0x10<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff8105f5bb>] ? kthread_worker_fn+0x139/0x139<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] [<ffffffff81356370>] ? gs_change+0x13/0x13<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] Code: 00 4c 8d 6d c8 74 0e 48 3b bb 10 02 00 00 74 05 e8 aa be c2 e0 48 89 ef e8 a7 f2 ff ff 48 8b 43 78 4c 89 ef 48 8b 80 90 02 00 00 <ff> 90 88 00 00 00 48 8b 2c 24 4c 39 e5 75 bc 48 83 c4 18 5b 5d<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] RIP [<ffffffffa04bf993>] transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] RSP <ffff8801a7015d90><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.829150] CR2: 0000000000000088<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:50 Storage1001B kernel: [ 756.864099] ---[ end trace bbe9d829bd42ef9a ]---<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.309687] BUG: unable to handle kernel NULL pointer dereference at 0000000000000088<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] IP: [<ffffffffa04bf993>] transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] PGD 1a6f1e067 PUD 1a6ffb067 PMD 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Oops: 0000 [#2] SMP<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] CPU 1<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Modules linked in: crc32c drbd lru_cache tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod configfs nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc bonding ext2 loop radeon ttm drm_kms_helper drm power_supply i2c_algo_bit i5000_edac iTCO_wdt iTCO_vendor_support snd_pcm snd_page_alloc snd_timer processor edac_core thermal_sys i5k_amb snd soundcore i2c_i801 psmouse joydev pcspkr serio_raw i2c_core coretemp evdev shpchp ioatdma rng_core dca container button ext4 crc16 jbd2 mbcache sr_mod cdrom dm_mod usbhid hid sg sd_mod crc_t10dif usb_storage uhci_hcd ahci libahci libata ehci_hcd usbcore usb_common 3w_9xxx scsi_mod e1000e [last unloaded: scsi_wait_scan]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Pid: 2905, comm: iscsi_ttx Tainted: G D 3.2.0-4-amd64 #1 Debian 3.2.51-1 Supermicro X7DBN/X7DBN<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RIP: 0010:[<ffffffffa04bf993>] [<ffffffffa04bf993>] transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RSP: 0018:ffff8801a74cfdf0 EFLAGS: 00010246<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RAX: 0000000000000000 RBX: ffff88019a161d00 RCX: ffff88019a161ee8<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RDX: dead000000200200 RSI: 0000000000000286 RDI: ffff8801a523ca80<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] RBP: ffff8801a523cab8 R08: ffff8801a74cfdf0 R09: ffff88019a161f20<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] R10: ffff8801a576d438 R11: ffff8801a576d438 R12: ffff8801a74cfdf0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] R13: ffff8801a523ca80 R14: 0000000000000000 R15: 0000000000000000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] FS: 0000000000000000(0000) GS:ffff8801afc40000(0000) knlGS:0000000000000000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] CR2: 0000000000000088 CR3: 00000001a686d000 CR4: 00000000000006e0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Process iscsi_ttx (pid: 2905, threadinfo ffff8801a74ce000, task ffff8801a5f68730)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Stack:<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] ffff8801a74cfdf0 ffff8801a74cfdf0 ffff8801a6d7f668 ffff88019a161d00<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] ffff880199ccfc00 ffff88019a161ac0 ffff88019a161b29 ffffffffa04c0d22<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] ffff88019a161ba8 ffffffffa04ef4db ffff8801a5f68730 ffffffff81039ac2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] Call Trace:<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.313623] [<ffffffffa04c0d22>] ? transport_generic_free_cmd+0x5c/0x65 [target_core_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffffa04ef4db>] ? iscsi_target_tx_thread+0x23d/0x112b [iscsi_target_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff81039ac2>] ? finish_task_switch+0x88/0xb9<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff8105fc83>] ? add_wait_queue+0x3c/0x3c<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffffa04ef29e>] ? iscsit_close_connection+0x4bb/0x4bb [iscsi_target_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff8105f631>] ? kthread+0x76/0x7e<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff81356374>] ? kernel_thread_helper+0x4/0x10<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff8105f5bb>] ? kthread_worker_fn+0x139/0x139<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] [<ffffffff81356370>] ? gs_change+0x13/0x13<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] Code: 00 4c 8d 6d c8 74 0e 48 3b bb 10 02 00 00 74 05 e8 aa be c2 e0 48 89 ef e8 a7 f2 ff ff 48 8b 43 78 4c 89 ef 48 8b 80 90 02 00 00 <ff> 90 88 00 00 00 48 8b 2c 24 4c 39 e5 75 bc 48 83 c4 18 5b 5d<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] RIP [<ffffffffa04bf993>] transport_free_dev_tasks+0xc6/0xe0 [target_core_mod]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] RSP <ffff8801a74cfdf0><o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.510614] CR2: 0000000000000088<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:51 Storage1001B kernel: [ 757.593194] ---[ end trace bbe9d829bd42ef9b ]---<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:53 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 111: monitor p_ping:0_monitor_5000 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:05:58 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=0, tag=nvpair, id=status-Storage1001A.xdomain.com-p_ping, name=p_ping, value=200, magic=NA, cib=0.264.47) : Transient attribute: update<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:00 Storage1001B kernel: [ 766.314376] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:00 Storage1001B kernel: [ 766.320550] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:03 Storage1001B kernel: [ 769.323055] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:03 Storage1001B kernel: [ 769.328979] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:04 Storage1001B kernel: [ 770.080981] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:04 Storage1001B kernel: [ 770.086770] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:06 Storage1001B kernel: [ 772.338555] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:06 Storage1001B kernel: [ 772.344101] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:07 Storage1001B kernel: [ 773.847066] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:07 Storage1001B kernel: [ 773.852207] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B kernel: [ 776.104111] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B kernel: [ 776.108920] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B lrmd: [3070]: WARN: p_target_data01:stop process (PID 18952) timed out (try 1). Killing with signal SIGTERM (15).<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B lrmd: [3070]: WARN: operation stop[111] on p_target_data01 for client 3073: pid 18952 timed out<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B crmd: [3073]: ERROR: process_lrm_event: LRM operation p_target_data01_stop_0 (111) Timed Out (timeout=20000ms)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B crmd: [3073]: WARN: status_from_rc: Action 23 (p_target_data01_stop_0) on Storage1001B.xdomain.com failed (target: 0 vs. rc: -2): Error<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B crmd: [3073]: WARN: update_failcount: Updating failcount for p_target_data01 on Storage1001B.xdomain.com after failed stop: rc=-2 (update=INFINITY, time=1387317970)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B crmd: [3073]: info: abort_transition_graph: match_graph_event:277 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=p_target_data01_last_failure_0, magic=2:-2;23:4:0:aad28f35-6727-40aa-9b27-f0f51cc082a6, cib=0.264.48) : Event failed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B crmd: [3073]: notice: run_graph: ==== Transition 4 (Complete=13, Pending=0, Fired=0, Skipped=9, Incomplete=0, Source=/var/lib/pengine/pe-error-0.bz2): Stopped<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_target_data01 (INFINITY)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B attrd: [3071]: notice: attrd_perform_update: Sent update 45: fail-count-p_target_data01=INFINITY<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B attrd: [3071]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_target_data01 (1387317970)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-Storage1001B.xdomain.com-fail-count-p_target_data01, name=fail-count-p_target_data01, value=INFINITY, magic=NA, cib=0.264.49) : Transient attribute: update<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B attrd: [3071]: notice: attrd_perform_update: Sent update 48: last-failure-p_target_data01=1387317970<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: WARN: unpack_rsc_op: Processing failed op p_target_data01_last_failure_0 on Storage1001B.xdomain.com: unknown exec error (-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node Storage1001B.xdomain.com will be fenced to recover from resource failure(s)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node Storage1001B.xdomain.com for STONITH<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: native_stop_constraints: Stop of failed resource p_target_data01 is implicit after Storage1001B.xdomain.com is fenced<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Stop st_node_a#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Move p_lvm01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip1#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip11#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Move p_target_data01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Start p_lu_data01_vol01#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B crmd: [3073]: info: abort_transition_graph: te_update_diff:176 - Triggered transition abort (complete=1, tag=nvpair, id=status-Storage1001B.xdomain.com-last-failure-p_target_data01, name=last-failure-p_target_data01, value=1387317970, magic=NA, cib=0.264.50) : Transient attribute: update<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Start p_email_admin1#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd1:0#011(Slave -> Master Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Demote p_drbd1:1#011(Master -> Stopped Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd2:0#011(Slave -> Master Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Demote p_drbd2:1#011(Master -> Stopped Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Stop p_target:1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B pengine: [3072]: notice: LogActions: Stop p_ping:1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:10 Storage1001B crmd: [3073]: info: handle_response: pe_calc calculation pe_calc-dc-1387317970-109 is obsolete<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: process_pe_message: Transition 5: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-180.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: unpack_rsc_op: Processing failed op p_target_data01_last_failure_0 on Storage1001B.xdomain.com: unknown exec error (-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node Storage1001B.xdomain.com will be fenced to recover from resource failure(s)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: common_apply_stickiness: Forcing p_target_data01 away from Storage1001B.xdomain.com after 1000000 failures (max=1000000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node Storage1001B.xdomain.com for STONITH<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: native_stop_constraints: Stop of failed resource p_target_data01 is implicit after Storage1001B.xdomain.com is fenced<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop st_node_a#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move p_lvm01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip1#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip11#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move p_target_data01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Start p_lu_data01_vol01#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Start p_email_admin1#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd1:0#011(Slave -> Master Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Demote p_drbd1:1#011(Master -> Stopped Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd2:0#011(Slave -> Master Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Demote p_drbd2:1#011(Master -> Stopped Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop p_target:1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop p_ping:1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: handle_response: pe_calc calculation pe_calc-dc-1387317970-110 is obsolete<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: process_pe_message: Transition 6: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-181.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: unpack_rsc_op: Processing failed op p_target_data01_last_failure_0 on Storage1001B.xdomain.com: unknown exec error (-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node Storage1001B.xdomain.com will be fenced to recover from resource failure(s)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: common_apply_stickiness: Forcing p_target_data01 away from Storage1001B.xdomain.com after 1000000 failures (max=1000000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node Storage1001B.xdomain.com for STONITH<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: native_stop_constraints: Stop of failed resource p_target_data01 is implicit after Storage1001B.xdomain.com is fenced<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop st_node_a#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move p_lvm01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip1#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip11#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Move p_target_data01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Start p_lu_data01_vol01#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Start p_email_admin1#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd1:0#011(Slave -> Master Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Demote p_drbd1:1#011(Master -> Stopped Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd2:0#011(Slave -> Master Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Demote p_drbd2:1#011(Master -> Stopped Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop p_target:1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: LogActions: Stop p_ping:1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B kernel: [ 777.611642] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B kernel: [ 777.616045] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: do_te_invoke: Processing graph 7 (ref=pe_calc-dc-1387317970-111) derived from /var/lib/pengine/pe-warn-182.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 1: cancel p_drbd1:0_monitor_4000 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 3: cancel p_drbd2:0_monitor_4000 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 145: notify p_drbd1:0_pre_notify_demote_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 147: notify p_drbd1:1_pre_notify_demote_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[112] (pid 19574)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 155: notify p_drbd2:0_pre_notify_demote_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 157: notify p_drbd2:1_pre_notify_demote_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[113] (pid 19575)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: notice: te_fence_node: Executing reboot fencing operation (118) on Storage1001B.xdomain.com (timeout=60000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B stonith-ng: [3069]: info: initiate_remote_stonith_op: Initiating remote operation reboot for Storage1001B.xdomain.com: 832bbb47-e241-4702-bc48-dcd387841ed7<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B stonith-ng: [3069]: info: can_fence_host_with_device: st_node_a can not fence Storage1001B.xdomain.com: static-list<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B stonith-ng: [3069]: info: stonith_command: Processed st_query from Storage1001B.xdomain.com: rc=0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: WARN: process_pe_message: Transition 7: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-182.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B pengine: [3072]: notice: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B lrmd: [3070]: info: operation notify[112] on p_drbd1:1 for client 3073: pid 19574 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_notify_0 (call=112, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B lrmd: [3070]: info: operation notify[113] on p_drbd2:1 for client 3073: pid 19575 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:11 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd2:1_notify_0 (call=113, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:13 Storage1001B kernel: [ 779.868256] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:13 Storage1001B kernel: [ 779.872318] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:15 Storage1001B kernel: [ 781.374749] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:15 Storage1001B kernel: [ 781.378561] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B stonith-ng: [3069]: ERROR: remote_op_done: Operation reboot of Storage1001B.xdomain.com by <no-one> for Storage1001B.xdomain.com[06b688b5-7fbe-4e14-8f63-6705013bd929]: Operation timed out<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: tengine_stonith_callback: StonithOp <remote-op state="0" st_target="Storage1001B.xdomain.com" st_op="reboot" /><o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: tengine_stonith_callback: Stonith operation 3 for Storage1001B.xdomain.com failed (Operation timed out): aborting transition.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: abort_transition_graph: tengine_stonith_callback:454 - Triggered transition abort (complete=0) : Stonith failed<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: tengine_stonith_notify: Peer Storage1001B.xdomain.com was not terminated (reboot) by <anyone> for Storage1001B.xdomain.com: Operation timed out (ref=832bbb47-e241-4702-bc48-dcd387841ed7)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: run_graph: ==== Transition 7 (Complete=16, Pending=0, Fired=0, Skipped=53, Incomplete=30, Source=/var/lib/pengine/pe-warn-182.bz2): Stopped<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B kernel: [ 783.630944] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B kernel: [ 783.634589] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: WARN: unpack_rsc_op: Processing failed op p_target_data01_last_failure_0 on Storage1001B.xdomain.com: unknown exec error (-2)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: WARN: pe_fence_node: Node Storage1001B.xdomain.com will be fenced to recover from resource failure(s)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: WARN: common_apply_stickiness: Forcing p_target_data01 away from Storage1001B.xdomain.com after 1000000 failures (max=1000000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: WARN: stage6: Scheduling Node Storage1001B.xdomain.com for STONITH<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: native_stop_constraints: Stop of failed resource p_target_data01 is implicit after Storage1001B.xdomain.com is fenced<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Stop st_node_a#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Move p_lvm01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip1#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Move p_ip11#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Move p_target_data01#011(Started Storage1001B.xdomain.com -> Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Start p_lu_data01_vol01#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Start p_email_admin1#011(Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd1:0#011(Slave -> Master Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Demote p_drbd1:1#011(Master -> Stopped Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Promote p_drbd2:0#011(Slave -> Master Storage1001A.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Demote p_drbd2:1#011(Master -> Stopped Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Stop p_target:1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B pengine: [3072]: notice: LogActions: Stop p_ping:1#011(Storage1001B.xdomain.com)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: do_te_invoke: Processing graph 8 (ref=pe_calc-dc-1387317977-120) derived from /var/lib/pengine/pe-warn-183.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 1: cancel p_drbd1:0_monitor_4000 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 3: cancel p_drbd2:0_monitor_4000 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 145: notify p_drbd1:0_pre_notify_demote_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 147: notify p_drbd1:1_pre_notify_demote_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B lrmd: [3070]: info: rsc:p_drbd1:1 notify[114] (pid 19824)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 155: notify p_drbd2:0_pre_notify_demote_0 on Storage1001A.xdomain.com<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: te_rsc_command: Initiating action 157: notify p_drbd2:1_pre_notify_demote_0 on Storage1001B.xdomain.com (local)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B lrmd: [3070]: info: rsc:p_drbd2:1 notify[115] (pid 19825)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: notice: te_fence_node: Executing reboot fencing operation (118) on Storage1001B.xdomain.com (timeout=60000)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001A.xdomain.com']//lrm_resource[@id='p_drbd1:0']/lrm_rsc_op[@id='p_drbd1:0_monitor_4000'] (origin=Storage1001A.xdomain.com/crmd/33, version=0.264.53): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B stonith-ng: [3069]: info: initiate_remote_stonith_op: Initiating remote operation reboot for Storage1001B.xdomain.com: cf2bcd82-7fe0-4316-8318-27e91a137adc<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B stonith-ng: [3069]: info: can_fence_host_with_device: st_node_a can not fence Storage1001B.xdomain.com: static-list<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B stonith-ng: [3069]: info: stonith_command: Processed st_query from Storage1001B.xdomain.com: rc=0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B cib: [3068]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='Storage1001A.xdomain.com']//lrm_resource[@id='p_drbd2:0']/lrm_rsc_op[@id='p_drbd2:0_monitor_4000'] (origin=Storage1001A.xdomain.com/crmd/34, version=0.264.54): ok (rc=0)<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B lrmd: [3070]: info: operation notify[114] on p_drbd1:1 for client 3073: pid 19824 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B lrmd: [3070]: info: operation notify[115] on p_drbd2:1 for client 3073: pid 19825 exited with return code 0<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd1:1_notify_0 (call=114, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:17 Storage1001B crmd: [3073]: info: process_lrm_event: LRM operation p_drbd2:1_notify_0 (call=115, rc=0, cib-update=0, confirmed=true) ok<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:18 Storage1001B pengine: [3072]: WARN: process_pe_message: Transition 8: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-183.bz2<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:18 Storage1001B pengine: [3072]: notice: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:19 Storage1001B kernel: [ 785.136801] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:19 Storage1001B kernel: [ 785.140293] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:21 Storage1001B kernel: [ 787.392813] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:21 Storage1001B kernel: [ 787.396060] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:22 Storage1001B kernel: [ 788.898172] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal>Dec 17 14:06:22 Storage1001B kernel: [ 788.901280] iSCSI Login negotiation failed.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>