<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.16.3">
</HEAD>
<BODY>
<FONT COLOR="#000000">Hi Emmanuel,</FONT><BR>
<BR>
<BLOCKQUOTE TYPE=CITE>
<FONT COLOR="#000000">i don't see any reason for put libvirtd as primitiv</FONT>e<FONT COLOR="#000000"> in pacemaker</FONT><BR>
</BLOCKQUOTE>
<BR>
Yes ... well, maybe. During my testing of failure scenarios (in particular, reboot of the VM host), several times the VM filesystem ended up corrupted and I needed to reinstall the VM. At least a couple of these failures occurred when I was testing with the system starting libvirtd and not controlling libvirtd start/stop via a cloned resource.<BR>
<BR>
And, those failures are the reason that I'm seeking the wisdom of others.<BR>
<BR>
Now that I understand more the issues, I will be again testing system start of libvirt, with more care.<BR>
<BR>
Thanks,<BR>
Bob Haxo<BR>
<BR>
<BR>
<BR>
On Thu, 2013-12-19 at 21:30 +0100, emmanuel segura wrote:
<BLOCKQUOTE TYPE=CITE>
<FONT COLOR="#000000">remove the libvirtd from pacemaker and chkconfig libvirtd on every node, like that the cluster just manage the vm, maybe i wrong but i don't see any reason for put libvirtd as primitivi in pacemaker</FONT><BR>
<BR>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BR>
<BR>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<FONT COLOR="#000000">2013/12/19 Bob Haxo <<A HREF="mailto:bhaxo@sgi.com">bhaxo@sgi.com</A>></FONT>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<FONT COLOR="#000000">Hi Emmanuel,</FONT><BR>
<BR>
<FONT COLOR="#000000">Thanks for the suggestions. It is pretty clear what is the problem; it's just not clear what is the fix or the work-around. </FONT><BR>
<BR>
<FONT COLOR="#000000">Search the Pacemaker email archive for the email of Andrew Beekhof, 12 Oct 2012, "Re: [Pacemaker] chicken-egg-problem with libvirtd and a VM within cluster", and the email to which he is responding (from Tom Fernandes).</FONT><BR>
<BR>
<FONT COLOR="#000000">The status/monitor function of VirtualDomain fails because the /var/run/libvirt/libvirt-sock has not been created. This socket is created by the lsb:libvirtd, but that is not started (as a resource) until Pacemaker has heard back from heartbeat:VirtualDomain, which will never happen until /var/run/libvirt/libvirt-sock has been created ("service libvirtd start" during this wait period does enable Pacemaker to continue starting resources). After the VirtualDomain monitor function timeout, Pacemaker deals with the failing logic loop, resulting in a re-start of the VM.</FONT><BR>
<BR>
<FONT COLOR="#000000">I hoping that "Unfortunately we still don't have a good answer for you." is no longer the case, and that there is a fix or that there is a community accepted workaround for the issue.</FONT><BR>
<BR>
<BR>
<FONT COLOR="#000000">Regards,</FONT><BR>
<FONT COLOR="#000000">Bob Haxo</FONT>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<FONT COLOR="#000000">On Thu, 2013-12-19 at 19:48 +0100, emmanuel segura wrote: </FONT><BR>
<BLOCKQUOTE TYPE=CITE>
<FONT COLOR="#000000">Maybe the problem is this, the cluster try to start the vm and libvirtd isn't started</FONT><BR>
<BR>
<BR>
<BR>
<FONT COLOR="#000000">2013/12/19 emmanuel segura <<A HREF="mailto:emi2fast@gmail.com">emi2fast@gmail.com</A>></FONT><BR>
<BLOCKQUOTE>
<FONT COLOR="#000000">if don't set your vm to start at boot time, you don't to put in cluster libvirtd, maybe the problem isn't this, but why put the os services in cluster, for example crond ...... :)</FONT><BR>
<BR>
<BR>
<BR>
<FONT COLOR="#000000">2013/12/19 Bob Haxo <<A HREF="mailto:bhaxo@sgi.com">bhaxo@sgi.com</A>> </FONT><BR>
<BLOCKQUOTE>
<FONT COLOR="#000000">Hello,</FONT><BR>
<BR>
<FONT COLOR="#000000">Earlier emails related to this topic:</FONT><BR>
<FONT COLOR="#000000">[pacemaker] chicken-egg-problem with libvirtd and a VM within cluster</FONT><BR>
<FONT COLOR="#000000">[pacemaker] VirtualDomain problem after reboot of one node</FONT><BR>
<BR>
<BR>
<FONT COLOR="#000000">My configuration:</FONT><BR>
<BR>
<FONT COLOR="#000000">RHEL6.5/CMAN/gfs2/Pacemaker/crmsh</FONT><BR>
<BR>
<FONT COLOR="#000000">pacemaker-libs-1.1.10-14.el6_5.1.x86_64</FONT><BR>
<FONT COLOR="#000000">pacemaker-cli-1.1.10-14.el6_5.1.x86_64</FONT><BR>
<FONT COLOR="#000000">pacemaker-1.1.10-14.el6_5.1.x86_64</FONT><BR>
<FONT COLOR="#000000">pacemaker-cluster-libs-1.1.10-14.el6_5.1.x86_64</FONT><BR>
<BR>
<FONT COLOR="#000000">Two node HA VM cluster using real shared drive, not drbd.</FONT><BR>
<BR>
<FONT COLOR="#000000">Resources (relevant to this discussion):</FONT><BR>
<FONT COLOR="#000000">primitive p_fs_images ocf:heartbeat:Filesystem \</FONT><BR>
<FONT COLOR="#000000">primitive p_libvirtd lsb:libvirtd \</FONT><BR>
<FONT COLOR="#000000">primitive virt ocf:heartbeat:VirtualDomain \</FONT><BR>
<BR>
<FONT COLOR="#000000">services chkconfig on: cman, clvmd, pacemaker</FONT><BR>
<FONT COLOR="#000000">services chkconfig off: corosync, gfs2, libvirtd</FONT><BR>
<BR>
<FONT COLOR="#000000">Observation:</FONT><BR>
<BR>
<FONT COLOR="#000000">Rebooting the NON-host system results in the restart of the VM merrily running on the host system.</FONT><BR>
<BR>
<FONT COLOR="#000000">Apparent cause:</FONT><BR>
<BR>
<FONT COLOR="#000000">Upon startup, Pacemaker apparently checks the status of configured resources. However, the status request for the virt (ocf:heartbeat:VirtualDomain) resource fails with:</FONT><BR>
<BR>
<PRE>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: warning: child_timeout_callback: virt_monitor_0 process (PID 4158) timed out</FONT>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: warning: operation_finished: virt_monitor_0:4158 - timed out after 200000ms</FONT>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: notice: operation_finished: virt_monitor_0:4158:stderr [ error: Failed to reconnect to the hypervisor ]</FONT>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: notice: operation_finished: virt_monitor_0:4158:stderr [ error: no valid connection ]</FONT>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: notice: operation_finished: virt_monitor_0:4158:stderr [ error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory ]</FONT>
</PRE>
<FONT COLOR="#000000">This failure then snowballs into an "orphan" situation in which the running VM is restarted.</FONT><BR>
<BR>
<FONT COLOR="#000000">There was the suggestion of chkconfig on libvirtd (and presumably deleting the resource) so that the /var/run/libvirt/libvirt-sock has been created by service libvirtd. With libvirtd started by the system, there is no un-needed reboot of the VM.</FONT><BR>
<BR>
<FONT COLOR="#000000">However, it may be that removing libvirtd from Pacemaker control leaves the VM vdisk filesystem susceptible to corruption during a reboot induced failover.</FONT><BR>
<BR>
<FONT COLOR="#000000">Question:</FONT><BR>
<BR>
<FONT COLOR="#000000">Is there an accepted Pacemaker configuration such that the un-needed restart of the VM does not occur with the reboot of the non-host system?</FONT><BR>
<BR>
<FONT COLOR="#000000">Regards,</FONT><BR>
<FONT COLOR="#000000">Bob Haxo</FONT><BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<FONT COLOR="#000000">_______________________________________________</FONT><BR>
<FONT COLOR="#000000">Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A></FONT><BR>
<FONT COLOR="#000000"><A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT><BR>
<BR>
<FONT COLOR="#000000">Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT><BR>
<FONT COLOR="#000000">Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT><BR>
<FONT COLOR="#000000">Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT><BR>
<BR>
</BLOCKQUOTE>
<BR>
<BR>
<BR>
<FONT COLOR="#888888">-- </FONT><BR>
<FONT COLOR="#888888">esta es mi vida e me la vivo hasta que dios quiera</FONT><FONT COLOR="#000000"> </FONT><BR>
</BLOCKQUOTE>
<BR>
<BR>
<BR>
<FONT COLOR="#000000">-- </FONT><BR>
<FONT COLOR="#000000">esta es mi vida e me la vivo hasta que dios quiera</FONT>
<PRE>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A></FONT>
<FONT COLOR="#000000"><A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT>
<FONT COLOR="#000000">Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT>
<FONT COLOR="#000000">Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT>
</PRE>
</BLOCKQUOTE>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
<FONT COLOR="#000000">_______________________________________________</FONT><BR>
<FONT COLOR="#000000">Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A></FONT><BR>
<FONT COLOR="#000000"><A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT><BR>
<BR>
<FONT COLOR="#000000">Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT><BR>
<FONT COLOR="#000000">Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT><BR>
<FONT COLOR="#000000">Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT><BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BR>
<BR>
<BR>
<FONT COLOR="#000000">-- </FONT><BR>
<FONT COLOR="#000000">esta es mi vida e me la vivo hasta que dios quiera</FONT>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A></FONT>
<FONT COLOR="#000000"><A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT>
<FONT COLOR="#000000">Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT>
<FONT COLOR="#000000">Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>