<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/3.16.3">
</HEAD>
<BODY>
Hi Emmanuel,<BR>
<BR>
Thanks for the suggestions. It is pretty clear what is the problem; it's just not clear what is the fix or the work-around. <BR>
<BR>
Search the Pacemaker email archive for the email of Andrew Beekhof, 12 Oct 2012, "Re: [Pacemaker] chicken-egg-problem with libvirtd and a VM within cluster", and the email to which he is responding (from Tom Fernandes).<BR>
<BR>
The status/monitor function of VirtualDomain fails because the <FONT COLOR="#000000">/var/run/libvirt/libvirt-sock</FONT> has not been created. This socket is created by the lsb:libvirtd, but that is not started (as a resource) until Pacemaker has heard back from heartbeat:VirtualDomain, which will never happen until <FONT COLOR="#000000">/var/run/libvirt/libvirt-sock</FONT> has been created ("service libvirtd start" during this wait period does enable Pacemaker to continue starting resources). After the VirtualDomain monitor function timeout, Pacemaker deals with the failing logic loop, resulting in a re-start of the VM.<BR>
<BR>
I hoping that "Unfortunately we still don't have a good answer for you." is no longer the case, and that there is a fix or that there is a community accepted workaround for the issue.<BR>
<BR>
<BR>
Regards,<BR>
Bob Haxo<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
On Thu, 2013-12-19 at 19:48 +0100, emmanuel segura wrote:
<BLOCKQUOTE TYPE=CITE>
<FONT COLOR="#000000">Maybe the problem is this, the cluster try to start the vm and libvirtd isn't started</FONT><BR>
<BR>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BR>
<BR>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<FONT COLOR="#000000">2013/12/19 emmanuel segura <<A HREF="mailto:emi2fast@gmail.com">emi2fast@gmail.com</A>></FONT>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<FONT COLOR="#000000">if don't set your vm to start at boot time, you don't to put in cluster libvirtd, maybe the problem isn't this, but why put the os services in cluster, for example crond ...... :)</FONT><BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<FONT COLOR="#000000">2013/12/19 Bob Haxo <<A HREF="mailto:bhaxo@sgi.com">bhaxo@sgi.com</A>></FONT>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BLOCKQUOTE>
<FONT COLOR="#000000">Hello,</FONT><BR>
<BR>
<FONT COLOR="#000000">Earlier emails related to this topic:</FONT><BR>
<FONT COLOR="#000000">[pacemaker] chicken-egg-problem with libvirtd and a VM within cluster</FONT><BR>
<FONT COLOR="#000000">[pacemaker] VirtualDomain problem after reboot of one node</FONT><BR>
<BR>
<BR>
<FONT COLOR="#000000">My configuration:</FONT><BR>
<BR>
<FONT COLOR="#000000">RHEL6.5/CMAN/gfs2/Pacemaker/crmsh</FONT><BR>
<BR>
<FONT COLOR="#000000">pacemaker-libs-1.1.10-14.el6_5.1.x86_64</FONT><BR>
<FONT COLOR="#000000">pacemaker-cli-1.1.10-14.el6_5.1.x86_64</FONT><BR>
<FONT COLOR="#000000">pacemaker-1.1.10-14.el6_5.1.x86_64</FONT><BR>
<FONT COLOR="#000000">pacemaker-cluster-libs-1.1.10-14.el6_5.1.x86_64</FONT><BR>
<BR>
<FONT COLOR="#000000">Two node HA VM cluster using real shared drive, not drbd.</FONT><BR>
<BR>
<FONT COLOR="#000000">Resources (relevant to this discussion):</FONT><BR>
<FONT COLOR="#000000">primitive p_fs_images ocf:heartbeat:Filesystem \</FONT><BR>
<FONT COLOR="#000000">primitive p_libvirtd lsb:libvirtd \</FONT><BR>
<FONT COLOR="#000000">primitive virt ocf:heartbeat:VirtualDomain \</FONT><BR>
<BR>
<FONT COLOR="#000000">services chkconfig on: cman, clvmd, pacemaker</FONT><BR>
<FONT COLOR="#000000">services chkconfig off: corosync, gfs2, libvirtd</FONT><BR>
<BR>
<FONT COLOR="#000000">Observation:</FONT><BR>
<BR>
<FONT COLOR="#000000">Rebooting the NON-host system results in the restart of the VM merrily running on the host system.</FONT><BR>
<BR>
<FONT COLOR="#000000">Apparent cause:</FONT><BR>
<BR>
<FONT COLOR="#000000">Upon startup, Pacemaker apparently checks the status of configured resources. However, the status request for the virt (ocf:heartbeat:VirtualDomain) resource fails with:</FONT><BR>
<BR>
<PRE>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: warning: child_timeout_callback: virt_monitor_0 process (PID 4158) timed out</FONT>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: warning: operation_finished: virt_monitor_0:4158 - timed out after 200000ms</FONT>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: notice: operation_finished: virt_monitor_0:4158:stderr [ error: Failed to reconnect to the hypervisor ]</FONT>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: notice: operation_finished: virt_monitor_0:4158:stderr [ error: no valid connection ]</FONT>
<FONT COLOR="#000000">Dec 18 12:19:30 [4147] mici-admin2 lrmd: notice: operation_finished: virt_monitor_0:4158:stderr [ error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory ]</FONT>
</PRE>
<FONT COLOR="#000000">This failure then snowballs into an "orphan" situation in which the running VM is restarted.</FONT><BR>
<BR>
<FONT COLOR="#000000">There was the suggestion of chkconfig on libvirtd (and presumably deleting the resource) so that the /var/run/libvirt/libvirt-sock has been created by service libvirtd. With libvirtd started by the system, there is no un-needed reboot of the VM.</FONT><BR>
<BR>
<FONT COLOR="#000000">However, it may be that removing libvirtd from Pacemaker control leaves the VM vdisk filesystem susceptible to corruption during a reboot induced failover.</FONT><BR>
<BR>
<FONT COLOR="#000000">Question:</FONT><BR>
<BR>
<FONT COLOR="#000000">Is there an accepted Pacemaker configuration such that the un-needed restart of the VM does not occur with the reboot of the non-host system?</FONT><BR>
<BR>
<FONT COLOR="#000000">Regards,</FONT><BR>
<FONT COLOR="#000000">Bob Haxo</FONT><BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BLOCKQUOTE>
<BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BLOCKQUOTE>
<FONT COLOR="#000000">_______________________________________________</FONT><BR>
<FONT COLOR="#000000">Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A></FONT><BR>
<FONT COLOR="#000000"><A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT><BR>
<BR>
<FONT COLOR="#000000">Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT><BR>
<FONT COLOR="#000000">Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT><BR>
<FONT COLOR="#000000">Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT><BR>
<BR>
</BLOCKQUOTE>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BLOCKQUOTE>
<BR>
<BR>
<BR>
<FONT COLOR="#888888">-- </FONT><BR>
<FONT COLOR="#888888">esta es mi vida e me la vivo hasta que dios quiera</FONT>
</BLOCKQUOTE>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<BR>
<BR>
<BR>
<FONT COLOR="#000000">-- </FONT><BR>
<FONT COLOR="#000000">esta es mi vida e me la vivo hasta que dios quiera</FONT>
</BLOCKQUOTE>
<BLOCKQUOTE TYPE=CITE>
<PRE>
<FONT COLOR="#000000">_______________________________________________</FONT>
<FONT COLOR="#000000">Pacemaker mailing list: <A HREF="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</A></FONT>
<FONT COLOR="#000000"><A HREF="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</A></FONT>
<FONT COLOR="#000000">Project Home: <A HREF="http://www.clusterlabs.org">http://www.clusterlabs.org</A></FONT>
<FONT COLOR="#000000">Getting started: <A HREF="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</A></FONT>
<FONT COLOR="#000000">Bugs: <A HREF="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</A></FONT>
</PRE>
</BLOCKQUOTE>
</BODY>
</HTML>