I fixed it. <div>Reason was in exportfs resource monitor. On startup it check was it already started:<div><div> # "grep -z" matches across newlines, which is necessary as</div><div> # exportfs output wraps lines for long export directory names</div>
<div> exportfs | grep -zqs "${OCF_RESKEY_directory}[[:space:]]*${OCF_RESKEY_clientspec}"</div><div>But after node reboot nfs-state files are not in clean state for a reason and it caused the monitor action to return 0 state that was a reason of failed cluster.</div>
<div><br></div><div>I fixed it by moving /var/lib/nfs to share, and i placed pipefs in /var/lib - so now exportfs, if /var/lib/nfs/etab isn't accessible, returns no exports on startup.</div><div><br></div><div>Filesystem resource is working as designed.</div>
<div><br></div><div>Thanks all for help!</div><br><div class="gmail_quote">2011/6/22 Aleksander Malaev <span dir="ltr"><<a href="mailto:amalaev@alt-lan.ru">amalaev@alt-lan.ru</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Also I've seen that resources started in parralel:<div><div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start ping:0#011(msk-nfs-gw01)</div><div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start ping:1#011(msk-nfs-gw02)</div>
<div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-fs:0#011(msk-nfs-gw01)</div><div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-fs:1#011(msk-nfs-gw02)</div><div>
Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-dlm:0#011(msk-nfs-gw01)</div><div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-dlm:1#011(msk-nfs-gw02)</div><div>
Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-o2cb:0#011(msk-nfs-gw01)</div>
<div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-o2cb:1#011(msk-nfs-gw02)</div><div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start st-null:0#011(msk-nfs-gw01)</div>
<div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start st-null:1#011(msk-nfs-gw02)</div><div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start portmap:0#011(msk-nfs-gw01)</div>
<div>
Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start nfs-kernel-server:0#011(msk-nfs-gw01)</div><div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start portmap:1#011(msk-nfs-gw02)</div>
<div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start nfs-kernel-server:1#011(msk-nfs-gw02)</div><div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-nfs2-ip#011(msk-nfs-gw02)</div>
<div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-nfs1-ip#011(msk-nfs-gw01)</div><div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-share:0#011(msk-nfs-gw01)</div>
<div>Jun 22 16:18:49 msk-nfs-gw01 pengine: [1907]: notice: LogActions: Start res-share:1#011(msk-nfs-gw02)</div><div><br></div><div>How can it be?</div><div><div></div><div class="h5"><br><div class="gmail_quote">2011/6/22 Aleksander Malaev <span dir="ltr"><<a href="mailto:amalaev@alt-lan.ru" target="_blank">amalaev@alt-lan.ru</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I created bug in bugzilla: <a href="http://developerbugs.linux-foundation.org/show_bug.cgi?id=2609" target="_blank">http://developerbugs.linux-foundation.org/show_bug.cgi?id=2609</a><div>
<br></div><div>It seems all is like I thought, Filesystem resource is returned success and pacemaker continue to start resources after it, but it have not mounted fs yet and exportfs failed on export operation.</div>
<div>In logs i can see:</div><div><div>Jun 22 16:18:51 msk-nfs-gw02 exportfs[1955]: ERROR: /media/media0/nfsroot/export1 does not exist or is not a directory </div>
<div>Jun 22 16:18:55 msk-nfs-gw02 exportfs[2042]: ERROR: /media/media0/nfsroot/export1 does not exist or is not a directory</div></div><div><div><div></div><div><br><div class="gmail_quote">2011/6/21 Dejan Muhamedagic <span dir="ltr"><<a href="mailto:dejanmm@fastmail.fm" target="_blank">dejanmm@fastmail.fm</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div></div><div>On Tue, Jun 21, 2011 at 08:30:39PM +0200, Pawel Warowny wrote:<br>
> Dnia Tue, 21 Jun 2011 16:23:07 +0200<br>
> Dejan Muhamedagic <<a href="mailto:dejanmm@fastmail.fm" target="_blank">dejanmm@fastmail.fm</a>> napisał(a):<br>
><br>
> Hi<br>
><br>
> Sorry to bother in this thread, but:<br>
><br>
> > If you need to do so (there's actually start-delay, but it<br>
> > should be deprecated),<br>
><br>
> I use start-delay for starting kvm virtualized guest one after another.<br>
> If they all start at once and in the background drbd is syncing, the<br>
> load is immense.<br>
><br>
> How can I achieve this without start-delay?<br>
<br>
</div></div>I think you cannot, not yet. There should be some support for<br>
this coming, but I'm not sure about the current plans.<br>
<br>
In this case the use of start-delay is fine. What I referred to<br>
was using it to make sure that the previous resource started.<br>
<br>
Thanks,<br>
<br>
Dejan<br>
<br>
> Best regards<br>
> --<br>
> Pawel Warowny<br>
<div><div></div><div><br>
<br>
<br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
<br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
</div></div></blockquote></div><br><br clear="all"><br></div></div>-- <br><div>С уважением,<br>Александр Малаев<br>+7-962-938-9323<br>
</div></div>
</blockquote></div><br><br clear="all"><br>-- <br>С уважением,<br>Александр Малаев<br>+7-962-938-9323<br>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br>С уважением,<br>Александр Малаев<br>+7-962-938-9323<br>
</div></div>