[Pacemaker] ERROR: Wrong stack o2cb

[Jake Smith]

----- Original Message -----
> From: "Denis Witt" <denis.witt at concepts-and-training.de>
> To: pacemaker at oss.clusterlabs.org
> Cc: "Jake Smith" <jsmith at argotec.com>
> Sent: Tuesday, June 25, 2013 11:47:36 AM
> Subject: Re: [Pacemaker] ERROR: Wrong stack o2cb
> 
> On Tue, 25 Jun 2013 11:37:15 -0400 (EDT)
> Jake Smith <jsmith at argotec.com> wrote:
> 
> > You probably already know but you're going to get failed "not
> > installed" from test4 always unless you install the same packages
> > there.
> > 
> > Do you have logs from test4-node[1|2] that are generating the not
> > installed for o2cb?  The log below is just from test4 if I'm not
> > mistaken which we expect doesn't have o2cb installed.
> 
> Hi Jake,
> 
> the log is from test4-node2, the machine was renamed and in the logs
> it
> still shows up as test4. It has o2cb installed. I can use the Drive
> fine on this machine when I start o2cb and ocfs2 by hand and mount
> the
> drive.
>  
> > A quick search for "ERROR: Wrong stack o2cb" indicates you may want
> > to verify o2cb isn't starting on boot?  But that's just a guess
> > without the logs from the affected nodes.
> 
> I've executed "update-rc.d o2cb disable" and "update-rc.d ocfs2
> disable". The services are stopped and pacemaker/corosync should
> handle
> everything. o2cb is still enabled in /etc/default/o2cb but the
> init-Script isn't executed on boot.
> 

This might help some - the second to last post:
http://comments.gmane.org/gmane.linux.highavailability.pacemaker/13918

I'll quote Bruno Macadre:

<quote>
I don't known if you solved your problem but I just have the same 
behavior on my fresh installed pacemaker.

	With the 2 lines :
p_o2cb:1_monitor_0 (node=nas1, call=10, rc=5,status=complete): not installed
p_o2cb:0_monitor_0 (node=nas2, call=10, rc=5, status=complete): not 
installed

	After some tries, I've found a bug in the resource agent ocf:pacemaker:o2cb

	When this agent start, his first action is to do a 'o2cb_monitor' to 
check if o2cb is already started. If not (ie $? == $OCF_NOT_RUNNING) it 
load all needed adn finally start.

	The bug is that 'o2cb_monitor' return $OCF_NOT_RUNNING if something is 
missing, except for the module 'ocfs2_user_stack' which he returns 
$OCF_ERR_INSTALLED. So if the module 'ocfs2_user_stack' is not loaded 
before starting ocf:pacemaker:o2cb resource agent it fails to start with 
'not installed' error.

	The workaround I've just find is to place 'ocfs2_user_stack' in my 
/etc/modules on all nodes and all works fine.

	I hope I helped someone and this bug was corrected in future release of 
o2cb RA.
</quote>

I took a look at the current RA and around lines 341-5 there is this check in the monitor code:
    grep user "$LOADED_PLUGINS_FILE" >/dev/null 2>&1; rc=$?
    if [ $rc != 0 ]; then
	ocf_log err "Wrong stack `cat $LOADED_PLUGINS_FILE`"
	return $OCF_ERR_INSTALLED
    fi

I'm guessing if you run:
grep user /sys/fs/ocfs2/loaded_cluster_plugins 2>&1; rc=$?

your going to return a 1 or something other than 0.

Also in the above thread the 4th to last post from Andreas Kurz @Hastexo mentions this:
     > This message was immediately followed by "Wrong stack" errors, and

     check the content of /sysfs/fs/ocfs2/loaded_cluster_plugins ... and if
     you have that configfile and it contains the value "user" this is a good
     sign you have started ocfs2/o2cb via init

HTH

Jake