[Pacemaker] Pacemaker and DRBD In Virtual Machines

[Ted Young]

Good afternoon,

I recently asked a question regarding resource stickiness.  In particular, I
set up two nodes with DRBD.  San1 was initially set primary while san2 was
secondary.  When I brought san1 down, san2 was promoted to primary.
However, when I brought san1 back online, it was promoted back to primary.
Ideally, I wanted to keep san2 as primary to avoid the unnecessary work of
moving the resources back.  Well, I found the source of my problem and it
was definitely not pacemaker.  I wanted to document this in hopes that it
might save other people considering pacemaker some trouble during testing
(and it might help others on this list diagnose strange problems).

My test nodes were actually virtual machines.  In itself this would not
normally be a problem.  But, I was simulating a node failure by pausing the
virtual machine.  When I paused san1, it would be taken off-line and san2
would be promoted.  When I resumed san1, it would come back online as if
nothing happened.  All its resources were configured identically (DRBD
master, mounted filesystem).  This is the part that I was missing.

When pacemaker noticed this new node coming back online preconfigured, it
simply gave back control to san1.  Actually, pacemaker made the most
effective choice here.  Unfortunately, if you are not paying attention to
what you are doing, the choice looks like it is violating resource
stickiness.

I guess if it were not for the fact that san1 was paused, this would have
actually created a split brain.

So, this is obviously not one of my brightest moments.  Many of you may be
thinking "duh".  I just wanted to bring this to light in case anyone else
runs into it.

For what it is worth, one can test the setup properly by actually shutting
the virtual machine down (rather than pausing it).  When the node is brought
back online, it will have all the resources disabled.  This gives pacemaker
a chance to bring up DRBD in secondary mode.

Thank you everyone for your help,

Ted

 

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.clusterlabs.org/pipermail/pacemaker/attachments/20111108/3919cc5a/attachment.html>