[ClusterLabs Developers] problem with master score limited to 1000000

[Jehan-Guillaume de Rorthais]

Hi all,

We are writing a new resource agent for PostgreSQL (I am open to discuss why
offlist to keep the thread clean) and are experiencing some limitation regarding
to the master scoring in Pacemaker.

The only way in PostgreSQL to define which node should be promoted is to
compare their location in their transaction log (called LSN). This LSN is
expressed as a size that is obviously growing quickly. Eg., After creating a
fresh PostgerSQL instance, it is already at a few MB.

Unfortunately, we can not use this LSN as a master core because as soon as
it is superior to 1,000,000, Pacemaker consider it internally as "infinity". In
this case, all the slave nodes having the same score of "infinity", Pacemaker
take a decision out of our control, probably promoting the wrong slave, leading
to the cluster corruption from the PostgreSQL point of view.

Even if we keep track of the highest LSN during the monitor action to
subtract it from the current LSN on pre-promote it can quickly grow over 1MB
depending on the database activity or the monitor interval.

On pre-promote notification, we could probably :

  * abort the transition by setting -inf as master score to all the node to
    forbid Pacemaker to promote a random slave
  * all the node set their LSN as an attribute called "lsn" as instance

Each resource agent would then be able to compare their LSN to each others and
set a master if it has the best one.

But then, we are not sure how to trigger this second round of the election.
Waiting for a monitor action to trigger on all the nodes sounds like an
eternity to wait. Moreover, this sounds like the wrong place to call crm_master
as we don't know for sure all the other node had a monitor call before the
promotion is triggered.

Any idea or advice, comments ? Thanks in advance !

Regards,
-- 
Jehan-Guillaume de Rorthais
Dalibo
http://www.dalibo.com