[ClusterLabs] Why shouldn't one store resource configuration in the CIB?

[Ken Gaillot]

On 04/13/2017 11:11 AM, Ferenc Wágner wrote:
> Hi,
> 
> I encountered several (old) statements on various forums along the lines
> of: "the CIB is not a transactional database and shouldn't be used as
> one" or "resource parameters should only uniquely identify a resource,
> not configure it" and "the CIB was not designed to be a configuration
> database but people still use it that way".  Sorry if I misquote these,
> I go by my memories now, I failed to dig up the links by a quick try.
> 
> Well, I've been feeling guilty in the above offenses for years, but it
> worked out pretty well that way which helped to suppress these warnings
> in the back of my head.  Still, I'm curious: what's the reason for these
> warnings, what are the dangers of "abusing" the CIB this way?
> /var/lib/pacemaker/cib/cib.xml is 336 kB with 6 nodes and 155 resources
> configured.  Old Pacemaker versions required tuning PCMK_ipc_buffer to
> handle this, but even the default is big enough nowadays (128 kB after
> compression, I guess).
> 
> Am I walking on thin ice?  What should I look out for?

That's a good question. Certainly, there is some configuration
information in most resource definitions, so it's more a matter of degree.

The main concerns I can think of are:

1. Size: Increasing the CIB size increases the I/O, CPU and networking
overhead of the cluster (and if it crosses the compression threshold,
significantly). It also marginally increases the time it takes the
policy engine to calculate a new state, which slows recovery.

2. Consistency: Clusters can become partitioned. If changes are made on
one or more partitions during the separation, the changes won't be
reflected on all nodes until the partition heals, at which time the
cluster will reconcile them, potentially losing one side's changes. I
suppose this isn't qualitatively different from using a separate
configuration file, but those tend to be more static, and failure to
modify all copies would be more obvious when doing them individually
rather than issuing a single cluster command.