[Pacemaker] placement-strategy=minimal - placing and logging

[Vladimir]

Hey everyone,

---
I built up a two node test setup using utilization attributes. In brief
it consists of:

node-1: provides cores="4"
node-2: provides cores="4"

res-dummy01 requires cores="1"
res-dummy02 requires cores="2"
res-dummy03 requires cores="3"
res-dummy04 requires cores="2"

---
All resources are only able to run if they are distributed in the right
combination. A working example could like:

res-dummy01 + res-dummy03 = require 4 cores in sum
res-dummy02 + res-dummy04 = require 4 cores in sum

---
My goal is to to build up a multi node setup (nodes >= 3). I want to
use utilization attributes for correct resource distribution since
cluster resources will require different amount of cpu cores. On os
level cgroups will restrict system resources. Via placement strategy
minimal I would like to pack resources on as few nodes as possible so
one node stays without resources. It should act as a kind of dedicated
failover node.

---
My issue: basing on the mentioned two node test setup the placing
strategy only works if the resources a started in the right order, e.g.
first res-dummy01 than res-dummy03 and so on. If I understood the
documentation
(https://github.com/ClusterLabs/pacemaker/blob/master/doc/Pacemaker_Explained/en-US/Ch-Utilization.txt)
correctly "the first eligible node listed in cib gets consumed first". 

Is there a possibility to configure it in another way?

Furthermore if there is a lack of configured cores and thus a resource
cannot be started I don't see any log messages or crm_mon output.

Is this the normal behaviour? 

Below I put some information about the used versions, my configs and
the only log messages I see when I try to start a resource and it can't
be started because of the lack of cores.



#####
# System information and configs:
#####

root at node-1:~
#> lsb_release -ir
Distributor ID:   Ubuntu
Release: 12.04
root at node-1:~
#> uname -r
3.2.0-38-generic
root at node-1:~
#> dpkg -l pacemaker | awk '/ii/{print $2,$3}'
pacemaker 1.1.6-2ubuntu3
root at node-1:~
#> crm configure show
node node-1 \
   utilization cores="4"
node node-2 \
   utilization cores="4"
primitive res-dummy01 ocf:heartbeat:Dummy \
   utilization cores="1" \
   meta target-role="Started"
primitive res-dummy02 ocf:heartbeat:Dummy \
   utilization cores="2" \
   meta target-role="Started"
primitive res-dummy03 ocf:heartbeat:Dummy \
   utilization cores="3" \
   meta target-role="Started"
primitive res-dummy04 ocf:heartbeat:Dummy \
   utilization cores="2" \
   meta target-role="Stopped"
property $id="cib-bootstrap-options" \
   dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \
   cluster-infrastructure="openais" \
   expected-quorum-votes="2" \
   stonith-enabled="false" \
   no-quorum-policy="ignore" \
   default-resource-stickiness="1000" \
   last-lrm-refresh="1362565374" \
   placement-strategy="minimal"
rsc_defaults $id="rsc-options" \
   migration-threshold="3"
root at node-1:~
#> crm configure ptest utilization | cat
Utilization information:
Original: node-1 capacity: cores=4
Original: node-2 capacity: cores=4
calculate_utilization: res-dummy02 utilization on node-1: cores=2
calculate_utilization: res-dummy01 utilization on node-1: cores=1
calculate_utilization: res-dummy03 utilization on node-2: cores=3
Remaining: node-1 capacity: cores=1
Remaining: node-2 capacity: cores=1
INFO: install graphviz to see a transition graph
root at node-1:~
#> crm_mon -fr1
============
Last updated: Thu Mar  7 15:19:17 2013
Last change: Thu Mar  7 15:17:43 2013 via cibadmin on node-1
Stack: openais
Current DC: node-1 - partition with quorum
Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
2 Nodes configured, 2 expected votes
4 Resources configured.
============

Online: [ node-1 node-2 ]

Full list of resources:

 res-dummy01   (ocf::heartbeat:Dummy): Started node-1
 res-dummy02   (ocf::heartbeat:Dummy): Started node-1
 res-dummy03   (ocf::heartbeat:Dummy): Started node-2
 res-dummy04   (ocf::heartbeat:Dummy): Stopped 

Migration summary:
* Node node-1: 
* Node node-2: 


#####
# Concrete situation:
#####

root at node-1:~
#> crm resource start res-dummy04

root at node-1:~
#> tailf /var/log/corosync/corosync.log:

Mar 07 15:22:35 node-1 cib: [7754]: info: cib:diff: - <cib
admin_epoch="0" epoch="181" num_updates="1" > Mar 07 15:22:35 node-1
cib: [7754]: info: cib:diff: -   <configuration > Mar 07 15:22:35
node-1 cib: [7754]: info: cib:diff: -     <resources > Mar 07 15:22:35
node-1 cib: [7754]: info: cib:diff: -       <primitive id="res-dummy04"
> Mar 07 15:22:35 node-1 cib: [7754]: info: cib:diff: -
> <meta_attributes id="res-dummy04-meta_attributes" > Mar 07 15:22:35
> node-1 cib: [7754]: info: cib:diff: -           <nvpair
> value="Stopped" id="res-dummy04-meta_attributes-target-role" />
Mar 07 15:22:35 node-1 cib: [7754]: info: cib:diff: -
</meta_attributes> Mar 07 15:22:35 node-1 cib: [7754]: info: cib:diff:
-       </primitive> Mar 07 15:22:35 node-1 cib: [7754]: info:
cib:diff: -     </resources> Mar 07 15:22:35 node-1 cib: [7754]: info:
cib:diff: -   </configuration> Mar 07 15:22:35 node-1 cib: [7754]:
info: cib:diff: - </cib> Mar 07 15:22:35 node-1 cib: [7754]: info:
cib:diff: + <cib epoch="182" num_updates="1" admin_epoch="0"
validate-with="pacemaker-1.2" crm_feature_set="3.0.5"
update-origin="node-1" update-client="cibadmin" cib-last-written="Thu
Mar  7 15:17:43 2013" have-quorum="1" dc-uuid="node-1" > Mar 07
15:22:35 node-1 cib: [7754]: info: cib:diff: +   <configuration > Mar
07 15:22:35 node-1 cib: [7754]: info: cib:diff: +     <resources > Mar
07 15:22:35 node-1 cib: [7754]: info: cib:diff: +       <primitive
class="ocf" id="res-dummy04" provider="heartbeat" type="Dummy" > Mar 07
15:22:35 node-1 cib: [7754]: info: cib:diff: +         <meta_attributes
id="res-dummy04-meta_attributes" > Mar 07 15:22:35 node-1 cib: [7754]:
info: cib:diff: +           <nvpair
id="res-dummy04-meta_attributes-target-role" name="target-role"
value="Started" /> Mar 07 15:22:35 node-1 cib: [7754]: info: cib:diff:
+         </meta_attributes> Mar 07 15:22:35 node-1 cib: [7754]: info:
cib:diff: +       </primitive> Mar 07 15:22:35 node-1 cib: [7754]:
info: cib:diff: +     </resources> Mar 07 15:22:35 node-1 cib: [7754]:
info: cib:diff: +   </configuration> Mar 07 15:22:35 node-1 cib:
[7754]: info: cib:diff: + </cib> Mar 07 15:22:35 node-1 cib: [7754]:
info: cib_process_request: Operation complete: op cib_replace for
section resources (origin=local/cibadmin/2, version=0.182.1): ok (rc=0)
Mar 07 15:22:35 node-1 crmd: [7758]: info: abort_transition_graph:
te_update_diff:124 - Triggered transition abort (complete=1, tag=diff,
id=(null), magic=NA, cib=0.182.1) : Non-status change Mar 07 15:22:35
node-1 crmd: [7758]: info: do_state_transition: State transition S_IDLE
-> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
origin=abort_transition_graph ] Mar 07 15:22:35 node-1 crmd: [7758]:
info: do_state_transition: All 2 cluster nodes are eligible to run
resources. Mar 07 15:22:35 node-1 crmd: [7758]: info: do_pe_invoke:
Query 448: Requesting the current CIB: S_POLICY_ENGINE Mar 07 15:22:35
node-1 crmd: [7758]: info: do_pe_invoke_callback: Invoking the PE:
query=448, ref=pe_calc-dc-1362666155-227, seq=179496, quorate=1 Mar 07
15:22:35 node-1 pengine: [7757]: notice: unpack_config: On loss of CCM
Quorum: Ignore Mar 07 15:22:35 node-1 pengine: [7757]: notice:
LogActions: Leave   res-dummy01  (Started node-1) Mar 07 15:22:35
node-1 pengine: [7757]: notice: LogActions: Leave   res-dummy02
(Started node-1) Mar 07 15:22:35 node-1 pengine: [7757]: notice:
LogActions: Leave   res-dummy03  (Started node-2) Mar 07 15:22:35
node-1 pengine: [7757]: notice: LogActions: Leave   res-dummy04
(Stopped) Mar 07 15:22:35 node-1 crmd: [7758]: info:
do_state_transition: State transition S_POLICY_ENGINE ->
S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE
origin=handle_response ] Mar 07 15:22:35 node-1 crmd: [7758]: info:
unpack_graph: Unpacked transition 39: 2 actions in 2 synapses Mar 07
15:22:35 node-1 crmd: [7758]: info: do_te_invoke: Processing graph 39
(ref=pe_calc-dc-1362666155-227) derived
from /var/lib/pengine/pe-input-293.bz2 Mar 07 15:22:35 node-1 crmd:
[7758]: info: te_pseudo_action: Pseudo action 3 fired and confirmed Mar
07 15:22:35 node-1 crmd: [7758]: info: te_pseudo_action: Pseudo action
2 fired and confirmed Mar 07 15:22:35 node-1 crmd: [7758]: info:
run_graph: ==================================================== Mar 07
15:22:35 node-1 crmd: [7758]: notice: run_graph: Transition 39
(Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-293.bz2): Complete Mar 07 15:22:35
node-1 crmd: [7758]: info: te_graph_trigger: Transition 39 is now
complete Mar 07 15:22:35 node-1 crmd: [7758]: info: notify_crmd:
Transition 39 status: done - <null> Mar 07 15:22:35 node-1 crmd:
[7758]: info: do_state_transition: State transition S_TRANSITION_ENGINE
-> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
] Mar 07 15:22:35 node-1 crmd: [7758]: info: do_state_transition:
Starting PEngine Recheck Timer Mar 07 15:22:35 node-1 pengine: [7757]:
notice: process_pe_message: Transition 39: PEngine Input stored
in: /var/lib/pengine/pe-input-293.bz2


(Node "node-2" does not get log entries on this action.)