<div dir="ltr">Jan,<div><br></div><div> Very appreciated on your help, I am getting further more, but still it looks very strange.</div><div><br></div><div>1. To use "debug-promote", I upgrade pacemaker from 1.12 to 1.16, pcs to 0.9.160.</div><div><br></div><div>2. Recreate resource with below commands</div><div><div>pcs resource create ovndb_servers ocf:ovn:ovndb-servers \</div><div> master_ip=192.168.0.99 \</div><div> op monitor interval="10s" \</div><div> op monitor interval="11s" role=Master</div><div>pcs resource master ovndb_servers-master ovndb_servers \</div><div> meta notify="true" master-max="1" master-node-max="1" clone-max="3" clone-node-max="1"</div><div>pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.99 \</div><div> op monitor interval=10s</div><div>pcs constraint colocation add VirtualIP with master ovndb_servers-master \</div><div> score=INFINITY</div></div><div><br></div><div>3. pcs status</div><div><div> Master/Slave Set: ovndb_servers-master [ovndb_servers]</div><div> Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]</div><div> VirtualIP<span style="white-space:pre"> </span>(ocf::heartbeat:IPaddr2):<span style="white-space:pre"> </span>Stopped</div></div><div><br></div><div>4. Manually run 'debug-start' on 3 nodes and 'debug-promote' on one of nodes</div><div>run below on [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]<br></div><div># pcs resource debug-start ovndb_servers --full</div><div>run below on [ node-1.domain.tld ]</div><div># pcs resource debug-promote ovndb_servers --full</div><div><br></div><div>5. pcs status</div><div> Master/Slave Set: ovndb_servers-master [ovndb_servers]</div><div> Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]</div><div> VirtualIP<span style="white-space:pre"> </span>(ocf::heartbeat:IPaddr2):<span style="white-space:pre"> </span>Stopped</div><div><br></div><div>6. However I have seen that one of ovndb_servers has been indeed promoted as master, but pcs status still showed all 'stopped'</div><div>what am I missing?</div><div><br></div><div> > stderr: + 17:45:59: ocf_log:327: __OCF_MSG='ovndb_servers: Promoting node-1.domain.tld as the master'</div><div> > stderr: + 17:45:59: ocf_log:329: case "${__OCF_PRIO}" in</div><div> > stderr: + 17:45:59: ocf_log:333: __OCF_PRIO=INFO</div><div> > stderr: + 17:45:59: ocf_log:338: '[' INFO = DEBUG ']'</div><div> > stderr: + 17:45:59: ocf_log:341: ha_log 'INFO: ovndb_servers: Promoting node-1.domain.tld as the master'</div><div> > stderr: + 17:45:59: ha_log:253: __ha_log 'INFO: ovndb_servers: Promoting node-1.domain.tld as the master'</div><div> > stderr: + 17:45:59: __ha_log:185: local ignore_stderr=false</div><div> > stderr: + 17:45:59: __ha_log:186: local loglevel</div><div> > stderr: + 17:45:59: __ha_log:188: '[' 'xINFO: ovndb_servers: Promoting node-1.domain.tld as the master' = x--ignore-stderr ']'</div><div> > stderr: + 17:45:59: __ha_log:190: '[' none = '' ']'</div><div> > stderr: + 17:45:59: __ha_log:192: tty</div><div> > stderr: + 17:45:59: __ha_log:193: '[' x = x0 -a x = xdebug ']'</div><div> > stderr: + 17:45:59: __ha_log:195: '[' false = true ']'</div><div> > stderr: + 17:45:59: __ha_log:199: '[' '' ']'</div><div> > stderr: + 17:45:59: __ha_log:202: echo 'INFO: ovndb_servers: Promoting node-1.domain.tld as the master'</div><div> > stderr: INFO: ovndb_servers: Promoting node-1.domain.tld as the master</div><div> > stderr: + 17:45:59: __ha_log:204: return 0</div><div> > stderr: + 17:45:59: ovsdb_server_promote:378: /usr/sbin/crm_attribute --type crm_config --name OVN_REPL_INFO -s ovn_ovsdb_master_server -v node-1.domain.tld</div><div> > stderr: + 17:45:59: ovsdb_server_promote:379: ovsdb_server_master_update 8</div><div> > stderr: + 17:45:59: ovsdb_server_master_update:214: case $1 in</div><div> > stderr: + 17:45:59: ovsdb_server_master_update:218: /usr/sbin/crm_master -l reboot -v 10</div><div> > stderr: + 17:45:59: ovsdb_server_promote:380: return 0</div><div> > stderr: + 17:45:59: 458: rc=0</div><div> > stderr: + 17:45:59: 459: exit 0</div><div><br></div><div><div><br></div><div><pre style="white-space:pre-wrap;color:rgb(0,0,0)">On 23/11/17 23:52 +0800, Hui Xiang wrote:
><i> I am working on HA with 3-nodes, which has below configurations:
</i>><i>
</i>><i> """
</i>><i> pcs resource create ovndb_servers ocf:ovn:ovndb-servers \
</i>><i> master_ip=168.254.101.2 \
</i>><i> op monitor interval="10s" \
</i>><i> op monitor interval="11s" role=Master
</i>><i> pcs resource master ovndb_servers-master ovndb_servers \
</i>><i> meta notify="true" master-max="1" master-node-max="1" clone-max="3"
</i>><i> clone-node-max="1"
</i>><i> pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=168.254.101.2 \
</i>><i> op monitor interval=10s
</i>><i> pcs constraint order promote ovndb_servers-master then VirtualIP
</i>><i> pcs constraint colocation add VirtualIP with master ovndb_servers-master \
</i>><i> score=INFINITY
</i>><i> """
</i>
(Out of curiosity, this looks like a mix of output from
pcs config export pcs-commands [or clufter cib2pcscmd -s]
and manual editing. Is this a good guess?)</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">It's the output of "pcs status".
><i> However, after setting it as above, the master is not being selected, all
</i>><i> are stopped, from pacemaker log, node-1 has been chosen as the master, I am
</i>><i> confuse where is wrong, can anybody give a help, it would be very
</i>><i> appreciated.
</i>><i>
</i>><i>
</i>><i> Master/Slave Set: ovndb_servers-master [ovndb_servers]
</i>><i> Stopped: [ node-1.domain.tld node-2.domain.tld node-3.domain.tld ]
</i>><i> VirtualIP (ocf::heartbeat:IPaddr2): Stopped
</i>><i>
</i>><i>
</i>><i> # pacemaker log
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ /cib/configuration/resources: <primitive class="ocf"
</i>><i> id="ovndb_servers" provider="ovn" type="ovndb-servers"/>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ <instance_attributes
</i>><i> id="ovndb_servers-instance_attributes">
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ <nvpair
</i>><i> id="ovndb_servers-instance_attributes-master_ip" name="master_ip"
</i>><i> value="168.254.101.2"/>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ </instance_attributes>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ <operations>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ <op
</i>><i> id="ovndb_servers-start-timeout-30s" interval="0s" name="start"
</i>><i> timeout="30s"/>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ <op
</i>><i> id="ovndb_servers-stop-timeout-20s" interval="0s" name="stop"
</i>><i> timeout="20s"/>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ <op
</i>><i> id="ovndb_servers-promote-timeout-50s" interval="0s" name="promote"
</i>><i> timeout="50s"/>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ <op
</i>><i> id="ovndb_servers-demote-timeout-50s" interval="0s" name="demote"
</i>><i> timeout="50s"/>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ <op
</i>><i> id="ovndb_servers-monitor-interval-10s" interval="10s" name="monitor"/>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ <op
</i>><i> id="ovndb_servers-monitor-interval-11s-role-Master" interval="11s"
</i>><i> name="monitor" role="Master"/>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ </operations>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++ </primitive>
</i>><i>
</i>><i> Nov 23 23:06:03 [665249] node-1.domain.tld attrd: info:
</i>><i> attrd_peer_update: Setting master-ovndb_servers[node-1.domain.tld]: (null)
</i>><i> -> 5 from node-1.domain.tld
</i>
If it's probable your ocf:ovn:ovndb-servers agent in master mode can
run something like "attrd_updater -n master-ovndb_servers -U 5", then
it was indeed launched OK, and if it does not continue to run as
expected, there may be a problem with the agent itself.
<br></pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">no change.</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">You can try running "pcs resource debug-promote ovndb_servers --full"
to examine the executation details (assuming the agent responds to
OCF_TRACE_RA=1 environment variable, which is what shell-based
agents built on top ocf-shellfuncs sourcable shell library from
resource-agents project, hence incl. also agents it ships,
customarily do).</pre><pre style="white-space:pre-wrap;color:rgb(0,0,0)">Yes, thank, it's helpful.
><i> Nov 23 23:06:03 [665251] node-1.domain.tld crmd: notice:
</i>><i> process_lrm_event: Operation ovndb_servers_monitor_0: ok
</i>><i> (node=node-1.domain.tld, call=185, rc=0, cib-update=88, confirmed=true)
</i>><i> <29>Nov 23 23:06:03 node-1 crmd[665251]: notice: process_lrm_event:
</i>><i> Operation ovndb_servers_monitor_0: ok (node=node-1.domain.tld, call=185,
</i>><i> rc=0, cib-update=88, confirmed=true)
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: Diff: --- 0.630.2 2
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: Diff: +++ 0.630.3 (null)
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: + /cib: @num_updates=3
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_perform_op: ++
</i>><i> /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:
</i>><i> <nvpair id="status-1-master-ovndb_servers" name="master-ovndb_servers"
</i>><i> value="5"/>
</i>><i> Nov 23 23:06:03 [665246] node-1.domain.tld cib: info:
</i>><i> cib_process_request: Completed cib_modify operation for section status: OK
</i>><i> (rc=0, origin=node-3.domain.tld/attrd/80, version=0.630.3)
</i>
Also depends if there's anything interesting after this point...
--
Jan (Poki)</pre></div></div></div>