<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Hi all,<div><br></div><div>I’m trying to use the crm history command in order to understand better why transitions are occurring and would appreciate some help using this tool. I am using crmsh version 4.3.1-ubuntu3 which is the highest available for my OS (Ubuntu 22.04.1 LTS). I can see that version 4.6 of crmsh is available for Ubuntu 24.04 and I’m a bit surprised to find that 22.04 is apparently no longer receiving updates - something I had noticed as well regarding pacemaker packages. Shouldn’t an LTS release still be receiving updates?</div><div><br></div><div>Anyway, see below for the commands I ran:</div><div><br></div><div>1. Use grep as recommended in <a href="https://clusterlabs.org/pacemaker/doc/2.1/Pacemaker_Administration/html/troubleshooting.html">https://clusterlabs.org/pacemaker/doc/2.1/Pacemaker_Administration/html/troubleshooting.html</a> </div><div><br></div><div><font face="Menlo">grep ’saving inputs’ /var/log/pacemaker/pacemaker.log</font></div><div><font face="Menlo">[…]</font></div><div><font face="Menlo">Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Promote DrbdData:0 ( Unpromoted -> Promoted cluster-nuc-c2 ) </font></div><div><font face="Menlo">Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Demote DrbdData:1 ( Promoted -> Unpromoted cluster-nuc-c1 ) </font></div><div><font face="Menlo">Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Move DrbdFS ( cluster-nuc-c1 -> cluster-nuc-c2 )</font></div><div><font face="Menlo">Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Move docker-services ( cluster-nuc-c1 -> cluster-nuc-c2 ) </font></div><div><font face="Menlo">Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Start virtual_ip ( cluster-nuc-c2 ) </font></div><div><font face="Menlo">Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (rsc_action_default) info: Leave PingChecks:0 (Started cluster-nuc-c2) </font></div><div><font face="Menlo">Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (rsc_action_default) info: Leave PingChecks:1 (Started cluster-nuc-c1) </font></div><div><font face="Menlo">Sep 10 11:18:37.890 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) notice: Calculated transition 505, saving inputs in /var/lib/pacemaker/pengine/pe-input-647.bz2 </font></div><div><font face="Menlo"><br></font></div><div>Okay so I want to find out more about this transition 505 with inputs in pe-input-647.</div><div><br></div><div>2. Run commands from <a href="https://clusterlabs.org/pacemaker/doc/2.1/Pacemaker_Administration/html/pcs-crmsh.html#log-analysis">https://clusterlabs.org/pacemaker/doc/2.1/Pacemaker_Administration/html/pcs-crmsh.html#log-analysis</a> - see below (I have added line breaks for better readability)</div><div><br></div><div><font face="Menlo">crm history </font></div><div><font face="Menlo"><br></font></div><div><div><font face="Menlo">crm(live/cluster-nuc-c1)history# peinputs</font></div><div><font face="Menlo">INFO: Retrieving information from cluster nodes, please wait...</font></div><div><br></div></div><div><font face="Menlo"># this is how the command is shown in the docs:</font></div><div><div><font face="Menlo">crm(live/cluster-nuc-c1)history# transition pe-input-647</font></div><div><font face="Menlo">INFO: Fetching updated logs from cluster nodes. Please wait...</font></div><div><font face="Menlo">/usr/lib/python3/dist-packages/parallax/manager.py:390: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used</font></div><div><font face="Menlo"> self.files[filename] = open(filename, 'wb', buffering=1)</font></div><div><font face="Menlo">ERROR: pe-input-647: path not found</font></div></div><div><br></div><div># tried with absolute path but same result</div><div><div><font face="Menlo">crm(live/cluster-nuc-c1)history# transition /var/lib/pacemaker/pengine/pe-input-647.tar.bz2</font></div><div><font face="Menlo">INFO: Fetching updated logs from cluster nodes. Please wait...</font></div><div><font face="Menlo">/usr/lib/python3/dist-packages/parallax/manager.py:390: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used</font></div><div><font face="Menlo"> self.files[filename] = open(filename, 'wb', buffering=1)</font></div><div><font face="Menlo">/usr/lib/python3/dist-packages/parallax/manager.py:390: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used</font></div><div><font face="Menlo"> self.files[filename] = open(filename, 'wb', buffering=1)</font></div><div><font face="Menlo">ERROR: /var/lib/pacemaker/pengine/pe-input-647.tar.bz2: path not found</font></div></div><div><font face="Menlo"><br></font></div><div>3. Running ‘transitions’ doesn’t show anything, despite at least 2 transitions having occurred within the last hour</div><div><br></div><div><div><font face="Menlo">crm(live/cluster-nuc-c1)history# transitions</font></div><div><font face="Menlo">Time Name Node Tags</font></div></div><div><font face="Menlo"><br></font></div><div>4. I also found the guide <a href="https://crmsh.github.io/history-guide/">https://crmsh.github.io/history-guide/</a> which led me to try some other commands:</div><div><br></div><div><font face="Menlo">crm report</font></div><div><font face="Menlo">[…]</font></div><div><div><font face="Menlo">INFO: cluster-nuc-c1# The report is saved in ./hb_report-Tue-10-Sep-2024.tar.bz2</font></div><div><font face="Menlo">INFO: cluster-nuc-c1# Report timespan: 09/10/24 02:06:00 - 09/10/24 14:06:43</font></div></div><div><font face="Menlo"><br></font></div><div><div><font face="Menlo">crm history</font></div><div><font face="Menlo">crm(live/cluster-nuc-c1)history# source ./hb_report-Tue-10-Sep-2024.tar.bz2</font></div><div><font face="Menlo">crm(live/cluster-nuc-c1)history# transitions</font></div><div><font face="Menlo">Time Name Node Tags</font></div></div><div><br></div><div>The time period covered by the report should include multiple transitions, so why does the transitions command not show anything?</div><div><br></div><div><font face="Menlo">grep 'saving inputs' /var/log/pacemaker/pacemaker.log</font></div><div><div><font face="Menlo">Sep 10 13:46:51.556 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 515, saving inputs in /var/lib/pacemaker/pengine/pe-input-649.bz2</font></div><div><font face="Menlo">Sep 10 13:49:54.844 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 516, saving inputs in /var/lib/pacemaker/pengine/pe-input-650.bz2</font></div><div><font face="Menlo">Sep 10 13:49:54.876 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 517, saving inputs in /var/lib/pacemaker/pengine/pe-input-651.bz2</font></div><div><font face="Menlo">Sep 10 13:49:55.096 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 518, saving inputs in /var/lib/pacemaker/pengine/pe-input-652.bz2</font></div><div><font face="Menlo">Sep 10 13:49:55.124 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 519, saving inputs in /var/lib/pacemaker/pengine/pe-input-653.bz2</font></div><div><font face="Menlo">Sep 10 13:50:04.840 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 520, saving inputs in /var/lib/pacemaker/pengine/pe-input-654.bz2</font></div><div><font face="Menlo">Sep 10 13:50:04.872 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 521, saving inputs in /var/lib/pacemaker/pengine/pe-input-655.bz2</font></div><div><font face="Menlo">Sep 10 13:50:05.112 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 522, saving inputs in /var/lib/pacemaker/pengine/pe-input-656.bz2</font></div><div><font face="Menlo">Sep 10 13:50:05.152 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 523, saving inputs in /var/lib/pacemaker/pengine/pe-input-657.bz2</font></div><div><font face="Menlo">Sep 10 13:50:12.868 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 524, saving inputs in /var/lib/pacemaker/pengine/pe-input-658.bz2</font></div><div><font face="Menlo">Sep 10 13:50:12.900 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 525, saving inputs in /var/lib/pacemaker/pengine/pe-input-659.bz2</font></div><div><font face="Menlo">Sep 10 13:50:13.124 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 526, saving inputs in /var/lib/pacemaker/pengine/pe-input-660.bz2</font></div><div><font face="Menlo">Sep 10 13:50:13.172 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 527, saving inputs in /var/lib/pacemaker/pengine/pe-input-661.bz2</font></div><div><font face="Menlo">Sep 10 13:50:26.228 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 528, saving inputs in /var/lib/pacemaker/pengine/pe-input-662.bz2</font></div><div><font face="Menlo">Sep 10 13:50:56.360 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 529, saving inputs in /var/lib/pacemaker/pengine/pe-input-663.bz2</font></div><div><font face="Menlo">Sep 10 13:50:56.512 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 530, saving inputs in /var/lib/pacemaker/pengine/pe-input-664.bz2</font></div><div><font face="Menlo">Sep 10 13:50:56.548 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 531, saving inputs in /var/lib/pacemaker/pengine/pe-input-665.bz2</font></div><div><font face="Menlo">Sep 10 13:51:06.704 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 532, saving inputs in /var/lib/pacemaker/pengine/pe-input-666.bz2</font></div><div><font face="Menlo">Sep 10 13:51:06.876 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 533, saving inputs in /var/lib/pacemaker/pengine/pe-input-667.bz2</font></div><div><font face="Menlo">Sep 10 13:51:06.908 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 534, saving inputs in /var/lib/pacemaker/pengine/pe-input-668.bz2</font></div><div><font face="Menlo">Sep 10 13:51:58.078 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 535, saving inputs in /var/lib/pacemaker/pengine/pe-input-669.bz2</font></div><div><font face="Menlo">Sep 10 13:52:07.463 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 536, saving inputs in /var/lib/pacemaker/pengine/pe-input-670.bz2</font></div><div><font face="Menlo">Sep 10 13:52:16.851 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 537, saving inputs in /var/lib/pacemaker/pengine/pe-input-671.bz2</font></div><div><font face="Menlo">Sep 10 13:52:26.207 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 538, saving inputs in /var/lib/pacemaker/pengine/pe-input-672.bz2</font></div><div><font face="Menlo">Sep 10 13:52:36.835 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 539, saving inputs in /var/lib/pacemaker/pengine/pe-input-673.bz2</font></div><div><font face="Menlo">Sep 10 14:07:50.063 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) <span class="Apple-tab-span" style="white-space:pre"> </span>notice: Calculated transition 540, saving inputs in /var/lib/pacemaker/pengine/pe-input-674.bz2</font></div></div><div><br></div><div><br></div><div>Any help and/or pointers to additional learning resources would be greatly appreciated.</div><div><br></div><div>Thanks and best regards,</div><div><br></div><div>Alex Eastwood</div><div><br></div></body></html>