[ClusterLabs] Help using crm history

Tue Sep 10 12:12:05 UTC 2024

Hi all,

I’m trying to use the crm history command in order to understand better why transitions are occurring and would appreciate some help using this tool. I am using crmsh version 4.3.1-ubuntu3 which is the highest available for my OS (Ubuntu 22.04.1 LTS). I can see that version 4.6 of crmsh is available for Ubuntu 24.04 and I’m a bit surprised to find that 22.04 is apparently no longer receiving updates - something I had noticed as well regarding pacemaker packages. Shouldn’t an LTS release still be receiving updates?

Anyway, see below for the commands I ran:

1. Use grep as recommended in https://clusterlabs.org/pacemaker/doc/2.1/Pacemaker_Administration/html/troubleshooting.html 

grep ’saving inputs’ /var/log/pacemaker/pacemaker.log
[…]
Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Promote DrbdData:0 ( Unpromoted -> Promoted cluster-nuc-c2 ) 
Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Demote DrbdData:1 ( Promoted -> Unpromoted cluster-nuc-c1 ) 
Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Move DrbdFS ( cluster-nuc-c1 -> cluster-nuc-c2 )
Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Move docker-services ( cluster-nuc-c1 -> cluster-nuc-c2 ) 
Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (log_list_item) notice: Actions: Start virtual_ip ( cluster-nuc-c2 ) 
Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (rsc_action_default) info: Leave PingChecks:0 (Started cluster-nuc-c2) 
Sep 10 11:18:37.878 cluster-nuc-c1 pacemaker-schedulerd[1010] (rsc_action_default) info: Leave PingChecks:1 (Started cluster-nuc-c1) 
Sep 10 11:18:37.890 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) notice: Calculated transition 505, saving inputs in /var/lib/pacemaker/pengine/pe-input-647.bz2 

Okay so I want to find out more about this transition 505 with inputs in pe-input-647.

2. Run commands from https://clusterlabs.org/pacemaker/doc/2.1/Pacemaker_Administration/html/pcs-crmsh.html#log-analysis - see below (I have added line breaks for better readability)

crm history 

crm(live/cluster-nuc-c1)history# peinputs
INFO: Retrieving information from cluster nodes, please wait...

# this is how the command is shown in the docs:
crm(live/cluster-nuc-c1)history# transition pe-input-647
INFO: Fetching updated logs from cluster nodes. Please wait...
/usr/lib/python3/dist-packages/parallax/manager.py:390: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.files[filename] = open(filename, 'wb', buffering=1)
ERROR: pe-input-647: path not found

# tried with absolute path but same result
crm(live/cluster-nuc-c1)history# transition /var/lib/pacemaker/pengine/pe-input-647.tar.bz2
INFO: Fetching updated logs from cluster nodes. Please wait...
/usr/lib/python3/dist-packages/parallax/manager.py:390: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.files[filename] = open(filename, 'wb', buffering=1)
/usr/lib/python3/dist-packages/parallax/manager.py:390: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
  self.files[filename] = open(filename, 'wb', buffering=1)
ERROR: /var/lib/pacemaker/pengine/pe-input-647.tar.bz2: path not found

3. Running ‘transitions’ doesn’t show anything, despite at least 2 transitions having occurred within the last hour

crm(live/cluster-nuc-c1)history# transitions
Time                            Name            Node            Tags

4. I also found the guide https://crmsh.github.io/history-guide/ which led me to try some other commands:

crm report
[…]
INFO: cluster-nuc-c1# The report is saved in ./hb_report-Tue-10-Sep-2024.tar.bz2
INFO: cluster-nuc-c1# Report timespan: 09/10/24 02:06:00 - 09/10/24 14:06:43

crm history
crm(live/cluster-nuc-c1)history# source ./hb_report-Tue-10-Sep-2024.tar.bz2
crm(live/cluster-nuc-c1)history# transitions
Time                            Name            Node            Tags

The time period covered by the report should include multiple transitions, so why does the transitions command not show anything?

grep 'saving inputs' /var/log/pacemaker/pacemaker.log
Sep 10 13:46:51.556 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 515, saving inputs in /var/lib/pacemaker/pengine/pe-input-649.bz2
Sep 10 13:49:54.844 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 516, saving inputs in /var/lib/pacemaker/pengine/pe-input-650.bz2
Sep 10 13:49:54.876 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 517, saving inputs in /var/lib/pacemaker/pengine/pe-input-651.bz2
Sep 10 13:49:55.096 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 518, saving inputs in /var/lib/pacemaker/pengine/pe-input-652.bz2
Sep 10 13:49:55.124 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 519, saving inputs in /var/lib/pacemaker/pengine/pe-input-653.bz2
Sep 10 13:50:04.840 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 520, saving inputs in /var/lib/pacemaker/pengine/pe-input-654.bz2
Sep 10 13:50:04.872 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 521, saving inputs in /var/lib/pacemaker/pengine/pe-input-655.bz2
Sep 10 13:50:05.112 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 522, saving inputs in /var/lib/pacemaker/pengine/pe-input-656.bz2
Sep 10 13:50:05.152 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 523, saving inputs in /var/lib/pacemaker/pengine/pe-input-657.bz2
Sep 10 13:50:12.868 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 524, saving inputs in /var/lib/pacemaker/pengine/pe-input-658.bz2
Sep 10 13:50:12.900 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 525, saving inputs in /var/lib/pacemaker/pengine/pe-input-659.bz2
Sep 10 13:50:13.124 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 526, saving inputs in /var/lib/pacemaker/pengine/pe-input-660.bz2
Sep 10 13:50:13.172 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 527, saving inputs in /var/lib/pacemaker/pengine/pe-input-661.bz2
Sep 10 13:50:26.228 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 528, saving inputs in /var/lib/pacemaker/pengine/pe-input-662.bz2
Sep 10 13:50:56.360 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 529, saving inputs in /var/lib/pacemaker/pengine/pe-input-663.bz2
Sep 10 13:50:56.512 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 530, saving inputs in /var/lib/pacemaker/pengine/pe-input-664.bz2
Sep 10 13:50:56.548 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 531, saving inputs in /var/lib/pacemaker/pengine/pe-input-665.bz2
Sep 10 13:51:06.704 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 532, saving inputs in /var/lib/pacemaker/pengine/pe-input-666.bz2
Sep 10 13:51:06.876 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 533, saving inputs in /var/lib/pacemaker/pengine/pe-input-667.bz2
Sep 10 13:51:06.908 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 534, saving inputs in /var/lib/pacemaker/pengine/pe-input-668.bz2
Sep 10 13:51:58.078 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 535, saving inputs in /var/lib/pacemaker/pengine/pe-input-669.bz2
Sep 10 13:52:07.463 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 536, saving inputs in /var/lib/pacemaker/pengine/pe-input-670.bz2
Sep 10 13:52:16.851 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 537, saving inputs in /var/lib/pacemaker/pengine/pe-input-671.bz2
Sep 10 13:52:26.207 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 538, saving inputs in /var/lib/pacemaker/pengine/pe-input-672.bz2
Sep 10 13:52:36.835 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 539, saving inputs in /var/lib/pacemaker/pengine/pe-input-673.bz2
Sep 10 14:07:50.063 cluster-nuc-c1 pacemaker-schedulerd[1010] (pcmk__log_transition_summary) 	notice: Calculated transition 540, saving inputs in /var/lib/pacemaker/pengine/pe-input-674.bz2

Any help and/or pointers to additional learning resources would be greatly appreciated.

Thanks and best regards,

Alex Eastwood

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20240910/d459bc7a/attachment.htm>