<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:12.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">This is my first attempt to use Grafana with ClusterLabs HA dashboard. I got Grafana, Prometheus and Prometheus node_exporter to work and I am able to see those metrics. Next step was to make “ClusterLabs
HA Cluster details” Grafana dashboard to work and I am unable to make it work. Appreciate if you can point me in the right direction.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://grafana.com/grafana/dashboards/12229">https://grafana.com/grafana/dashboards/12229</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I am running Grafana on default 3000 port. Similarly using defaults for Prometheus. Prometheus node_exporter is using 9100.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I installed “ha_cluster_exporter” on all NFS-HA nodes (node1, node2 and corosync quorum (qdevice) node. I see they use
<span style="color:red">9664 </span>port. <o:p></o:p></span></p>
<p class="MsoNormal"><b><u><span style="font-size:11.0pt"><o:p><span style="text-decoration:none"> </span></o:p></span></u></b></p>
<p class="MsoNormal"><b><u><span style="font-size:11.0pt">nfs-server-1<o:p></o:p></span></u></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@nfs-server-1 ha_cluster_exporter]# systemctl status ha_cluster_exporter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">● ha_cluster_exporter.service - Prometheus exporter for Pacemaker HA clusters metrics<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Loaded: loaded (/usr/lib/systemd/system/ha_cluster_exporter.service; enabled; vendor preset: disabled)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Active: active (running) since Sun 2021-02-07 11:39:07 GMT; 1h 3min ago<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Main PID: 18547 (ha_cluster_expo)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> Memory: 6.7M<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> CGroup: /system.slice/ha_cluster_exporter.service<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> └─18547 /root/go/bin/ha_cluster_exporter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 11:39:07 nfs-server-1 systemd[1]: Started Prometheus exporter for Pacemaker HA clusters metrics.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 11:39:07 nfs-server-1 ha_cluster_exporter[18547]: time="2021-02-07T11:39:07Z" level=warning msg="Config File \"ha_cluster_exporter\" Not Found in \"[/ /.config /etc /usr/etc]\""<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 11:39:07 nfs-server-1 ha_cluster_exporter[18547]: time="2021-02-07T11:39:07Z" level=info msg="Default config values will be used"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 11:39:07 nfs-server-1 ha_cluster_exporter[18547]: time="2021-02-07T11:39:07Z" level=warning msg="Registration failure: could not initialize 'drbd' collector: '/sbin/drbdsetup' does not exist"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 11:39:07 nfs-server-1 ha_cluster_exporter[18547]: time="2021-02-07T11:39:07Z" level=info msg="'pacemaker' collector registered."<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 11:39:07 nfs-server-1 ha_cluster_exporter[18547]: time="2021-02-07T11:39:07Z" level=info msg="'corosync' collector registered."<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 11:39:07 nfs-server-1 ha_cluster_exporter[18547]: time="2021-02-07T11:39:07Z" level=info msg="'sbd' collector registered."<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 11:39:07 nfs-server-1 ha_cluster_exporter[18547]: time="2021-02-07T11:39:07Z" level=info msg="Serving metrics on 0.0.0.0:<span style="color:red">9664</span>"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@nfs-server-1 ha_cluster_exporter]#<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Similarly on <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">nfs-server-2 and qdevice node.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@nfs-server-2 ha_cluster_exporter]# systemctl status ha_cluster_exporter<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">…….<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">…..<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 12:15:10 nfs-server-2 ha_cluster_exporter[11895]: time="2021-02-07T12:15:10Z" level=warning msg="Registration failure: could not initialize 'drbd' collector: '/sbin/drbdsetup' does not exist"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 12:15:10 nfs-server-2 ha_cluster_exporter[11895]: time="2021-02-07T12:15:10Z" level=info msg="'pacemaker' collector registered."<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 12:15:10 nfs-server-2 ha_cluster_exporter[11895]: time="2021-02-07T12:15:10Z" level=info msg="'corosync' collector registered."<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 12:15:10 nfs-server-2 ha_cluster_exporter[11895]: time="2021-02-07T12:15:10Z" level=info msg="'sbd' collector registered."<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Feb 07 12:15:10 nfs-server-2 ha_cluster_exporter[11895]: time="2021-02-07T12:15:10Z" level=info msg="Serving metrics on 0.0.0.0:9664"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I copied this file <a href="https://github.com/ClusterLabs/ha_cluster_exporter/blob/master/dashboards/provider-sleha.yaml%20to%20/etc/grafana/provisioning/dashboards/%20and%20copied%20ha-cluster-details_rev2.json">
https://github.com/ClusterLabs/ha_cluster_exporter/blob/master/dashboards/provider-sleha.yaml to /etc/grafana/provisioning/dashboards/ and copied ha-cluster-details_rev2.json</a> file to /etc/grafana/dashboards/sleha as mentioned in manual steps section of
this page: <a href="https://github.com/ClusterLabs/ha_cluster_exporter/tree/master/dashboards">
https://github.com/ClusterLabs/ha_cluster_exporter/tree/master/dashboards</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I see these errors in Grafana log and the Grafana UI says “No data”.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">t=2021-02-07T12:51:18+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/1/api/v1/query_range remote_addr=10.0.0.2 referer="http://localhost:3000/d/Q5YJpwtZk1/clusterlabs-ha-cluster-details?orgId=1"
error="http: proxy error: context canceled"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">t=2021-02-07T12:51:18+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/api/v1/query_range status=502 remote_addr=10.0.0.2 time_ms=13
size=0 referer="http://localhost:3000/d/Q5YJpwtZk1/clusterlabs-ha-cluster-details?orgId=1"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">t=2021-02-07T12:51:48+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/1/api/v1/query remote_addr=10.0.0.2 referer="http://localhost:3000/d/Q5YJpwtZk1/clusterlabs-ha-cluster-details?orgId=1&var-DS_PROMETHEUS=Prometheus&var-cluster=nfs-ha&var-dc_instance="
error="http: proxy error: context canceled"<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">t=2021-02-07T12:51:48+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/api/v1/query status=502 remote_addr=10.0.0.2 time_ms=4 size=0
referer=<a href="http://localhost:3000/d/Q5YJpwtZk1/clusterlabs-ha-cluster-details?orgId=1&var-DS_PROMETHEUS=Prometheus&var-cluster=nfs-ha&var-dc_instance=">http://localhost:3000/d/Q5YJpwtZk1/clusterlabs-ha-cluster-details?orgId=1&var-DS_PROMETHEUS=Prometheus&var-cluster=nfs-ha&var-dc_instance=</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><img border="0" width="1623" height="758" style="width:16.9062in;height:7.8958in" id="Picture_x0020_1" src="cid:image001.png@01D6FD1B.4492BD20" alt="A screenshot of a computer
Description automatically generated with medium confidence"></span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><u><span style="font-size:11.0pt">This is my monitoring node running Grafana:
<o:p></o:p></span></u></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@client-1 log]# ls -l /etc/grafana/dashboards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">total 92<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">-rw-r--r--. 1 root root 94021 Feb 7 10:15 node-exporter.json<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">drwxr-xr-x. 2 root root 42 Feb 7 12:58 sleha<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@client-1 log]# ls -l /etc/grafana/dashboards/sleha/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">total 44<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">-rw-r--r--. 1 root root 41665 Feb 7 12:27 ha-cluster-details_rev2.json<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@client-1 log]#<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt">cat /etc/grafana/provisioning/dashboards/provider-sleha.yaml<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt">apiVersion: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">providers:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> - name: SUSE Linux Enterprise High Availability Extension<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> folder: SUSE Linux Enterprise<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> folderUid: 3b1e0b26-fc28-4254-88a1-2d3516b5e404<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> type: file<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> allowUiUpdates: true<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> editable: true<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> options:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> path: /etc/grafana/dashboards/sleha<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Copied file (<b>ha-cluster-details_rev2.json</b>) from here:
<a href="https://grafana.com/grafana/dashboards/12229/revisions">https://grafana.com/grafana/dashboards/12229/revisions</a> to /etc/grafana/dashboards/sleha<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt">cat /etc/grafana/provisioning/dashboards/node_exporter.yaml<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt">apiVersion: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">providers:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> - name: 'NFS HA Dashboard'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> type: file<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> updateIntervalSeconds: 10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> options:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> path: /etc/grafana/dashboards/node-exporter.json<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[root@client-1 log]#<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I added the <b>job_name: 'nfs-ha' section
</b>to /etc/prometheus/prometheus.yml , since the ClusterLabs HA Grafan dashboard says : “It is built on top of ha_cluster_exporter, but it also requires Prometheus node_exporter to be configured on the target nodes, and it also assumes that the
<b>target nodes in each cluster are grouped via the job label</b>.”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><a href="https://grafana.com/grafana/dashboards/12229">https://grafana.com/grafana/dashboards/12229</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> - job_name: 'quorum'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> # Override the global default and scrape targets from this job every 5 seconds.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> scrape_interval: 5s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> static_configs:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> - targets: ['qdevice.storage.nfs.oraclevcn.com:9100']<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> labels:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> group: 'quorum'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> - job_name: 'nfs_server'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> # Override the global default and scrape targets from this job every 5 seconds.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> scrape_interval: 5s<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> static_configs:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> - targets: ['nfs-server-1.storage.nfs.oraclevcn.com:9100', 'nfs-server-2.storage.nfs.oraclevcn.com:9100']<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> labels:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"> group: 'nfs_server'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"> - job_name: 'nfs-ha'<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"> scrape_interval: 5s<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"> static_configs:<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"> - targets: ['nfs-server-1.storage.nfs.oraclevcn.com:9100', 'nfs-server-2.storage.nfs.oraclevcn.com:9100', 'qdevice.storage.nfs.oraclevcn.com:9100']<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"> labels:<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt"> group: 'nfs-ha'<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Pinkesh Valdria<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Principal Solutions Architect – HPC<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>