[ClusterLabs] pacemaker_remoted XML parse error

Narayanamoorthy Srinivasan narayanamoorthys at gmail.com
Thu Jun 9 05:24:24 UTC 2016 

Don't see any issues in network traffic.

Some more logs where the XML tags are incomplete:

2016-06-09T03:06:03.096449+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
<lrm_rsc_op id="fs-postgresql_last_0" operation_key="fs-postgresql_stop_0"
operation="stop" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="225:116:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
transition-magic="0:0;225:116:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
on_node="d00-50-56-94-24-dd" call-id="489" rc-code="0" op-status="0"
interval="0" last-run="1459491026" last-rc-change="1459491026"
exec-time="158" queue-time="0" op-digest="dfb0c861
2016-06-09T03:06:03.097136+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
<lrm_rsc_op id="fs-postgresql_last_failure_0"
operation_key="fs-postgresql_monitor_0" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="41:4:7:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
transition-magic="0:0;41:4:7:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
on_node="d00-50-56-94-24-dd" call-id="5" rc-code="0" op-status="0"
interval="0" last-run="1459429072" last-rc-change="1459429072"
exec-time="315" queue-time="0" op-digest="df
2016-06-09T03:06:03.097361+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
<lrm_rsc_op id="fs-postgresql_monitor_10000"
operation_key="fs-postgresql_monitor_10000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="224:107:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
transition-magic="0:0;224:107:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
on_node="d00-50-56-94-24-dd" call-id="365" rc-code="0" op-status="0"
interval="10000" last-rc-change="1459490849" exec-time="185" queue-time="0"
op-digest="cd8d3642c
2016-06-09T03:06:03.097582+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
</lrm_resource>
2016-06-09T03:06:03.097690+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
<lrm_resource id="vip-admin-database-default-proposal-controller"
type="IPaddr2" class="ocf" provider="heartbeat">
2016-06-09T03:06:03.097797+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
<lrm_rsc_op id="vip-admin-database-default-proposal-controller_last_0"
operation_key="vip-admin-database-default-proposal-controller_stop_0"
operation="stop" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="228:116:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
transition-magic="0:0;228:116:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
on_node="d00-50-56-94-24-dd" call-id="487" rc-code="0" op-status="0"
interval="0" last-run="1459491026" last-rc-chan
2016-06-09T03:06:03.098013+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
<lrm_rsc_op
id="vip-admin-database-default-proposal-controller_monitor_10000"
operation_key="vip-admin-database-default-proposal-controller_monitor_10000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="227:107:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
transition-magic="0:0;227:107:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
on_node="d00-50-56-94-24-dd" call-id="369" rc-code="0" op-status="0"
interval="10000" last-rc-chang
2016-06-09T03:06:03.098230+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
</lrm_resource>
2016-06-09T03:06:03.098337+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
<lrm_resource id="postgresql" type="pgsql" class="ocf" provider="heartbeat">
2016-06-09T03:06:03.098468+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
<lrm_rsc_op id="postgresql_last_0" operation_key="postgresql_stop_0"
operation="stop" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.10"
transition-key="231:116:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
transition-magic="0:0;231:116:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
on_node="d00-50-56-94-24-dd" call-id="481" rc-code="0" op-status="0"
interval="0" last-run="1459491025" last-rc-change="1459491025"
exec-time="1334" queue-time="0" op-digest="f2317cad3d54c
2016-06-09T03:06:03.099061+05:30 d18-fb-7b-18-f1-8e
pacemaker_remoted[6153]:    error: Partial
<lrm_rsc_op id="postgresql_monitor_10000"
operation_key="postgresql_monitor_10000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.10"
transition-key="230:107:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
transition-magic="0:0;230:107:0:8fbf83fd-241b-4623-8bbe-31d92e4dfce1"
on_node="d00-50-56-94-24-dd" call-id="372" rc-code="0" op-status="0"
interval="10000" last-rc-change="1459490852" exec-time="424" queue-time="0"
op-digest="873ed4f07792aa8


On Wed, Jun 8, 2016 at 10:28 PM, Narayanamoorthy Srinivasan <
narayanamoorthys at gmail.com> wrote:

> No recent network changes. Will check for abnormal traffic using wireshark.
>
> I also notice that the XML lines are partial (no ending '>', closing " and
> sometimes partial words) in logs. Any lines > 472 characters are truncated
> to 472 characters. Wondering is it due to anyother limitations.
>
> I can post some line tomorrow when i am back to work.
>
>
> On Wed, Jun 8, 2016 at 8:00 PM, Ken Gaillot <kgaillot at redhat.com> wrote:
>
>> On 06/08/2016 06:14 AM, Narayanamoorthy Srinivasan wrote:
>> > I have a pacemaker cluster with two pacemaker remote nodes. Recently the
>> > remote nodes started throwing below errors and SDB started self-fencing.
>> > Appreciate if someone throws light on what could be the issue and the
>> fix.
>> >
>> > OS - SLES 12 SP1
>> > Pacemaker Remote version - pacemaker-remote-1.1.13-14.7.x86_64
>> >
>> > 2016-06-08T14:11:46.009073+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
>> > error : AttValue: ' expected
>> > 2016-06-08T14:11:46.009314+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error:
>> > key="neutron-ha-tool_monitor_0" operation="monitor"
>> > crm-debug-origin="do_update_
>> > 2016-06-08T14:11:46.009443+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error:
>> >                                                      ^
>> > 2016-06-08T14:11:46.009567+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
>> > error : attributes construct error
>> > 2016-06-08T14:11:46.009697+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error:
>> > key="neutron-ha-tool_monitor_0" operation="monitor"
>> > crm-debug-origin="do_update_
>> > 2016-06-08T14:11:46.009824+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error:
>> >                                                      ^
>> > 2016-06-08T14:11:46.009948+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
>> > error : Couldn't find end of Start Tag lrm_rsc_op line 1
>> > 2016-06-08T14:11:46.010070+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error:
>> > key="neutron-ha-tool_monitor_0" operation="monitor"
>> > crm-debug-origin="do_update_
>> > 2016-06-08T14:11:46.010191+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error:
>> >                                                      ^
>> > 2016-06-08T14:11:46.010460+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
>> > error : Premature end of data in tag lrm_resource line 1
>> > 2016-06-08T14:11:46.010718+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error:
>> > key="neutron-ha-tool_monitor_0" operation="monitor"
>> > crm-debug-origin="do_update_
>> > 2016-06-08T14:11:46.010977+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error:
>> >                                                      ^
>> > 2016-06-08T14:11:46.011234+05:30 d18-fb-7b-18-f1-8e
>> > pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
>> > error : Premature end of data in tag lrm_resources line 1
>> >
>> >
>> > --
>> > Thanks & Regards
>> > Moorthy
>>
>> This sounds like the network traffic between the cluster nodes and the
>> remote nodes is being corrupted. Have there been any network changes
>> lately? Switch/firewall/etc. equipment/settings? MTU?
>>
>> You could try using a packet sniffer such as wireshark to see if the
>> traffic looks abnormal in some way. The payload is XML so it should be
>> more or less readable.
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
>
>
> --
> Thanks & Regards
> Moorthy
>



-- 
Thanks & Regards
Moorthy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160609/0cdd74f4/attachment.htm>

More information about the Users mailing list