[ClusterLabs] pacemaker_remoted XML parse error

[Ken Gaillot]

On 06/08/2016 06:14 AM, Narayanamoorthy Srinivasan wrote:
> I have a pacemaker cluster with two pacemaker remote nodes. Recently the
> remote nodes started throwing below errors and SDB started self-fencing.
> Appreciate if someone throws light on what could be the issue and the fix.
> 
> OS - SLES 12 SP1
> Pacemaker Remote version - pacemaker-remote-1.1.13-14.7.x86_64
> 
> 2016-06-08T14:11:46.009073+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
> error : AttValue: ' expected
> 2016-06-08T14:11:46.009314+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error:
> key="neutron-ha-tool_monitor_0" operation="monitor"
> crm-debug-origin="do_update_
> 2016-06-08T14:11:46.009443+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error:                          
>                                                      ^
> 2016-06-08T14:11:46.009567+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
> error : attributes construct error
> 2016-06-08T14:11:46.009697+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error:
> key="neutron-ha-tool_monitor_0" operation="monitor"
> crm-debug-origin="do_update_
> 2016-06-08T14:11:46.009824+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error:                          
>                                                      ^
> 2016-06-08T14:11:46.009948+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
> error : Couldn't find end of Start Tag lrm_rsc_op line 1
> 2016-06-08T14:11:46.010070+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error:
> key="neutron-ha-tool_monitor_0" operation="monitor"
> crm-debug-origin="do_update_
> 2016-06-08T14:11:46.010191+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error:                          
>                                                      ^
> 2016-06-08T14:11:46.010460+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
> error : Premature end of data in tag lrm_resource line 1
> 2016-06-08T14:11:46.010718+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error:
> key="neutron-ha-tool_monitor_0" operation="monitor"
> crm-debug-origin="do_update_
> 2016-06-08T14:11:46.010977+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error:                          
>                                                      ^
> 2016-06-08T14:11:46.011234+05:30 d18-fb-7b-18-f1-8e
> pacemaker_remoted[6190]:    error: XML Error: Entity: line 1: parser
> error : Premature end of data in tag lrm_resources line 1
> 
> 
> -- 
> Thanks & Regards
> Moorthy

This sounds like the network traffic between the cluster nodes and the
remote nodes is being corrupted. Have there been any network changes
lately? Switch/firewall/etc. equipment/settings? MTU?

You could try using a packet sniffer such as wireshark to see if the
traffic looks abnormal in some way. The payload is XML so it should be
more or less readable.