[ClusterLabs] [Problem] The `attrd` process may encounter an IPC communication error, rendering it unable to communicate and triggering a node reboot.

[Chris Lumens]

> When these operations—stopping nodes and migrating resources—are repeated, the `attrd` process occasionally encounters the following error (though the issue is not consistently reproducible):
>   -  (pcmk__ipc_send_iov)    notice: Response 45877 to pid 4671 failed: Resource temporarily unavailable | bytes=xxx rc=-11 ipcs=xxxxx

Incidentally, 6fa18439 removes this logging message since it's handled 
elsewhere, which should drastically shorten this log file.

> Oct 24 07:14:30.196 rh102-03 pacemaker-controld  [4671] (do_state_transition)   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
> Oct 24 07:14:35.080 rh102-03 pacemaker-based     [4666] (cib_process_ping)      info: Reporting our current digest to rh102-03: a584dfda8bbde766eccd6586c5c6d4bd for 100000.111769.0
> Oct 24 07:15:03.108 rh102-03 pacemaker-controld  [4671] (pcmk__update_peer_expected)    info: handle_request: Node rh102-04[4] - expected state is now down (was member)
> Oct 24 07:15:03.108 rh102-03 pacemaker-controld  [4671] (handle_shutdown_request)       info: Creating shutdown request for rh102-04 (state=S_IDLE)
> Oct 24 07:15:03.108 rh102-03 pacemaker-attrd     [4669] (pcmk__ipc_send_iov)    notice: Response 45877 to pid 4671 failed: Resource temporarily unavailable | bytes=129 rc=-11 ipcs=0x563355afbed0
> Oct 24 07:15:03.108 rh102-03 pacemaker-attrd     [4669] (pcmk__ipc_send_iov)    notice: Response 45877 to pid 4671 failed: Resource temporarily unavailable | bytes=129 rc=-11 ipcs=0x563355afbed0
> (snip)
> Oct 24 07:15:38.074 rh102-03 pacemaker-attrd     [4669] (pcmk__ipc_send_iov)    notice: Response 45877 to pid 4671 failed: Resource temporarily unavailable | bytes=129 rc=-11 ipcs=0x563355afbed0

I think what's happening here is that attrd is so busy trying to send 
something to PID 4671 (pacemaker-controld) that it doesn't have time to 
respond to any other messages.  Thus, it can't respond to pacemakerd's 
"are you alive?" message, so pacemakerd thinks it's dead and kills it.

So the question is:  what's controld doing that's keeping it from 
responding to attrd?

The first thing that comes to mind to me is the bug fixed by 39f99f1b. 
Does the commit message on that seem like it might be relevant to your 
setup?  In particular, do you have a lot of resources or nodes that 
might be generating a lot of transitions all at once?

- Chris