<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html40/strict.dtd">
<html><head><meta name="qrichtext" content="1" /><style type="text/css">
p, li { white-space: pre-wrap; }
</style></head><body style=" font-family:'Sans Serif'; font-size:10pt; font-weight:400; font-style:normal;">
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">On Wednesday, June 15, 2011 17:14:49 Dejan Muhamedagic wrote:</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> Hi,</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> On Wed, Jun 15, 2011 at 10:29:15PM +0200, Jelle de Jong wrote:</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > Hello everybody,</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > I was doing some testing/experiments with stonith:meatware using the</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > following configuration: http://paste.debian.net/119991/</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > question 1: does somebody know if I should add a pingd location to the</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > meatware stonith (see configuration)</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> No.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > question 2: I had my resources running on node hennessy, pulled out all</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > network cables to see what happens next...</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > The viktoriya node detects hennessy is gone and sends out the following</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > distress call:</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > info: client tengine [pid: 1776] requests a STONITH operation RESET on</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > node hennessy</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > info: stonith_operate_locally::2713: sending fencing op RESET for</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > hennessy to stonith_hennessy (meatware) (pid=8029)</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > CRIT: OPERATOR INTERVENTION REQUIRED to reset hennessy.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > CRIT: Run "meatclient -c hennessy" AFTER power-cycling the machine.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > So I am waiting and waiting until viktoriya starts its kvm guests but</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > nothing happens it just sits there... http://paste.debian.net/119984/</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > _until_ I executed meatclient -c hennessy (y) and the kvm guests started</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> > Is this what stonith:meatware is expected to do? I was hoping viktoriya</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> Yes. meatware is software run by meat (aka operator). It says so</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> above:</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> OPERATOR INTERVENTION REQUIRED to reset hennessy.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">> Funny but it looks fairly unequivocal to me.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Yes and no. The message is clear but unless you have someone sitting at a console 24/7 running tail on the log file, it has little value. According to the ClusterLabs stonith docs (which I just realized you wrote, haha):</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">"Whenever invoked, <span style=" font-family:'Courier New,courier';">meatware</span> logs a CRIT severity message which should show up on the node’s console."</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">While meatware seems to provide the fencing action as expected I've never seen any console messages either. There are at least 2 possible reasons I've found:</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">1) You need to check /etc/rsylog.conf (or equivalent) to see that .crit messages are indeed logged to the console. On Fedora 15 at least there is no special routing for .crit messages and they are sent to /var/log/messages by default. You might try inserting a line "*.crit /dev/console" although it didn't help me.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">2) Even though fencing works I still get lots of warn and error messages in the logs. Here's a sample:</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">[...]</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:34 JeffDesk stonith-ng: [560]: info: remote_op_done: Notifing clients of be940d4a-27d0-43a5-9d31-21b1fa8e761f (reboot of Server2.LAN from 8a883947-7022-4cad-9dba-d42110819bf9 by JeffDesk.LAN): 0, rc=1</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:34 JeffDesk stonith-ng: [560]: info: stonith_notify_client: Sending st_fence-notification to client 565/4ab08c7b-b795-404c-b1d6-dc2215a60248</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:34 JeffDesk crmd: [565]: ERROR: stonith_error2string: Unknown Stonith error code: 1</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:34 JeffDesk crmd: [565]: ERROR: tengine_stonith_notify: Peer Server2.LAN could not be terminated (reboot) by JeffDesk.LAN for Server4.LAN (ref=be940d4a-27d0-43a5-9d31-21b1fa8e761f): <unknown error></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:40 JeffDesk stonith-ng: [560]: info: stonith_queryQuery <stonith_command t="stonith-ng" st_async_id="62604f54-1777-45d5-b88e-9a8e53209e35" st_op="st_query" st_callid="0" st_callopt="0" st_remote_op="62604f54-1777-45d5-b88e-9a8e53209e35" st_target="Server2.LAN" st_device_action="reboot" st_clientid="8a883947-7022-4cad-9dba-d42110819bf9" st_timeout="6000" src="Server4.LAN" seq="80" /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:40 JeffDesk stonith-ng: [560]: info: can_fence_host_with_device: Meatware-Fence:0 can fence Server2.LAN: dynamic-list</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:40 JeffDesk stonith-ng: [560]: info: stonith_query: Found 1 matching devices for 'Server2.LAN'</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:40 JeffDesk stonith-ng: [560]: info: stonith_fenceExec <stonith_command t="stonith-ng" st_async_id="62604f54-1777-45d5-b88e-9a8e53209e35" st_op="st_fence" st_callid="0" st_callopt="0" st_remote_op="62604f54-1777-45d5-b88e-9a8e53209e35" st_target="Server2.LAN" st_device_action="reboot" st_timeout="54000" src="Server4.LAN" seq="82" /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:40 JeffDesk stonith-ng: [560]: info: can_fence_host_with_device: Meatware-Fence:0 can fence Server2.LAN: dynamic-list</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:40 JeffDesk stonith-ng: [560]: info: stonith_fence: Found 1 matching devices for 'Server2.LAN'</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:12:51 JeffDesk stonith: meatware device OK.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:21 JeffDesk stonith: meatware device OK.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:34 JeffDesk stonith-ng: [560]: WARN: 62604f54-1777-45d5-b88e-9a8e53209e35 process (PID 31288) timed out (try 1). Killing with signal SIGTERM (15).</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:34 JeffDesk stonith-ng: [560]: ERROR: log_operation: Operation 'reboot' [31288] for host 'Server2.LAN' with device 'Meatware-Fence:0' returned: 1 (call 0 from (null))</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:34 JeffDesk stonith-ng: [560]: info: process_remote_stonith_execExecResult <st-reply st_origin="stonith_construct_async_reply" t="stonith-ng" st_op="st_notify" st_remote_op="62604f54-1777-45d5-b88e-9a8e53209e35" st_callid="0" st_callopt="0" st_rc="1" st_output="Performing: stonith -t meatware -T reset Server2.LAN#012** INFO: parse config info info=JeffDesk.LAN Server2.LAN Server4.LAN#012#012** (process:31289): CRITICAL **: OPERATOR INTERVENTION REQUIRED to reset server2.lan.#012#012** (process:31289): CRITICAL **: Run "meatclient -c server2.lan" AFTER power-cycling the machine.#012failed: Server2.LAN 0.05859375#012" src="JeffDesk.LAN" seq="3" /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:34 JeffDesk stonith-ng: [560]: info: remote_op_done: Notifing clients of 62604f54-1777-45d5-b88e-9a8e53209e35 (reboot of Server2.LAN from 8a883947-7022-4cad-9dba-d42110819bf9 by JeffDesk.LAN): 0, rc=1</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:34 JeffDesk stonith-ng: [560]: info: stonith_notify_client: Sending st_fence-notification to client 565/4ab08c7b-b795-404c-b1d6-dc2215a60248</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:34 JeffDesk crmd: [565]: ERROR: stonith_error2string: Unknown Stonith error code: 1</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:34 JeffDesk crmd: [565]: ERROR: tengine_stonith_notify: Peer Server2.LAN could not be terminated (reboot) by JeffDesk.LAN for Server4.LAN (ref=62604f54-1777-45d5-b88e-9a8e53209e35): <unknown error></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:40 JeffDesk stonith-ng: [560]: info: stonith_queryQuery <stonith_command t="stonith-ng" st_async_id="fe6c7bcf-b444-408b-88d1-72abd3f13b21" st_op="st_query" st_callid="0" st_callopt="0" st_remote_op="fe6c7bcf-b444-408b-88d1-72abd3f13b21" st_target="Server2.LAN" st_device_action="reboot" st_clientid="8a883947-7022-4cad-9dba-d42110819bf9" st_timeout="6000" src="Server4.LAN" seq="83" /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:40 JeffDesk stonith-ng: [560]: info: can_fence_host_with_device: Refreshing port list for Meatware-Fence:0</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:40 JeffDesk stonith-ng: [560]: WARN: parse_host_line: Could not parse (0 2): ** INFO: parse config info info=JeffDesk.LAN Server2.LAN Server4.LAN</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:40 JeffDesk stonith-ng: [560]: WARN: parse_host_line: Could not parse (0 0): </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:40 JeffDesk stonith-ng: [560]: info: can_fence_host_with_device: Meatware-Fence:0 can fence Server2.LAN: dynamic-list</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:40 JeffDesk stonith-ng: [560]: info: stonith_query: Found 1 matching devices for 'Server2.LAN'</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:40 JeffDesk stonith-ng: [560]: info: stonith_fenceExec <stonith_command t="stonith-ng" st_async_id="fe6c7bcf-b444-408b-88d1-72abd3f13b21" st_op="st_fence" st_callid="0" st_callopt="0" st_remote_op="fe6c7bcf-b444-408b-88d1-72abd3f13b21" st_target="Server2.LAN" st_device_action="reboot" st_timeout="54000" src="Server4.LAN" seq="85" /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:40 JeffDesk stonith-ng: [560]: info: can_fence_host_with_device: Meatware-Fence:0 can fence Server2.LAN: dynamic-list</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:40 JeffDesk stonith-ng: [560]: info: stonith_fence: Found 1 matching devices for 'Server2.LAN'</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:13:51 JeffDesk stonith: meatware device OK.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:14:21 JeffDesk stonith: meatware device OK.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:14:34 JeffDesk stonith-ng: [560]: WARN: fe6c7bcf-b444-408b-88d1-72abd3f13b21 process (PID 31469) timed out (try 1). Killing with signal SIGTERM (15).</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:14:34 JeffDesk stonith-ng: [560]: ERROR: log_operation: Operation 'reboot' [31469] for host 'Server2.LAN' with device 'Meatware-Fence:0' returned: 1 (call 0 from (null))</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Jun 15 18:14:34 JeffDesk stonith-ng: [560]: info: process_remote_stonith_execExecResult <st-reply st_origin="stonith_construct_async_reply" t="stonith-ng" st_op="st_notify" st_remote_op="fe6c7bcf-b444-408b-88d1-72abd3f13b21" st_callid="0" st_callopt="0" st_rc="1" st_output="Performing: stonith -t meatware -T reset Server2.LAN#012** INFO: parse config info info=JeffDesk.LAN Server2.LAN Server4.LAN#012#012** (process:31470): CRITICAL **: OPERATOR INTERVENTION REQUIRED to reset server2.lan.#012#012** (process:31470): CRITICAL **: Run "meatclient -c server2.lan" AFTER power-cycling the machine.#012failed: Server2.LAN 0.05859375#012" src="JeffDesk.LAN" seq="5" /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">[...]</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">These messages repeat until I release the fence with meatclient.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br /></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Thanks, Jeff</p></body></html>