[Pacemaker] ccm returning with exit code 100 and system rebooting

[Andrew Beekhof]

On Tue, Jan 18, 2011 at 4:04 AM, akshay punja <akshay.punja at gmail.com> wrote:
> Please let me know if any one has solved this issue.

Can you try "crm respawn" instead of "crm on" so the node stays up
long enough to see why the ccm is unhappy.

Lars, you really aught to think about changing the default behavior
and adding "crm fatal" or something.

> CCM exiting with return code 100 and system rebooting
>
> On Mon, Jan 17, 2011 at 1:29 PM, akshay punja <akshay.punja at gmail.com>
> wrote:
>>
>> Hi All,
>>
>> We am using pacemaker(pacemaker-1.0.9.1-1.15.el5.i386.rpm) with
>> heartbeat(heartbeat-3.0.3-2.3.el5.i386.rpm) for a production deployment.
>>
>> Node : we are using two node in a cluster and hosting a bunch of
>> application on the HA.
>>
>> We are seeing a strange rebooting of one of the nodes Managed
>> /usr/lib/heartbeat/ccm process 22115 exited with return code 100. What could
>> be possible issue and how could we fix it.
>>
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: yes
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support:
>> false
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: WARN: Logging daemon is
>> disabled --enabling logging daemon is recommended
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info:
>> **************************
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Configuration
>> validated. Starting heartbeat 3.0.2
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: heartbeat: version
>> 3.0.2
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Heartbeat generation:
>> 1293182645
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: write
>> socket priority set to IPTOS_LOWDELAY on eth0
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound send
>> socket to device: eth0
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound
>> receive socket to device: eth0
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: started on
>> port 694 interface eth0 to 172.21.52.135
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
>> G_main_add_TriggerHandler: Added signal manual handler
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
>> G_main_add_TriggerHandler: Added signal manual handler
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
>> G_main_add_SignalHandler: Added signal handler for signal 17
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: ERROR: Unable to set
>> scheduler parameters.: Operation not permitted
>> Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Local status now set
>> to: 'up'
>> Jan 17 07:50:39 mysqlis1 heartbeat: [17627]: ERROR: Unable to set
>> scheduler parameters.: Operation not permitted
>> Jan 17 07:50:39 mysqlis1 heartbeat: [17629]: ERROR: Unable to set
>> scheduler parameters.: Operation not permitted
>> Jan 17 07:50:39 mysqlis1 heartbeat: [17628]: ERROR: Unable to set
>> scheduler parameters.: Operation not permitted
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: node mysql3: is dead
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Comm_now_up(): updating
>> status to active
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Local status now set
>> to: 'active'
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
>> "/usr/lib/heartbeat/ccm" (100,101)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
>> "/usr/lib/heartbeat/cib" (100,101)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
>> "/usr/lib/heartbeat/lrmd -r" (0,0)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
>> "/usr/lib/heartbeat/stonithd" (0,0)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
>> "/usr/lib/heartbeat/attrd" (100,101)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
>> "/usr/lib/heartbeat/crmd" (100,101)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19576]: info: Starting
>> "/usr/lib/heartbeat/ccm" as uid 100  gid 101 (pid 19576)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19577]: info: Starting
>> "/usr/lib/heartbeat/cib" as uid 100  gid 101 (pid 19577)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19578]: info: Starting
>> "/usr/lib/heartbeat/lrmd -r" as uid 0  gid 0 (pid 19578)
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
>> Added signal handler for signal 15
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
>> Added signal handler for signal 17
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: enabling coredumps
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
>> Added signal handler for signal 10
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
>> Added signal handler for signal 12
>> Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: Started.
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19579]: info: Starting
>> "/usr/lib/heartbeat/stonithd" as uid 0  gid 0 (pid 19579)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19580]: info: Starting
>> "/usr/lib/heartbeat/attrd" as uid 100  gid 101 (pid 19580)
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: Managed
>> /usr/lib/heartbeat/ccm process 19576 exited with return code 100.
>> Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: EMERG: Rebooting system.
>> Reason: /usr/lib/heartbeat/ccm
>> Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info:
>> G_main_add_SignalHandler: Added signal handler for signal 10
>> Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info:
>> G_main_add_SignalHandler: Added signal handler for signal 12
>> Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: crm_cluster_connect:
>> Connecting to Heartbeat
>> Jan 17 07:52:39 mysqlis1 heartbeat: [19581]: info: Starting
>> "/usr/lib/heartbeat/crmd" as uid 100  gid 101 (pid 19581)
>> Jan 17 07:52:41 mysqlis1 heartbeat: [17620]: EMERG: ALL REBOOT OPTIONS
>> FAILED: /sbin/reboot -nf returned 0
>> Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR:
>> register_heartbeat_conn: Cannot sign on with heartbeat:
>> Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: failed to connect to
>> cluster
>> Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR:
>> /usr/lib/heartbeat/stonithd abnormally abort.
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown:
>> Master Control process died.
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17620 with
>> SIGTERM
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17628 with
>> SIGTERM
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17629 with
>> SIGTERM
>> Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown(MCP
>> dead): Killing ourselves.
>>
>> Regards,
>> Akshay
>>
>>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs:
> http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker
>
>