[Pacemaker] ccm returning with exit code 100 and system rebooting

Tue Jan 18 04:29:18 EST 2011

Hi,

On Tue, Jan 18, 2011 at 08:34:57AM +0530, akshay punja wrote:
> Please let me know if any one has solved this issue. CCM exiting with return
> code 100 and system rebooting

Either bad installation or some kind of security mechanism
preventing heartbeat/ccm from operating normally. 

For instance, this looks suspicious:

Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: ERROR: Unable to set scheduler parameters.: Operation not permitted

Thanks,

Dejan

> On Mon, Jan 17, 2011 at 1:29 PM, akshay punja <akshay.punja at gmail.com>wrote:
> 
> > Hi All,
> >
> > We am using pacemaker(pacemaker-1.0.9.1-1.15.el5.i386.rpm) with
> > heartbeat(heartbeat-3.0.3-2.3.el5.i386.rpm) for a production deployment.
> >
> > Node : we are using two node in a cluster and hosting a bunch of
> > application on the HA.
> >
> > We are seeing a strange rebooting of one of the nodes *Managed
> > /usr/lib/heartbeat/ccm process 22115 exited with return code 100. What could
> > be possible issue and how could we fix it.
> > *
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: yes
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Pacemaker support: false
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: WARN: Logging daemon is
> > disabled --enabling logging daemon is recommended
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info:
> > **************************
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17619]: info: Configuration validated.
> > Starting heartbeat 3.0.2
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: heartbeat: version 3.0.2
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Heartbeat generation:
> > 1293182645
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: write
> > socket priority set to IPTOS_LOWDELAY on eth0
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound send
> > socket to device: eth0
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: bound
> > receive socket to device: eth0
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: glib: ucast: started on
> > port 694 interface eth0 to 172.21.52.135
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
> > G_main_add_TriggerHandler: Added signal manual handler
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
> > G_main_add_TriggerHandler: Added signal manual handler
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info:
> > G_main_add_SignalHandler: Added signal handler for signal 17
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: ERROR: Unable to set scheduler
> > parameters.: Operation not permitted
> > Jan 17 07:50:38 mysqlis1 heartbeat: [17620]: info: Local status now set to:
> > 'up'
> > Jan 17 07:50:39 mysqlis1 heartbeat: [17627]: ERROR: Unable to set scheduler
> > parameters.: Operation not permitted
> > Jan 17 07:50:39 mysqlis1 heartbeat: [17629]: ERROR: Unable to set scheduler
> > parameters.: Operation not permitted
> > Jan 17 07:50:39 mysqlis1 heartbeat: [17628]: ERROR: Unable to set scheduler
> > parameters.: Operation not permitted
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: node mysql3: is dead
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Comm_now_up(): updating
> > status to active
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Local status now set to:
> > 'active'
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> > "/usr/lib/heartbeat/ccm" (100,101)
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> > "/usr/lib/heartbeat/cib" (100,101)
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> > "/usr/lib/heartbeat/lrmd -r" (0,0)
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> > "/usr/lib/heartbeat/stonithd" (0,0)
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> > "/usr/lib/heartbeat/attrd" (100,101)
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: info: Starting child client
> > "/usr/lib/heartbeat/crmd" (100,101)
> > Jan 17 07:52:39 mysqlis1 heartbeat: [19576]: info: Starting
> > "/usr/lib/heartbeat/ccm" as uid 100  gid 101 (pid 19576)
> > Jan 17 07:52:39 mysqlis1 heartbeat: [19577]: info: Starting
> > "/usr/lib/heartbeat/cib" as uid 100  gid 101 (pid 19577)
> > Jan 17 07:52:39 mysqlis1 heartbeat: [19578]: info: Starting
> > "/usr/lib/heartbeat/lrmd -r" as uid 0  gid 0 (pid 19578)
> > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
> > Added signal handler for signal 15
> > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
> > Added signal handler for signal 17
> > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: enabling coredumps
> > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
> > Added signal handler for signal 10
> > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: G_main_add_SignalHandler:
> > Added signal handler for signal 12
> > Jan 17 07:52:39 mysqlis1 lrmd: [19578]: info: Started.
> > Jan 17 07:52:39 mysqlis1 heartbeat: [19579]: info: Starting
> > "/usr/lib/heartbeat/stonithd" as uid 0  gid 0 (pid 19579)
> > Jan 17 07:52:39 mysqlis1 heartbeat: [19580]: info: Starting
> > "/usr/lib/heartbeat/attrd" as uid 100  gid 101 (pid 19580)
> > *Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: WARN: Managed
> > /usr/lib/heartbeat/ccm process 19576 exited with return code 100.
> > Jan 17 07:52:39 mysqlis1 heartbeat: [17620]: EMERG: Rebooting system.
> > Reason: /usr/lib/heartbeat/ccm*
> > Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler:
> > Added signal handler for signal 10
> > Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: G_main_add_SignalHandler:
> > Added signal handler for signal 12
> > Jan 17 07:52:39 mysqlis1 stonithd: [19579]: info: crm_cluster_connect:
> > Connecting to Heartbeat
> > Jan 17 07:52:39 mysqlis1 heartbeat: [19581]: info: Starting
> > "/usr/lib/heartbeat/crmd" as uid 100  gid 101 (pid 19581)
> > Jan 17 07:52:41 mysqlis1 heartbeat: [17620]: EMERG: ALL REBOOT OPTIONS
> > FAILED: /sbin/reboot -nf returned 0
> > Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: register_heartbeat_conn:
> > Cannot sign on with heartbeat:
> > Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR: failed to connect to
> > cluster
> > Jan 17 07:52:41 mysqlis1 stonithd: [19579]: ERROR:
> > /usr/lib/heartbeat/stonithd abnormally abort.
> > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown:
> > Master Control process died.
> > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17620 with
> > SIGTERM
> > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17628 with
> > SIGTERM
> > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Killing pid 17629 with
> > SIGTERM
> > Jan 17 07:52:42 mysqlis1 heartbeat: [17627]: CRIT: Emergency Shutdown(MCP
> > dead): Killing ourselves.*
> >
> > Regards,
> > Akshay
> >
> >
> > *

> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker