<html><body>
<p>beekhof@gmail.com wrote on 04/28/2009 10:31:43 AM:<br>
><br>
> On Mon, Apr 27, 2009 at 22:25, Mark Hamzy <hamzy@us.ibm.com> wrote:<br>
> > beekhof@gmail.com wrote on 04/24/2009 11:00:01 AM:<br>
> >><br>
> >> On Thu, Apr 23, 2009 at 17:49, Mark Hamzy <hamzy@us.ibm.com> wrote:<br>
> >> ><br>
> > Not only does this has to be done for all of the resources, but new health<br>
> > metrics must<br>
> > be known to the administrators.<br>
> <br>
> I'd suggest that they should know this sort of thing - much like they<br>
> do with pingd.<br>
> <br>
> This is particularly true when there are multiple health devices as<br>
> the admin needs to decide how to combine the "scores" (and rules are<br>
> designed for exactly this sort of function).<br>
<br>
If you give administrators the choice of how to combine scores, its seems<br>
that there is another potential point of failure being introduced. Lucky<br>
admins will not see problems where unlucky admins will have to wait until<br>
their machines truly fail before HA moves resources over.<br>
<br>
How about a PE option for pacemaker to automatically calculate health?<br>
Admins could then enable these calculations if they do not want to go<br>
through the effort to investigate all of the health variables and how<br>
they might affect system operations?<br>
<br>
> <br>
> ><br>
> > This is a logistical nightmare. ?What I am proposing is that pacemaker add<br>
> > health<br>
> > scores to nodes. ?Currently, nodes with no rules applied to them start at<br>
> > zero.<br>
> ><br>
> > We want the constraints left alone.<br>
> <br>
> But it is a constraint that you want the system to observe.<br>
> Granted we may not expose it as a constraint (eg. migration-failure or<br>
> standby), but that doesn't change the fact that it is one.<br>
<br>
By setting a PE option, does this now reinforce the notion that constraints<br>
are being implicitly added?<br>
<br>
> <br>
> > <constraints><br>
> > ?<rsc_location id="rsc_location_apache_1" rsc="apache_1"><br>
> > ? ?<rule id="prefered_location_apache_1" score="100"><br>
> > ? ? ?<expression attribute="#uname" id="prefered_location_apache_1_expr"<br>
> > operation="eq" value="hs21c"/><br>
> > ? ?</rule><br>
> > ?</rsc_location><br>
> > </constraints><br>
> ><br>
> > How were you proposing that this should be done under the current<br>
> > rsc_location?<br>
> <br>
> In 1.0.x, rules are reusable, so in the worst case, each additional<br>
> resource "only" need have<br>
> <br>
> <rsc_location id="other_health" rsc="other"><br>
> <rule id-ref="health_location_1_apache_1"/><br>
> <rule id-ref="health_location_2_apache_1"/><br>
> </rsc_location><br>
<br>
Let me work out what it would look like for a system with two resources<br>
and two health variables. Currently, an admin would have to write out<br>
four constraints to weight certain resources on certain nodes. They would<br>
then have to add the two health variables to each resource weighting.<br>
<br>
<constraints><br>
<rsc_location id="prefered_location_apache_1" node="hs21c" rsc="apache_1" score="100"/><br>
<rsc_location id="health_apache_1" rsc="apache_1"><br>
<rule id="health_location_1_apache_1" score-attribute="#health-ipmi"><br>
<expression attribute="#health-ipmi" id="apache_1_ipmi_expr" operation="defined"/><br>
</rule><br>
<rule id="health_location_2_apache_1" score-attribute="#health-smart"><br>
<expression attribute="#health-smart" id="apache_1_smart_expr" operation="defined"/><br>
</rule><br>
</rsc_location><br>
<rsc_location id="prefered_location_nfs_1" node="hs21c" rsc="nfs_1" score="100"/><br>
<rsc_location id="health_nfs_1" rsc="nfs_1"><br>
<rule id="health_location_1_nfs_1" score-attribute="#health-ipmi"><br>
<expression attribute="#health-ipmi" id="nfs_1_ipmi_expr" operation="defined"/><br>
</rule><br>
<rule id="health_location_2_nfs_1" score-attribute="#health-smart"><br>
<expression attribute="#health-smart" id="nfs_1_smart_expr" operation="defined"/><br>
</rule><br>
</rsc_location><br>
</constraints><br>
<br>
This is a lot of work for N resources and M health variables.<br>
<br>
In 1.0.x, you can reference previous rule ids in other resources. So,<br>
you leave the first resource "as is" and then switch the longer health<br>
rules to health rules references in the other resources.<br>
<br>
<constraints><br>
<rsc_location id="prefered_location_apache_1" node="hs21c" rsc="apache_1" score="100"/><br>
<rsc_location id="health_apache_1" rsc="apache_1"><br>
<rule id="health_location_1" score-attribute="#health-ipmi"><br>
<expression attribute="#health-ipmi" id="apache_1_ipmi_expr" operation="defined"/><br>
</rule><br>
<rule id="health_location_2" score-attribute="#health-smart"><br>
<expression attribute="#health-smart" id="apache_1_smart_expr" operation="defined"/><br>
</rule><br>
</rsc_location><br>
<rsc_location id="prefered_location_nfs_1" node="hs21c" rsc="nfs_1" score="100"/><br>
<rsc_location id="health_nfs_1" rsc="nfs_1"><br>
<rule id-ref="health_location_1"/><br>
<rule id-ref="health_location_2"/><br>
</rsc_location><br>
</constraints><br>
<br>
While a little typing is saved, you still have to enter information<br>
for each N x M combination.<br>
<br>
> I've also been mulling over the idea of allowing one rule to match<br>
> multiple resources somehow...<br>
> Something like <rsc_location id="rsc_location_apache_3" rsc-pattern="*">...<br>
> <br>
> Which would again significantly reduce the configuration overhead.<br>
> <br>
> Is this starting to sound a little less aggravating?<br>
<br>
Does this mean that you would have to create a fake resource with<br>
health rules in it? Something like this:<br>
<br>
<constraints><br>
<rsc_location id="other_health" rsc="*"><br>
<rule id="health_location_1" score-attribute="#health-ipmi"><br>
<expression attribute="#health-ipmi" id="health_location_1_ipmi_expr" operation="defined"/><br>
</rule><br>
<rule id="health_location_2" score-attribute="#health-smart"><br>
<expression attribute="#health-smart" id="health_location_2_smart_expr" operation="defined"/><br>
</rule><br>
</rsc_location><br>
<rsc_location id="prefered_location_apache_1" node="hs21c" rsc="apache_1" score="100"/><br>
<rsc_location id="prefered_location_nfs_1" node="hs21c" rsc="nfs_1" score="100"/><br>
</constraints><br>
<br>
That is a little bit cleaner. The health calculations are consolidated<br>
and every resource does not have to bring them in. However, why should<br>
you have to list them all out if you don't want to? It could introduce<br>
errors if you forget one.<br>
<br>
> > lmb@suse.de wrote:<br>
> >> Basically your mechanism modifies the "base score" for a node, somewhat<br>
> >> similar to the cluster-wide default set by symmetric-cluster true (0) or<br>
> >> false (-INFINITY).<br>
> >><br>
> >> Sure, but I'd go even beyond this and just add the mechanism for setting<br>
> >> the base score; translating health scores into these can be done outside<br>
> >> the core system.<br>
> <br>
> Also an interesting way to think about it.<br>
> I could see this working if there was only ever one health agent per<br>
> node, but from what you're saying it sounds as if there is likely to<br>
> be many.<br>
> <br>
> Actually, it would still work if the entity responsible for updating<br>
> the node health combined the readings from the different sources into<br>
> a single value.<br>
> However, then you start to require a daemon and some way to configure<br>
> it (in order to specify how the sources should be combined).<br>
> And of course eventually people will want more detail than the<br>
> combined score... "why is the health red? ".<br>
<br>
Yes. There is really only one overall health status of a system. It<br>
can be summed from multiple, independent reporting mechanisms (ipmi,<br>
smart, mcelog, etc).<br>
<br>
I don't think that complex rules need to exist in order to combine<br>
scores. Any "red" health forces -INF. All "yellow" healths<br>
keep weighting the resource more and more to a different node. -INF<br>
plus anything equals -INF. So a simple merge_weights would work.<br>
<br>
I envision different daemons for each reporting interface. For example,<br>
if you system supports ipmi, then it will run a daemon that listens to<br>
ipmi events, determines if the event deals with system health, and<br>
then set a #health-ipmi variable to notify HA.<br>
<br>
With this architecture, you don't want another daemon summing up<br>
all #health-* variables that it sees. There would be a delay as an<br>
event needs to be reported; detected and summed; and then HA detecting<br>
the change and acting. Also, HA would detect changes to all of the<br>
sub-health variables and not really care about them. And what<br>
happens if the combining daemon dies?<br>
<br>
> And of course eventually people will want more detail than the<br>
> combined score... "why is the health red? ".<br>
<br>
Yes. I think adequate logging would work here. The ipmi daemon will<br>
notify the log of a fault that it detected as it sets the HA #health-ipmi<br>
variable for example.<br>
<br>
Mark</body></html>