<div dir="ltr"><div>Hi,</div><div><br></div><div>yes, by "service" I meant the apache-clone resource.</div><div><br></div><div>Maybe I can give a more stripped down and detailed example:</div><div><br></div><div><b>Given the following configuration:</b></div><div style="margin-left:40px">[root@pacemaker-test-1 cluster]# pcs cluster cib --config<br></div><div style="margin-left:40px"><div style="margin-left:40px"><configuration><br> <crm_config><br> <cluster_property_set id="cib-bootstrap-options"><br> <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/><br> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.23-1.el7_9.1-9acf116022"/><br> <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/><br> <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="pacemaker-test"/><br> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/><br> <nvpair id="cib-bootstrap-options-symmetric-cluster" name="symmetric-cluster" value="false"/><br> <nvpair id="cib-bootstrap-options-last-lrm-refresh" name="last-lrm-refresh" value="1628511747"/><br> </cluster_property_set><br> </crm_config><br> <nodes><br> <node id="1" uname="pacemaker-test-1"/><br> <node id="2" uname="pacemaker-test-2"/><br> </nodes><br> <resources><br> <clone id="apache-clone"><br> <primitive class="ocf" id="apache" provider="heartbeat" type="apache"><br> <instance_attributes id="apache-instance_attributes"><br> <nvpair id="apache-instance_attributes-port" name="port" value="80"/><br> <nvpair id="apache-instance_attributes-statusurl" name="statusurl" value="<a href="http://localhost/server-status" target="_blank">http://localhost/server-status</a>"/><br> </instance_attributes><br> <operations><br> <op id="apache-monitor-interval-10s" interval="10s" name="monitor" timeout="20s"/><br> <op id="apache-start-interval-0s" interval="0s" name="start" timeout="40s"/><br> <op id="apache-stop-interval-0s" interval="0s" name="stop" timeout="60s"/><br> </operations><br> </primitive><br> <meta_attributes id="apache-meta_attributes"><br> <nvpair id="apache-clone-meta_attributes-clone-max" name="clone-max" value="2"/><br> <nvpair id="apache-clone-meta_attributes-clone-node-max" name="clone-node-max" value="1"/><br> <nvpair id="apache-clone-meta_attributes-interleave" name="interleave" value="true"/><br> </meta_attributes><br> </clone><br> </resources><br> <constraints><br> <rsc_location id="location-apache-clone-pacemaker-test-1-100" node="pacemaker-test-1" rsc="apache-clone" score="100" resource-discovery="exclusive"/><br> <rsc_location id="location-apache-clone-pacemaker-test-2-0" node="pacemaker-test-2" rsc="apache-clone" score="0" resource-discovery="exclusive"/><br> </constraints><br> <rsc_defaults><br> <meta_attributes id="rsc_defaults-options"><br> <nvpair id="rsc_defaults-options-resource-stickiness" name="resource-stickiness" value="50"/><br> </meta_attributes><br> </rsc_defaults><br></configuration><br></div></div><div><br></div><div><br></div><div><b>With the cluster in a running state:</b></div><div><br></div><div style="margin-left:40px">[root@pacemaker-test-1 cluster]# pcs status<br></div><div style="margin-left:80px">Cluster name: pacemaker-test<br>Stack: corosync<br>Current DC: pacemaker-test-2 (version 1.1.23-1.el7_9.1-9acf116022) - partition with quorum<br>Last updated: Mon Aug 9 14:45:38 2021<br>Last change: Mon Aug 9 14:43:14 2021 by hacluster via crmd on pacemaker-test-1<br><br>2 nodes configured<br>2 resource instances configured<br><br>Online: [ pacemaker-test-1 pacemaker-test-2 ]<br><br>Full list of resources:<br><br> Clone Set: apache-clone [apache]<br> Started: [ pacemaker-test-1 pacemaker-test-2 ]<br><br>Daemon Status:<br> corosync: active/disabled<br> pacemaker: active/disabled<br> pcsd: active/enabled<br></div><div><br></div><div><b>When simulating an error by killing the <b>apache-resource </b>on pacemaker-test-1:</b></div><div><br></div><div style="margin-left:40px">[root@pacemaker-test-1 ~]# killall httpd</div><div style="margin-left:40px"><br></div><div><b>After a few seconds, the cluster notices that the <b>apache-resource</b> is down on pacemaker-test-1 and restarts it on pacemaker-test-1 (this is fine):</b></div><div><br></div><div style="margin-left:40px">[root@pacemaker-test-1 cluster]# cat corosync.log | grep crmd:<br></div><div style="margin-left:80px">Aug 09 14:49:30 [10336] pacemaker-test-1 crmd: info: process_lrm_event: Result of monitor operation for apache on pacemaker-test-1: 7 (not running) | call=12 key=apache_monitor_10000 confirmed=false cib-update=22<br>Aug 09 14:49:30 [10336] pacemaker-test-1 crmd: info: do_lrm_rsc_op: Performing key=3:4:0:0fe9a8dd-1a73-4770-a36e-b14a6bb37d68 op=apache_stop_0<br>Aug 09 14:49:30 [10336] pacemaker-test-1 crmd: info: process_lrm_event: Result of monitor operation for apache on pacemaker-test-1: Cancelled | call=12 key=apache_monitor_10000 confirmed=true<br>Aug 09 14:49:30 [10336] pacemaker-test-1 crmd: notice: process_lrm_event: Result of stop operation for apache on pacemaker-test-1: 0 (ok) | call=14 key=apache_stop_0 confirmed=true cib-update=24<br>Aug 09 14:49:32 [10336] pacemaker-test-1 crmd: info: do_lrm_rsc_op: Performing key=5:4:0:0fe9a8dd-1a73-4770-a36e-b14a6bb37d68 op=apache_start_0<br>Aug 09 14:49:33 [10336] pacemaker-test-1 crmd: notice: process_lrm_event: Result of start operation for apache on pacemaker-test-1: 0 (ok) | call=15 key=apache_start_0 confirmed=true cib-update=26<br>Aug 09 14:49:33 [10336] pacemaker-test-1 crmd: info: do_lrm_rsc_op: Performing key=6:4:0:0fe9a8dd-1a73-4770-a36e-b14a6bb37d68 op=apache_monitor_10000<br>Aug 09 14:49:34 [10336] pacemaker-test-1 crmd: info: process_lrm_event: Result of monitor operation for apache on pacemaker-test-1: 0 (ok) | call=16 key=apache_monitor_10000 confirmed=false cib-update=28<br></div><div><br></div><div><b>BUT the cluster also restarts the apache-resource on pacemaker-test-2. Which it should not do because the apache-resource on pacemaker-test-2 did not crash:</b></div><div><br></div><div style="margin-left:40px">[root@pacemaker-test-2 cluster]# cat corosync.log | grep crmd:</div><div style="margin-left:80px">Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: update_failcount: Updating failcount for apache on pacemaker-test-1 after failed monitor: rc=7 (update=value++, time=1628513370)<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: process_graph_event: Detected action (2.6) apache_monitor_10000.12=not running: failed<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: do_te_invoke: Processing graph 3 (ref=pe_calc-dc-1628513370-25) derived from /var/lib/pacemaker/pengine/pe-input-51.bz2<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: notice: abort_transition_graph: Transition aborted by status-1-fail-count-apache.monitor_10000 doing modify fail-count-apache#monitor_10000=2: Transient attribute change | cib=0.33.33 source=abort_unless_down:356 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-fail-count-apache.monitor_10000'] complete=false<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: abort_transition_graph: Transition aborted by status-1-last-failure-apache.monitor_10000 doing modify last-failure-apache#monitor_10000=1628513370: Transient attribute change | cib=0.33.34 source=abort_unless_down:356 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']/nvpair[@id='status-1-last-failure-apache.monitor_10000'] complete=false<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: notice: run_graph: Transition 3 (Complete=1, Pending=0, Fired=0, Skipped=2, Incomplete=9, Source=/var/lib/pacemaker/pengine/pe-input-51.bz2): Stopped<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1628513370-26) derived from /var/lib/pacemaker/pengine/pe-input-52.bz2<br><b>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: notice: te_rsc_command: Initiating stop operation apache_stop_0 locally on pacemaker-test-2 | action 4</b><br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: do_lrm_rsc_op: Performing key=4:4:0:0fe9a8dd-1a73-4770-a36e-b14a6bb37d68 op=apache_stop_0<br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: notice: te_rsc_command: Initiating stop operation apache_stop_0 on pacemaker-test-1 | action 3<br><b>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: process_lrm_event: Result of monitor operation for apache on pacemaker-test-2: Cancelled | call=12 key=apache_monitor_10000 confirmed=true</b><br>Aug 09 14:49:30 [18553] pacemaker-test-2 crmd: info: match_graph_event: Action apache_stop_0 (3) confirmed on pacemaker-test-1 (rc=0)<br><b>Aug 09 14:49:32 [18553] pacemaker-test-2 crmd: notice: process_lrm_event: Result of stop operation for apache on pacemaker-test-2: 0 (ok) | call=14 key=apache_stop_0 confirmed=true cib-update=50</b><br>Aug 09 14:49:32 [18553] pacemaker-test-2 crmd: info: match_graph_event: Action apache_stop_0 (4) confirmed on pacemaker-test-2 (rc=0)<br>Aug 09 14:49:32 [18553] pacemaker-test-2 crmd: notice: te_rsc_command: Initiating start operation apache_start_0 on pacemaker-test-1 | action 5<br><b>Aug 09 14:49:32 [18553] pacemaker-test-2 crmd: notice: te_rsc_command: Initiating start operation apache_start_0 locally on pacemaker-test-2 | action 7</b><br>Aug 09 14:49:32 [18553] pacemaker-test-2 crmd: info: do_lrm_rsc_op: Performing key=7:4:0:0fe9a8dd-1a73-4770-a36e-b14a6bb37d68 op=apache_start_0<br><b>Aug 09 14:49:32 [18553] pacemaker-test-2 crmd: notice: process_lrm_event: Result of start operation for apache on pacemaker-test-2: 0 (ok) | call=15 key=apache_start_0 confirmed=true cib-update=52</b><br>Aug 09 14:49:32 [18553] pacemaker-test-2 crmd: info: match_graph_event: Action apache_start_0 (7) confirmed on pacemaker-test-2 (rc=0)<br>Aug 09 14:49:32 [18553] pacemaker-test-2 crmd: notice: te_rsc_command: Initiating monitor operation apache_monitor_10000 locally on pacemaker-test-2 | action 8<br>Aug 09 14:49:32 [18553] pacemaker-test-2 crmd: info: do_lrm_rsc_op: Performing key=8:4:0:0fe9a8dd-1a73-4770-a36e-b14a6bb37d68 op=apache_monitor_10000<br>Aug 09 14:49:33 [18553] pacemaker-test-2 crmd: info: process_lrm_event: Result of monitor operation for apache on pacemaker-test-2: 0 (ok) | call=16 key=apache_monitor_10000 confirmed=false cib-update=54<br>Aug 09 14:49:33 [18553] pacemaker-test-2 crmd: info: match_graph_event: Action apache_monitor_10000 (8) confirmed on pacemaker-test-2 (rc=0)<br>Aug 09 14:49:33 [18553] pacemaker-test-2 crmd: info: match_graph_event: Action apache_start_0 (5) confirmed on pacemaker-test-1 (rc=0)<br>Aug 09 14:49:33 [18553] pacemaker-test-2 crmd: notice: te_rsc_command: Initiating monitor operation apache_monitor_10000 on pacemaker-test-1 | action 6<br>Aug 09 14:49:34 [18553] pacemaker-test-2 crmd: info: match_graph_event: Action apache_monitor_10000 (6) confirmed on pacemaker-test-1 (rc=0)<br>Aug 09 14:49:34 [18553] pacemaker-test-2 crmd: notice: run_graph: Transition 4 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-52.bz2): Complete<br>Aug 09 14:49:34 [18553] pacemaker-test-2 crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd<br>Aug 09 14:49:34 [18553] pacemaker-test-2 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd<br></div><div><br></div><div>So my questions are:</div><div>- Why is the apache-resource being restarted on pacemaker-test-2 in this scenario?</div><div>- Is it possible to configure the cluster to not restart the apache-resource on pacemaker-test-2 in this scenario?</div><div><br></div><div>Regards,</div><div><br></div><div>Andreas<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Mo., 9. Aug. 2021 um 14:14 Uhr schrieb Andrei Borzenkov <<a href="mailto:arvidjaar@gmail.com" target="_blank">arvidjaar@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Mon, Aug 9, 2021 at 3:07 PM Andreas Janning<br>
<<a href="mailto:andreas.janning@qaware.de" target="_blank">andreas.janning@qaware.de</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> I have just tried your suggestion by adding<br>
> <nvpair id="apache-clone-meta_attributes-interleave" name="interleave" value="true"/><br>
> to the clone configuration.<br>
> Unfortunately, the behavior stays the same. The service is still restarted on the passive node when crashing it on the active node.<br>
><br>
<br>
What is "service"? Is it the resource with id=apache-clone in your<br>
configuration?<br>
<br>
Logs from DC around time of crash would certainly be useful here.<br>
<br>
> Regards<br>
><br>
> Andreas<br>
><br>
> Am Mo., 9. Aug. 2021 um 13:45 Uhr schrieb Vladislav Bogdanov <<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>>:<br>
>><br>
>> Hi.<br>
>> I'd suggest to set your clone meta attribute 'interleaved' to 'true'<br>
>><br>
>> Best,<br>
>> Vladislav<br>
>><br>
>> On August 9, 2021 1:43:16 PM Andreas Janning <<a href="mailto:andreas.janning@qaware.de" target="_blank">andreas.janning@qaware.de</a>> wrote:<br>
>>><br>
>>> Hi all,<br>
>>><br>
>>> we recently experienced an outage in our pacemaker cluster and I would like to understand how we can configure the cluster to avoid this problem in the future.<br>
>>><br>
>>> First our basic setup:<br>
>>> - CentOS7<br>
>>> - Pacemaker 1.1.23<br>
>>> - Corosync 2.4.5<br>
>>> - Resource-Agents 4.1.1<br>
>>><br>
>>> Our cluster is composed of multiple active/passive nodes. Each software component runs on two nodes simultaneously and all traffic is routed to the active node via Virtual IP.<br>
>>> If the active node fails, the passive node grabs the Virtual IP and immediately takes over all work of the failed node. Since the software is already up and running on the passive node, there should be virtually no downtime.<br>
>>> We have tried achieved this in pacemaker by configuring clone-sets for each software component.<br>
>>><br>
>>> Now the problem:<br>
>>> When a software component fails on the active node, the Virtual-IP is correctly grabbed by the passive node. BUT the software component is also immediately restarted on the passive Node.<br>
>>> That unfortunately defeats the purpose of the whole setup, since we now have a downtime until the software component is restarted on the passive node and the restart might even fail and lead to a complete outage.<br>
>>> After some investigating I now understand that the cloned resource is restarted on all nodes after a monitoring failure because the default "on-fail" of "monitor" is restart. But that is not what I want.<br>
>>><br>
>>> I have created a minimal setup that reproduces the problem:<br>
>>><br>
>>>> <configuration><br>
>>>> <crm_config><br>
>>>> <cluster_property_set id="cib-bootstrap-options"><br>
>>>> <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/><br>
>>>> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.23-1.el7_9.1-9acf116022"/><br>
>>>> <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/><br>
>>>> <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="pacemaker-test"/><br>
>>>> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/><br>
>>>> <nvpair id="cib-bootstrap-options-symmetric-cluster" name="symmetric-cluster" value="false"/><br>
>>>> </cluster_property_set><br>
>>>> </crm_config><br>
>>>> <nodes><br>
>>>> <node id="1" uname="active-node"/><br>
>>>> <node id="2" uname="passive-node"/><br>
>>>> </nodes><br>
>>>> <resources><br>
>>>> <primitive class="ocf" id="vip" provider="heartbeat" type="IPaddr2"><br>
>>>> <instance_attributes id="vip-instance_attributes"><br>
>>>> <nvpair id="vip-instance_attributes-ip" name="ip" value="{{infrastructure.virtual_ip}}"/><br>
>>>> </instance_attributes><br>
>>>> <operations><br>
>>>> <op id="psa-vip-monitor-interval-10s" interval="10s" name="monitor" timeout="20s"/><br>
>>>> <op id="psa-vip-start-interval-0s" interval="0s" name="start" timeout="20s"/><br>
>>>> <op id="psa-vip-stop-interval-0s" interval="0s" name="stop" timeout="20s"/><br>
>>>> </operations><br>
>>>> </primitive><br>
>>>> <clone id="apache-clone"><br>
>>>> <primitive class="ocf" id="apache" provider="heartbeat" type="apache"><br>
>>>> <instance_attributes id="apache-instance_attributes"><br>
>>>> <nvpair id="apache-instance_attributes-port" name="port" value="80"/><br>
>>>> <nvpair id="apache-instance_attributes-statusurl" name="statusurl" value="<a href="http://localhost/server-status" rel="noreferrer" target="_blank">http://localhost/server-status</a>"/><br>
>>>> </instance_attributes><br>
>>>> <operations><br>
>>>> <op id="apache-monitor-interval-10s" interval="10s" name="monitor" timeout="20s"/><br>
>>>> <op id="apache-start-interval-0s" interval="0s" name="start" timeout="40s"/><br>
>>>> <op id="apache-stop-interval-0s" interval="0s" name="stop" timeout="60s"/><br>
>>>> </operations><br>
>>>> </primitive><br>
>>>> <meta_attributes id="apache-meta_attributes"><br>
>>>> <nvpair id="apache-clone-meta_attributes-clone-max" name="clone-max" value="2"/><br>
>>>> <nvpair id="apache-clone-meta_attributes-clone-node-max" name="clone-node-max" value="1"/><br>
>>>> </meta_attributes><br>
>>>> </clone><br>
>>>> </resources><br>
>>>> <constraints><br>
>>>> <rsc_location id="location-apache-clone-active-node-100" node="active-node" rsc="apache-clone" score="100" resource-discovery="exclusive"/><br>
>>>> <rsc_location id="location-apache-clone-passive-node-0" node="passive-node" rsc="apache-clone" score="0" resource-discovery="exclusive"/><br>
>>>> <rsc_location id="location-vip-clone-active-node-100" node="active-node" rsc="vip" score="100" resource-discovery="exclusive"/><br>
>>>> <rsc_location id="location-vip-clone-passive-node-0" node="passive-node" rsc="vip" score="0" resource-discovery="exclusive"/><br>
>>>> <rsc_colocation id="colocation-vip-apache-clone-INFINITY" rsc="vip" score="INFINITY" with-rsc="apache-clone"/><br>
>>>> </constraints><br>
>>>> <rsc_defaults><br>
>>>> <meta_attributes id="rsc_defaults-options"><br>
>>>> <nvpair id="rsc_defaults-options-resource-stickiness" name="resource-stickiness" value="50"/><br>
>>>> </meta_attributes><br>
>>>> </rsc_defaults><br>
>>>> </configuration><br>
>>><br>
>>><br>
>>><br>
>>> When this configuration is started, httpd will be running on active-node and passive-node. The VIP runs only on active-node.<br>
>>> When crashing the httpd on active-node (with killall httpd), passive-node immediately grabs the VIP and restarts its own httpd.<br>
>>><br>
>>> How can I change this configuration so that when the resource fails on active-node:<br>
>>> - passive-node immediately grabs the VIP (as it does now).<br>
>>> - active-node tries to restart the failed resource, giving up after x attempts.<br>
>>> - passive-node does NOT restart the resource.<br>
>>><br>
>>> Regards<br>
>>><br>
>>> Andreas Janning<br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> ________________________________<br>
>>><br>
>>> Beste Arbeitgeber ITK 2021 - 1. Platz für QAware<br>
>>> ausgezeichnet von Great Place to Work<br>
>>><br>
>>> ________________________________<br>
>>><br>
>>> Andreas Janning<br>
>>> Expert Software Engineer<br>
>>><br>
>>> QAware GmbH<br>
>>> Aschauer Straße 32<br>
>>> 81549 München, Germany<br>
>>> Mobil +49 160 1492426<br>
>>> <a href="mailto:andreas.janning@qaware.de" target="_blank">andreas.janning@qaware.de</a><br>
>>> <a href="http://www.qaware.de" rel="noreferrer" target="_blank">www.qaware.de</a><br>
>>><br>
>>> ________________________________<br>
>>><br>
>>> Geschäftsführer: Christian Kamm, Johannes Weigend, Dr. Josef Adersberger<br>
>>> Registergericht: München<br>
>>> Handelsregisternummer: HRB 163761<br>
>>><br>
>>> _______________________________________________<br>
>>> Manage your subscription:<br>
>>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
>>><br>
>>> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
>>><br>
>><br>
><br>
><br>
> --<br>
> ________________________________<br>
><br>
> Beste Arbeitgeber ITK 2021 - 1. Platz für QAware<br>
> ausgezeichnet von Great Place to Work<br>
><br>
> ________________________________<br>
><br>
> Andreas Janning<br>
> Expert Software Engineer<br>
><br>
> QAware GmbH<br>
> Aschauer Straße 32<br>
> 81549 München, Germany<br>
> Mobil +49 160 1492426<br>
> <a href="mailto:andreas.janning@qaware.de" target="_blank">andreas.janning@qaware.de</a><br>
> <a href="http://www.qaware.de" rel="noreferrer" target="_blank">www.qaware.de</a><br>
><br>
> ________________________________<br>
><br>
> Geschäftsführer: Christian Kamm, Johannes Weigend, Dr. Josef Adersberger<br>
> Registergericht: München<br>
> Handelsregisternummer: HRB 163761<br>
><br>
> _______________________________________________<br>
> Manage your subscription:<br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
><br>
> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div>
<hr style="height:1px;background-color:rgb(204,204,204);border:medium none" width="100%" size="1" align="center">
<div style="font-size:8pt;font-family:sans-serif">
<p style="margin:0pt 1pt">
<b>Beste Arbeitgeber ITK 2021 - 1. Platz für QAware</b><br>
ausgezeichnet von
<a href="https://www.qaware.de/news/platz-1-bei-beste-arbeitgeber-in-der-itk-2021/" rel="nofollow" target="_blank">Great Place to Work</a>
</p>
<hr style="height:1px;background-color:rgb(204,204,204);border:medium none" width="100%" size="1" align="center">
<p style="margin:0pt 1pt 8pt">
Andreas Janning<br>
Expert Software Engineer<br>
</p>
<p style="margin:0pt 1pt">
QAware GmbH<br>
Aschauer Straße 32<br>
81549 München, Germany<br>
Mobil <span>+49 160 1492426</span><br>
<a href="mailto:andreas.janning@qaware.de" rel="nofollow" target="_blank">andreas.janning@qaware.de</a><br>
<a href="https://www.qaware.de" rel="nofollow" target="_blank">www.qaware.de</a><br>
</p>
</div>
<hr style="height:1px;background-color:rgb(204,204,204);border:medium none" width="100%" size="1" align="center">
<div style="font-size:7pt;font-family:sans-serif">
<p style="margin:0pt 1pt 14pt">
Geschäftsführer: Christian Kamm, Johannes Weigend, Dr. Josef Adersberger<br>
Registergericht: München<br>
Handelsregisternummer: HRB 163761<br>
</p>
</div>
</div></div></div>