[ClusterLabs] The proxy server received an invalid response from an upstream server.
Jason Long
hack3rcon at yahoo.com
Wed Mar 17 16:07:49 EDT 2021
Thanks.
The 192.168.1.4 is my secondary VM.
I want to follow "https://www.digitalocean.com/community/tutorials/how-to-set-up-an-apache-active-passive-cluster-using-pacemaker-on-centos-7" tutorial.
At "Step 8 — Adding the Apache Resource", I got below problems:
1- In the tutorial example, you will see:
Full list of resources:
* Cluster_VIP (ocf::heartbeat:IPaddr2): Started webnode01
* WebServer (ocf::heartbeat:apache): Started webnode02
But mine:
Full List of Resources:
* Cluster_VIP (ocf::heartbeat:IPaddr2): Started node1
* WebServer (ocf::heartbeat:apache): Starting node2
Please be careful about "Started" and "Starting" words.
2- I wanted to restart the Apache resource, but:
[root at node1 log]# sudo pcs resource restart WebServer
Error: crm_resource: Error performing operation: Timer expired
Set 'WebServer' option: id=WebServer-meta_attributes-target-role set=WebServer-meta_attributes name=target-role value=stopped
Waiting for 1 resources to stop:
* WebServer
Deleted 'WebServer' option: id=WebServer-meta_attributes-target-role name=target-role
Waiting for 1 resources to start again:
* WebServer
Could not complete restart of WebServer, 1 resources remaining
* WebServer
Logs are:
https://paste.ubuntu.com/p/nHfTRFh4RD/
Why?
On Wednesday, March 17, 2021, 11:26:03 PM GMT+3:30, Reid Wahl <nwahl at redhat.com> wrote:
On Wed, Mar 17, 2021 at 11:51 AM Jason Long <hack3rcon at yahoo.com> wrote:
> Hello,
> I changed "IP" to my Apache web server:
>
> $ sudo pcs resource update floating_ip ocf:heartbeat:IPaddr2 ip=192.168.1.4 cidr_netmask=24 op monitor interval=5s
>
> And did:
>
> $ sudo pcs status
> Cluster name: mycluster
> Cluster Summary:
> * Stack: corosync
> * Current DC: node1 (version 2.0.5-10.fc33-ba59be7122) - partition with quorum
> * Last updated: Wed Mar 17 21:55:58 2021
> * Last change: Wed Mar 17 21:55:02 2021 by root via cibadmin on node1
> * 2 nodes configured
> * 2 resource instances configured
>
> Node List:
> * Online: [ node1 node2 ]
> Full List of Resources:
> * floating_ip (ocf::heartbeat:IPaddr2): Started node1
> * http_server (ocf::heartbeat:apache): Stopped
>
> Failed Resource Actions:
> * http_server_start_0 on node1 'error' (1): call=10, status='Timed Out', exitreason='', last-rc-change='2021-03-17 21:50:31 +03:30', queued=0ms, exec=40002ms
> * http_server_start_0 on node2 'error' (1): call=11, status='Timed Out', exitreason='', last-rc-change='2021-03-17 21:51:11 +03:30', queued=0ms, exec=40002ms
>
> Daemon Status:
> corosync: active/enabled
> pacemaker: active/enabled
> pcsd: active/enabled
>
>
> Why "http_server (ocf::heartbeat:apache): Stopped" ?
Like Ken said, the apache agent works by using wget to request the status URL. If this fails, then it will request the HTTP header of the index page.
You can run `pcs resource disable http_server` and then `pcs resource debug-start http_server --full` to get more detail about where the start operation is hanging. This likely needs to be done on the node where the floating IP resource is running, but you're welcome to try it on both nodes.
>
> I think you misunderstand my goal, please examine "https://paste.ubuntu.com/p/Nx2ptqZjFg/". I just have one Apache server and two Reverse Proxy servers, when a Reverse Proxy server stopped then another one work.
> In this scenario, is group resources mandatory?
>
Does the 192.168.1.4 IP address need to be on the same machine as the apache resource, or on a different machine?
>
>
>
>
> On Wednesday, March 17, 2021, 01:50:35 AM GMT+3:30, Reid Wahl <nwahl at redhat.com> wrote:
>
>
>
>
>
>
>
> On Tue, Mar 16, 2021 at 3:13 PM Jason Long <hack3rcon at yahoo.com> wrote:
>> I'm using CentOS.
>
> Ah okay. I think I had made an assumption based on the pastebins URLs.
>
>> Thus, I must use my Apache web server IP instead of node2?
>
> Yes, it's never a good idea to configure a node's constant IP address within an IPaddr2 resource. That will almost inevitably result in Pacemaker taking down the IP address at some point.
>
> For an IPaddr2 resource, you configure the IP address that's free to move around the cluster. In this case, that's the Apache web server IP. Node2's IP address isn't free to move to node1.
>
>> About resource group, is you mean "https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Pacemaker_Explained/index.html#group-resources" ?
>
> Yes, that's correct. And if you have access to the Red Hat docs, you can also refer to the following:
> - Chapter 5. Configuring an active/passive Apache HTTP server in a Red Hat High Availability cluster (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_configuring-active-passive-http-server-in-a-cluster-configuring-and-managing-high-availability-clusters)
> - Configuring resource groups (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_high_availability_clusters/assembly_configuring-cluster-resources-configuring-and-managing-high-availability-clusters#assembly_resource-groups-configuring-cluster-resources)
>
>>
>>
>>
>>
>>
>>
>> On Wednesday, March 17, 2021, 01:10:33 AM GMT+3:30, Reid Wahl <nwahl at redhat.com> wrote:
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Mar 16, 2021 at 1:47 PM Jason Long <hack3rcon at yahoo.com> wrote:
>>> Thanks.
>>> Excuse me, did you read how did I set my cluster up? Please look at: https://paste.ubuntu.com/p/Nx2ptqZjFg/
>>> Which part of my configuration is wrong?
>>
>> 1. You configured the IPaddr2 resource to use node2's IP address (192.168.1.10) instead of the floating IP address (192.168.1.4).
>> 2. You didn't configure the resources into a resource group. As a result, the floating IP may end up on a different node compared to the web server.
>>
>> Both of these are explained in more detail in previous emails :)
>>
>> I also thought that Ubuntu used /etc/apache2 instead of /etc/httpd, but maybe not.
>>
>>> Both of the main and secondary servers are an Apache Reverse Proxy Server. I want when main server failed, then the secondary server handle the requests.
>>> How can I achieve this goal?
>>
>> I don't know anything about reverse proxies, sorry. I can only really comment on general principles here, like "an IPaddr2 resource shouldn't manage an IP address that's expected to stay on one particular node" and "if two resources need to run on the same node and start in a particular order, they need to be grouped."
>>
>>>
>>>
>>>
>>>
>>> On Tuesday, March 16, 2021, 11:57:13 PM GMT+3:30, Reid Wahl <nwahl at redhat.com> wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 16, 2021 at 1:03 PM Jason Long <hack3rcon at yahoo.com> wrote:
>>>> Thanks.
>>>> I changed it to the IP address of node2 and I can see my Apache Web Server.
>>>
>>> Like I said, you don't want to do that. You should change it to an IP address that you want the cluster to manage. If you set it to node2's IP address, Pacemaker will try to remove node2's IP address and assign it to node1 if the resource fails over to node1. If node2 is using that address for anything else (e.g., corosync communication), then that would be a big problem.
>>>
>>> The managed floating IP address should be an IP address dedicated to the web server, that can move between cluster nodes as needed.
>>>
>>>>
>>>> $ sudo pcs resource update floating_ip ocf:heartbeat:IPaddr2 ip=192.168.1.10 cidr_netmask=24 op monitor interval=5s
>>>>
>>>> Now, I want to test my cluster and stop node1. On node1 I did:
>>>>
>>>> # pcs cluster stop http_server
>>>> Error: nodes 'http_server' do not appear to exist in configuration
>>>>
>>>> Why?
>>>
>>> The `pcs cluster stop` command stops pacemaker and corosync services on a particular node (the local node if you don't specify one). You've specified `http_server`, so the command is trying to connect to a node called "http_server" and stop services there.
>>>
>>> If you want to stop node1, then run `pcs cluster stop node1`.
>>>
>>> If you want to prevent the http_server resource from running anywhere, then run `pcs resource disable http_server`.
>>>
>>> If you want to prevent the http_server resource from running on node2, then run `pcs resource ban http_server node2`. If you want to remove that constraint later and allow it to run on node2 again, run `pcs resource clear http_server`.
>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tuesday, March 16, 2021, 11:05:48 PM GMT+3:30, Reid Wahl <nwahl at redhat.com> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 16, 2021 at 12:11 PM Jason Long <hack3rcon at yahoo.com> wrote:
>>>>> Thank you so much.
>>>>> I forgot to ask a question. In below command, what should be the ip="IP" value? Is it the IP address of my Apache or node2?
>>>>>
>>>>> $ sudo pcs resource create floating_ip ocf:heartbeat:IPaddr2 ip="IP" cidr_netmask=24 op monitor interval=5s
>>>>
>>>> It's the IP address that you want the cluster to manage. That sounds like it would be your web server IP address. You definitely don't want to set the ip option to some IP address that resides statically on one of the nodes. An IP managed by an IPaddr2 resource can be moved around the cluster.
>>>>
>>>> If that's your web server IP address, you'll also want to put it in a resource group with your apache resource. Otherwise, the floating IP may end up on a different node from your web server, which renders the IP address useless.
>>>>
>>>> For resources that already exist, you can use the `pcs resource group add` command. For example: `pcs resource group add apache_group floating_ip http_server`.
>>>>
>>>> For resources that you're newly creating, you can use the `--group` option of `pcs resource create`. For example, `pcs resource create new_IP IPaddr2 <options> --group apache_group`. That adds the new resource to the end of the group.
>>>>
>>>> The pcs help outputs have more details on these options.
>>>>
>>>> If you're new to resource groups, you can check them out here: https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/2.0/html-single/Pacemaker_Explained/index.html#group-resources
>>>>
>>>>>
>>>>> Logs are:
>>>>> https://paste.ubuntu.com/p/86YHRX6rdC/
>>>>> https://paste.ubuntu.com/p/HHVzNvhRM2/
>>>>> https://paste.ubuntu.com/p/kNxynhfyc2/
>>>>>
>>>>> I have not any "status.conf" file:
>>>>>
>>>>> # cat /etc/httpd/conf.d/status.conf
>>>>> cat: /etc/httpd/conf.d/status.conf: No such file or directory
>>>>>
>>>>
>>>> If you're using Ubuntu, I believe it's in a different location -- somewhere in /etc/apache2 if memory serves.
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tuesday, March 16, 2021, 07:20:32 PM GMT+3:30, Klaus Wenninger <kwenning at redhat.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 3/16/21 3:18 PM, Ken Gaillot wrote:
>>>>>> On Tue, 2021-03-16 at 09:42 +0000, Jason Long wrote:
>>>>>>> Hello,
>>>>>>> I want to launch a Clustering for my Apache Web Server. I have three
>>>>>>> servers:
>>>>>>>
>>>>>>> 1- Main server that acts as a Reverse Proxy
>>>>>>> 2- The secondary server that when my main server stopped, work as a
>>>>>>> Reverse Proxy
>>>>>>> 3- Apache Web Server
>>>>>>>
>>>>>>> The client ---> Reverse Proxy Server ---> Apache Web Server
>>>>>>>
>>>>>>> The IP addresses are:
>>>>>>> Main Server (node1) : 192.168.1.3
>>>>>>> Secondary Server (node2) : 192.168.1.10
>>>>>>> Apache Web Server (node3) : 192.168.1.4
>>>>>>>
>>>>>>> On the main and secondary servers, I installed and configured Apache
>>>>>>> as a Reverse Proxy Server. I created a Virtual Host and my Reverse
>>>>>>> Configuration is:
>>>>>>>
>>>>>>> <VirtualHost *:80>
>>>>>>> ProxyPreserveHost On
>>>>>>> ProxyPass / http://192.168.1.4/
>>>>>>> ProxyPassReverse / http://192.168.1.4/
>>>>>>> </VirtualHost>
>>>>>>>
>>>>>>> As you see, it forward all requests to the Apache Web Server.
>>>>>>>
>>>>>>> I installed and configured Corosync and Pacemaker as below:
>>>>>>>
>>>>>>> On the main and secondary servers, I opened "/etc/hosts" files and
>>>>>>> added my servers IP addresses and host names:
>>>>>>>
>>>>>>> 192.168.1.3 node1
>>>>>>> 192.168.1.10 node2
>>>>>>>
>>>>>>> Then installed Pacemaker, Corosync, and Pcs packages on both servers
>>>>>>> and started its service:
>>>>>>>
>>>>>>> $ sudo yum install corosync pacemaker pcs
>>>>>>> $ sudo systemctl enable pcsd
>>>>>>> $ sudo systemctl start pcsd
>>>>>>> $ sudo systemctl status pcsd
>>>>>>>
>>>>>>> Then Configured the firewall on both servers as below:
>>>>>>>
>>>>>>> $ sudo firewall-cmd --permanent --add-service=http
>>>>>>> $ sudo firewall-cmd --permanent --add-service=high-
>>>>>>> availability
>>>>>>> $ sudo firewall-cmd --reload
>>>>>>>
>>>>>>> After it, on both servers, I created a password for the "hacluster"
>>>>>>> user, then on the main server:
>>>>>>>
>>>>>>> $ sudo pcs host auth node1 node2 -u hacluster -p password
>>>>>>> node1: Authorized
>>>>>>> node2: Authorized
>>>>>>>
>>>>>>> Then:
>>>>>>> $ sudo pcs cluster setup mycluster node1 node2 --start --enable
>>>>>>> $ sudo pcs cluster enable --all
>>>>>>> node1: Cluster Enabled
>>>>>>> node2: Cluster Enabled
>>>>>>>
>>>>>>> After it:
>>>>>>> $ sudo pcs cluster start --all
>>>>>>> node1: Starting Cluster...
>>>>>>> node2: Starting Cluster...
>>>>>>>
>>>>>>> I checked my clusters with below command and they are up and running:
>>>>>>> $ sudo pcs status
>>>>>>> ...
>>>>>>> Node List:
>>>>>>> * Online: [ node1 node2 ]
>>>>>>> ....
>>>>>>>
>>>>>>> And finally, I tried to add a resource:
>>>>>>> $ sudo pcs resource create floating_ip ocf:heartbeat:IPaddr2
>>>>>>> ip=192.168.1.4 cidr_netmask=24 op monitor interval=5s
>>>>> Shouldn't the virtual-IP moved between node1 & node2 be
>>>>> different from the IP used for the web-server on node3?
>>>>> And having just one instance of the reverse-proxy that
>>>>> should probably be colocated with the virtual-IP - right?
>>>>>
>>>>> Klaus
>>>>>
>>>>>>> $ sudo pcs resource create http_server ocf:heartbeat:apache
>>>>>>> configfile="/etc/httpd/conf.d/VirtualHost.conf" op monitor
>>>>>>> timeout="5s" interval="5s"
>>>>>>>
>>>>>>> On both servers (Main and Secondary), pcsd service is enabled, but
>>>>>>> when I want to see my Apache Web Server then it show me below error:
>>>>>>>
>>>>>>> Proxy Error
>>>>>>> The proxy server received an invalid response from an upstream
>>>>>>> server.
>>>>>>> The proxy server could not handle the request
>>>>>>> Reason: Error reading from remote server
>>>>>>>
>>>>>>> Why? Which parts of my configuration is wrong?
>>>>>>> The output of "sudo pcs status" command is:
>>>>>>> https://paste.ubuntu.com/p/V9KvHKwKtC/
>>>>>>>
>>>>>>> Thank you.
>>>>>> The thing to investigate is:
>>>>>>
>>>>>> Failed Resource Actions:
>>>>>> * http_server_start_0 on node2 'error' (1): call=12, status='Timed Out', exitreason='', last-rc-change='2021-03-16 12:28:14 +03:30', queued=0ms, exec=40004ms
>>>>>> * http_server_start_0 on node1 'error' (1): call=14, status='Timed Out', exitreason='', last-rc-change='2021-03-16 12:28:52 +03:30', queued=0ms, exec=40002ms
>>>>>>
>>>>>> The web server start timed out. Check the system, pacemaker and apache
>>>>>> logs around those times for any hints.
>>>>>>
>>>>>> Did you enable and test the status URL? The ocf:heartbeat:apache agent
>>>>>> checks the status as part of its monitor (which is also done for
>>>>>> start). It would be something like:
>>>>>>
>>>>>> cat <<-END >/etc/httpd/conf.d/status.conf
>>>>>> <Location /server-status>
>>>>>> SetHandler server-status
>>>>>> Require local
>>>>>> </Location>
>>>>>> END
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Manage your subscription:
>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> ClusterLabs home: https://www.clusterlabs.org/
>>>>>
>>>>> _______________________________________________
>>>>> Manage your subscription:
>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> ClusterLabs home: https://www.clusterlabs.org/
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Reid Wahl, RHCA
>>>> Senior Software Maintenance Engineer, Red Hat
>>>> CEE - Platform Support Delivery - ClusterHA
>
>>
>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Manage your subscription:
>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>> ClusterLabs home: https://www.clusterlabs.org/
>>>> _______________________________________________
>>>> Manage your subscription:
>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>> ClusterLabs home: https://www.clusterlabs.org/
>>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Reid Wahl, RHCA
>>> Senior Software Maintenance Engineer, Red Hat
>>> CEE - Platform Support Delivery - ClusterHA
>>>
>>> _______________________________________________
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/
>>> _______________________________________________
>>> Manage your subscription:
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> ClusterLabs home: https://www.clusterlabs.org/
>>>
>>
>>
>> --
>> Regards,
>>
>> Reid Wahl, RHCA
>> Senior Software Maintenance Engineer, Red Hat
>> CEE - Platform Support Delivery - ClusterHA
>>
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
>>
>
>
> --
> Regards,
>
> Reid Wahl, RHCA
> Senior Software Maintenance Engineer, Red Hat
> CEE - Platform Support Delivery - ClusterHA
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
--
Regards,
Reid Wahl, RHCA
Senior Software Maintenance Engineer, Red Hat
CEE - Platform Support Delivery - ClusterHA
_______________________________________________
Manage your subscription:
https://lists.clusterlabs.org/mailman/listinfo/users
ClusterLabs home: https://www.clusterlabs.org/
More information about the Users
mailing list