[ClusterLabs] The proxy server received an invalid response from an upstream server.

[Ken Gaillot]

On Tue, 2021-03-16 at 09:42 +0000, Jason Long wrote:
> Hello,
> I want to launch a Clustering for my Apache Web Server. I have three
> servers:
> 
> 1- Main server that acts as a Reverse Proxy
> 2- The secondary server that when my main server stopped, work as a
> Reverse Proxy
> 3- Apache Web Server
> 
> The client ---> Reverse Proxy Server ---> Apache Web Server
> 
> The IP addresses are:
> Main Server (node1)                    : 192.168.1.3 
> Secondary Server (node2)           : 192.168.1.10 
> Apache Web Server (node3)        : 192.168.1.4
> 
> On the main and secondary servers, I installed and configured Apache
> as a Reverse Proxy Server. I created a Virtual Host and my Reverse
> Configuration is:
> 
> <VirtualHost *:80>
>     ProxyPreserveHost On
>     ProxyPass / http://192.168.1.4/
>     ProxyPassReverse / http://192.168.1.4/
> </VirtualHost>
> 
> As you see, it forward all requests to the Apache Web Server.
> 
> I installed and configured Corosync and Pacemaker as below:
> 
> On the main and secondary servers, I opened "/etc/hosts" files and
> added my servers IP addresses and host names:
> 
> 192.168.1.3    node1
> 192.168.1.10  node2
> 
> Then installed Pacemaker, Corosync, and Pcs packages on both servers
> and started its service:
> 
> $ sudo yum install corosync pacemaker pcs
> $ sudo systemctl enable pcsd
> $ sudo systemctl start pcsd
> $ sudo systemctl status pcsd
> 
> Then Configured the firewall on both servers as below:
> 
> $ sudo firewall-cmd --permanent --add-service=http
> $ sudo firewall-cmd --permanent --add-service=high-
> availability        
> $ sudo firewall-cmd --reload
> 
> After it, on both servers, I created a password for the "hacluster"
> user, then on the main server:
> 
> $ sudo pcs host auth node1 node2 -u hacluster -p password
> node1: Authorized
> node2: Authorized
> 
> Then:
> $ sudo pcs cluster setup mycluster node1 node2 --start --enable
> $ sudo pcs cluster enable --all
> node1: Cluster Enabled
> node2: Cluster Enabled
> 
> After it:
> $ sudo pcs cluster start --all
> node1: Starting Cluster...
> node2: Starting Cluster...
> 
> I checked my clusters with below command and they are up and running:
> $ sudo pcs status
> ...
> Node List:
>   * Online: [ node1 node2 ]
> ....
> 
> And finally, I tried to add a resource:
> $ sudo pcs resource create floating_ip ocf:heartbeat:IPaddr2
> ip=192.168.1.4 cidr_netmask=24 op monitor interval=5s
> $ sudo pcs resource create http_server ocf:heartbeat:apache
> configfile="/etc/httpd/conf.d/VirtualHost.conf" op monitor
> timeout="5s" interval="5s"
> 
> On both servers (Main and Secondary), pcsd service is enabled, but
> when I want to see my Apache Web Server then it show me below error:
> 
> Proxy Error
> The proxy server received an invalid response from an upstream
> server.
> The proxy server could not handle the request
> Reason: Error reading from remote server
> 
> Why? Which parts of my configuration is wrong?
> The output of "sudo pcs status" command is: 
> https://paste.ubuntu.com/p/V9KvHKwKtC/
> 
> Thank you.

The thing to investigate is:

Failed Resource Actions:
  * http_server_start_0 on node2 'error' (1): call=12, status='Timed Out', exitreason='', last-rc-change='2021-03-16 12:28:14 +03:30', queued=0ms, exec=40004ms
  * http_server_start_0 on node1 'error' (1): call=14, status='Timed Out', exitreason='', last-rc-change='2021-03-16 12:28:52 +03:30', queued=0ms, exec=40002ms

The web server start timed out. Check the system, pacemaker and apache
logs around those times for any hints.

Did you enable and test the status URL? The ocf:heartbeat:apache agent
checks the status as part of its monitor (which is also done for
start). It would be something like:

cat <<-END >/etc/httpd/conf.d/status.conf
 <Location /server-status>
    SetHandler server-status
    Require local
 </Location>
END

-- 
Ken Gaillot <kgaillot at redhat.com>