[ClusterLabs] killing postgres wal progress in slave makes cluster crashed

Ken Gaillot kgaillot at redhat.com
Wed May 2 12:35:01 EDT 2018


On Sat, 2018-04-28 at 08:46 +0000, 范国腾 wrote:
> Hi,
> There are three nodes: node1,node2,node3。node1 is master, node2 and
> node3 is slave。
> We execute the “truncate table” in 14:25:36 and kill the WAL progress
> in db2。Then the db2 pacemaker is down and db1 is reboot.
> But I could not find any information from the /var/log/messages. The
> flowing is log, could you help find any clue?
>  
>  
> Current DC: db3 (version 1.1.15-11.el7-e174ec8) - partition with
> quorum

The DC is the node that schedules all actions, so its logs will be
helpful too. If there was any fencing, it should be mentioned there.

I don't see anything obvious in the logs you've posted here. But if the
nodes were fenced or crashed, the most recent logs may have not been
written to disk yet.

> Last updated: Sat Apr 28 16:37:53 2018          Last change: Sat Apr
> 28 16:02:25 2018 by hacluster via crmd on db3
>  
> 3 nodes and 19 resources configured
>  
> Node db2: pending
> Online: [ db3 ]
> OFFLINE: [ db1 ]
>  
> Full list of resources:
>  
> ipmi_node1     (stonith:fence_ipmilan):        Started db3
> ipmi_node2     (stonith:fence_ipmilan):        Started db3
> ipmi_node3     (stonith:fence_ipmilan):        Stopped
> Clone Set: dlm-clone [dlm]
>      Started: [ db3 ]
>      Stopped: [ db1 db2 ]
> Clone Set: clvmd-clone [clvmd]
>      Started: [ db3 ]
>      Stopped: [ db1 db2 ]
> Clone Set: clusterfs-clone [clusterfs]
>      Started: [ db3 ]
>      Stopped: [ db1 db2 ]
> Master/Slave Set: pgsql-ha [pgsqld]
>      Masters: [ db3 ]
>      Stopped: [ db1 db2 ]
> Resource Group: mastergroup
>      master-vip (ocf::heartbeat:IPaddr2):       Started db3
>      rep-vip    (ocf::heartbeat:IPaddr2):       Started db3
> slave1-vip     (ocf::heartbeat:IPaddr2):       Stopped
> slave2-vip     (ocf::heartbeat:IPaddr2):       Stopped
>  
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/enabled
>  
> DB1 /var/log/messages
> 
>  
> DB2 /var/log/messages
> 
>  
>  
> DB1 postgres log
> 
> DB2 postgres log
> 
>  
> 发件人: 徐晓菲 
> 发送时间: 2018年4月27日 9:57
> 收件人: 邵大明 <shaodaming at highgo.com>; 范国腾 <fanguoteng at highgo.com>; 王亮 <wa
> ngliang at highgo.com>
> 主题: 回复: 回复: message+pglog
>  
> 嗯嗯,知道了。
>  
> 还有昨天邮件发log的那个问题,不知道是不是跟truncate tb有关,因为跟下面这种情况一样都做过truncate tb
>  
> 这还有一种情况:
> 操作步骤:
> (1)1主2备(db1主  db2备  db3备),psql -h master-vip 
> (2)create tb1;  insert tb1执行中
> (3)kill一个备机(db3)的流复制进程
> (4)该备机重启流复制进程,pcs status仍为原有的1主2备
>  
> (5)truncate tb1
> (6)重新kill一个备机(db3)的流复制进程(没有执行insert)
> (7)原主机db1被关机
> (8)db2上执行pcs status和查看进程
> [root at sds2 ~]# pcs status
> Cluster name: hgpurog
> Stack: corosync
> Current DC: db2 (version 1.1.15-11.el7-e174ec8) - partition with
> quorum
> Last updated: Fri Apr 27 09:47:15 2018      Last change: Fri Apr 27
> 09:28:01 2018 by root via crm_attribute on db1
>  
> 3 nodes and 19 resources configured
>  
> Node db3: pending
> Online: [ db2 ]
> OFFLINE: [ db1 ]
>  
> Full list of resources:
>  
>  ipmi_node1    (stonith:fence_ipmilan):    Started db2
>  ipmi_node2    (stonith:fence_ipmilan):    Stopped
>  ipmi_node3    (stonith:fence_ipmilan):    Started db2
>  Clone Set: dlm-clone [dlm]
>      Started: [ db2 ]
>      Stopped: [ db1 db3 ]
>  Clone Set: clvmd-clone [clvmd]
>      Started: [ db2 ]
>      Stopped: [ db1 db3 ]
>  Clone Set: clusterfs-clone [clusterfs]
>      Started: [ db2 ]
>      Stopped: [ db1 db3 ]
>  Master/Slave Set: pgsql-ha [pgsqld]
>      Slaves: [ db2 ]
>      Stopped: [ db1 db3 ]
>  Resource Group: mastergroup
>      master-vip  (ocf::heartbeat:IPaddr2):   Stopped
>      rep-vip (ocf::heartbeat:IPaddr2):   Stopped
>  slave1-vip    (ocf::heartbeat:IPaddr2):   Stopped
>  slave2-vip    (ocf::heartbeat:IPaddr2):   Stopped
>  
> Failed Actions:
> * pgsqld_promote_0 on db2 'unknown error' (1): call=94, status=Timed
> Out, exitreason='none',
>     last-rc-change='Fri Apr 27 09:36:06 2018', queued=0ms,
> exec=300002ms
>  
>  
> Daemon Status:
>   corosync: active/disabled
>   pacemaker: active/disabled
>   pcsd: active/enabled
> [root at sds2 ~]# 
>  
> [highgo at sds2 data]$ ps -ef|grep postgres
> highgo   29499 28255  0 09:51 pts/1    00:00:00 grep --color=auto
> postgres
>  
> db3上pcs staus 和查看进程
> [root at sds3 ~]# pcs status
> Error: cluster is not currently running on this node
>  
> [root at sds3 ~]# ps -ef |grep postgres
> highgo    4388     1  0 09:19 ?        00:00:13
> /home/highgo/hgdb/bin/postgres -D /home/highgo/hgdb/data
> highgo    4449  4388  0 09:19 ?        00:00:00 postgres: logger
> process   
> highgo   10723  4388  2 09:28 ?        00:00:35 postgres: startup
> process   recovering 0000000900000000000000EF
> highgo   10732  4388  0 09:28 ?        00:00:00 postgres:
> checkpointer process   
> highgo   10733  4388  0 09:28 ?        00:00:00 postgres: writer
> process   
> highgo   11261  4388  0 09:28 ?        00:00:00 postgres: stats
> collector process   
> highgo   12229  4388  0 09:50 ?        00:00:00 postgres: wal
> receiver process   
> root     12231 17313  0 09:50 pts/0    00:00:00 grep --color=auto
> postgres
> [root at sds3 ~]# 
>  
>  
> 祝工作顺利!
> ----------------------------------
> 徐晓菲  产品检测部
> 瀚高基础软件股份有限公司
> 网址:www.highgo.com 
> 地址:济南市高新区新泺大街2117号铭盛大厦20层
> 手机:183-6307-3951  邮箱:xuxiaofei at highgo.com
>  
>  
> 发件人: shaodaming at highgo.com
> 发送时间: 2018-04-27 09:37
> 收件人: xuxiaofei at highgo.com; fanguoteng; 王亮
> 主题: 回复: 回复: message+pglog
> hi, xiaofei
>  
> 交叉就是, 如果两个机器作为client server.
> 一个机器建立400个client访问 备1 数据库1
> 一个机器建立400 个client 访问 备 2 数据库2
> 交叉10% 就是 360个访问备1的数据库1, 40个访问备1的数据库2.
>                 就是 360个访问备2的数据库2, 40个访问备2的数据库1.
> 其他的情况类似按比例改变如上
>  
> thanks.
> Br.
> Bret
> shaodaming at highgo.com
>  
> 发件人: xuxiaofei at highgo.com
> 发送时间: 2018-04-27 09:18
> 收件人: 范国腾; wangliang; shaodaming
> 主题: 回复: message+pglog
> 哈喽
>     这里的交叉是指,比如100%交叉是同时发select,比如10%交叉是备一读一段时间之后,备二再读 么
>  
>  
> 祝工作顺利!
> ----------------------------------
> 徐晓菲  产品检测部
> 瀚高基础软件股份有限公司
> 网址:www.highgo.com 
> 地址:济南市高新区新泺大街2117号铭盛大厦20层
> 手机:183-6307-3951  邮箱:xuxiaofei at highgo.com
>  
>  
> 发件人: xuxiaofei at highgo.com
> 发送时间: 2018-04-26 16:33
> 收件人: 范国腾; wangliang; shaodaming
> 主题: message+pglog
>  
>  
>  
> 祝工作顺利!
> ----------------------------------
> 徐晓菲  产品检测部
> 瀚高基础软件股份有限公司
> 网址:www.highgo.com 
> 地址:济南市高新区新泺大街2117号铭盛大厦20层
> 手机:183-6307-3951  邮箱:xuxiaofei at highgo.com
>  
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.
> pdf
> Bugs: http://bugs.clusterlabs.org
-- 
Ken Gaillot <kgaillot at redhat.com>


More information about the Users mailing list