<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1569002193;
mso-list-type:hybrid;
mso-list-template-ids:-1971811568 -868590514 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='color:#1F497D'>Hi all.<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>I formatted drbd disk to get rid of the corrupted postmaster.pid file. After this everything works fine. I couldn’t reproduce the issue anymore.<o:p></o:p></span></p><div><p class=MsoNormal><span style='color:#1F497D;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='color:#1F497D;mso-fareast-language:EN-GB'>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D;mso-fareast-language:EN-GB'>Michal Mistina<o:p></o:p></span></p></div><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='mso-fareast-language:EN-GB'>From:</span></b><span lang=EN-US style='mso-fareast-language:EN-GB'> Mistina Michal [mailto:Michal.Mistina@virte.sk] <br><b>Sent:</b> Monday, August 19, 2013 9:39 AM<br><b>To:</b> The Pacemaker cluster resource manager<br><b>Subject:</b> [Pacemaker] PostgreSQL failed to stop after streaming replication established<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoPlainText>Dear community.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>The scenario of redundant environment is in the "graphic" representation...<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText> +------------------------------------+<o:p></o:p></p><p class=MsoPlainText> | WAN |<o:p></o:p></p><p class=MsoPlainText> + v<o:p></o:p></p><p class=MsoPlainText>+------------+------------+ +------------+------------+<o:p></o:p></p><p class=MsoPlainText>|pgsql |pgsql | |pgsql |pgsql |<o:p></o:p></p><p class=MsoPlainText>+------------+------------+ +------------+------------+<o:p></o:p></p><p class=MsoPlainText>|drbd-pri |drbd-sec | |drbd-pri |drbd-sec |<o:p></o:p></p><p class=MsoPlainText>+------------+------------+ +------------+------------+<o:p></o:p></p><p class=MsoPlainText>| pacemaker | | pacemaker |<o:p></o:p></p><p class=MsoPlainText>+-------------------------+ +--------------------------+<o:p></o:p></p><p class=MsoPlainText>| corosync | | corosync |<o:p></o:p></p><p class=MsoPlainText>+------------+------------+ +------------+------------+<o:p></o:p></p><p class=MsoPlainText>|node1 |node2 | |node1 |node2 |<o:p></o:p></p><p class=MsoPlainText>+------------+------------+ +------------+------------+<o:p></o:p></p><p class=MsoPlainText> TC1 TC2<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Within each technical center everything worked fine when migrating resources between nodes. <o:p></o:p></p><p class=MsoPlainText>Then I’ve set up streaming replication from TC1 to TC2. <o:p></o:p></p><p class=MsoPlainText>Now migration from one node to another failes. Pacemaker operation FAILED to stop resource postgres. However postgresql was stopped but postmaster.pid stayed corrupted.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>Now I ended up like this…<o:p></o:p></p><p class=MsoPlainText>I am unable to stop postgresql service correctly on TC1 (streaming replication master). After issuing /etc/init.d/postgresql-9.2 stop the postmaster.pid remains on the filesystem and moreover it is corrupted. I am unable to delete it with rm command.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>It looks like this:<o:p></o:p></p><p class=MsoPlainText>[root@pcmk1 ~]# ll /var/lib/pgsql/9.2/data/<o:p></o:p></p><p class=MsoPlainText>ls: cannot access /var/lib/pgsql/9.2/data/postmaster.pid: No such file or directory total 56<o:p></o:p></p><p class=MsoPlainText>drwx------ 7 postgres postgres 62 Jun 26 17:13 base<o:p></o:p></p><p class=MsoPlainText>drwx------ 2 postgres postgres 4096 Aug 18 00:25 global<o:p></o:p></p><p class=MsoPlainText>drwx------ 2 postgres postgres 17 Jun 26 09:54 pg_clog<o:p></o:p></p><p class=MsoPlainText>-rw------- 1 postgres postgres 5127 Aug 17 16:24 pg_hba.conf<o:p></o:p></p><p class=MsoPlainText>-rw------- 1 postgres postgres 1636 Jun 26 09:54 pg_ident.conf<o:p></o:p></p><p class=MsoPlainText>drwx------ 2 postgres postgres 4096 Jul 2 00:00 pg_log<o:p></o:p></p><p class=MsoPlainText>drwx------ 4 postgres postgres 34 Jun 26 09:53 pg_multixact<o:p></o:p></p><p class=MsoPlainText>drwx------ 2 postgres postgres 17 Aug 18 00:23 pg_notify<o:p></o:p></p><p class=MsoPlainText>drwx------ 2 postgres postgres 6 Jun 26 09:53 pg_serial<o:p></o:p></p><p class=MsoPlainText>drwx------ 2 postgres postgres 6 Jun 26 09:53 pg_snapshots<o:p></o:p></p><p class=MsoPlainText>drwx------ 2 postgres postgres 6 Aug 18 00:25 pg_stat_tmp<o:p></o:p></p><p class=MsoPlainText>drwx------ 2 postgres postgres 17 Jun 26 09:54 pg_subtrans<o:p></o:p></p><p class=MsoPlainText><span lang=DE>drwx------ 2 postgres postgres 6 Jun 26 09:53 pg_tblspc<o:p></o:p></span></p><p class=MsoPlainText><span lang=DE>drwx------ 2 postgres postgres 6 Jun 26 09:53 pg_twophase<o:p></o:p></span></p><p class=MsoPlainText><span lang=DE>-rw------- 1 postgres postgres 4 Jun 26 09:53 PG_VERSION<o:p></o:p></span></p><p class=MsoPlainText><span lang=DE>drwx------ 3 postgres postgres 4096 Aug 18 00:25 pg_xlog<o:p></o:p></span></p><p class=MsoPlainText><span lang=DE>-rw------- 1 postgres postgres 19884 Aug 17 22:54 postgresql.conf<o:p></o:p></span></p><p class=MsoPlainText>-rw------- 1 postgres postgres 71 Aug 18 00:23 postmaster.opts<o:p></o:p></p><p class=MsoPlainText>?????????? ? ? ? ? ? postmaster.pid<o:p></o:p></p><p class=MsoPlainText>-rw-r--r-- 1 postgres postgres 491 Aug 17 16:33 recovery.done<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>I don’t know if the resource agent did something wrong while pacemaker tried stopping postgres or actually the postgres is the source component, which failed to stop correctly. What do you think? Has somebody experienced problem like this?<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>I am using:<o:p></o:p></p><p class=MsoPlainText style='margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>pacemaker-1.1.7-6<o:p></o:p></p><p class=MsoPlainText style='margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>corosync-1.4.1-7<o:p></o:p></p><p class=MsoPlainText style='margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>resource-agents-3.9.2-12<o:p></o:p></p><p class=MsoPlainText style='margin-left:36.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span><![endif]>drbd-8.4.3-2<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>CONFIGURATION<o:p></o:p></p><p class=MsoPlainText>[root@pcmk2 9.2]# crm configure show<o:p></o:p></p><p class=MsoPlainText>node pcmk1 \<o:p></o:p></p><p class=MsoPlainText> attributes standby="off"<o:p></o:p></p><p class=MsoPlainText>node pcmk2 \<o:p></o:p></p><p class=MsoPlainText> attributes standby="off"<o:p></o:p></p><p class=MsoPlainText>primitive drbd_pg ocf:linbit:drbd \<o:p></o:p></p><p class=MsoPlainText> params drbd_resource="postgres" \<o:p></o:p></p><p class=MsoPlainText> op monitor interval="15" role="Master" \<o:p></o:p></p><p class=MsoPlainText> op monitor interval="16" role="Slave" \<o:p></o:p></p><p class=MsoPlainText> op start interval="0" timeout="240" \<o:p></o:p></p><p class=MsoPlainText> op stop interval="0" timeout="120"<o:p></o:p></p><p class=MsoPlainText>primitive pg_fs ocf:heartbeat:Filesystem \<o:p></o:p></p><p class=MsoPlainText> params device="/dev/vg_local-lv_pgsql/lv_pgsql" directory="/var/lib/pgsql/9.2/data" options="noatime,nodiratime" fstype="xfs" \<o:p></o:p></p><p class=MsoPlainText> op start interval="0" timeout="60" \<o:p></o:p></p><p class=MsoPlainText> op stop interval="0" timeout="120"<o:p></o:p></p><p class=MsoPlainText>primitive pg_lsb lsb:postgresql-9.2 \<o:p></o:p></p><p class=MsoPlainText> op monitor interval="30" timeout="60" \<o:p></o:p></p><p class=MsoPlainText> op start interval="0" timeout="60" \<o:p></o:p></p><p class=MsoPlainText> op stop interval="0" timeout="60"<o:p></o:p></p><p class=MsoPlainText>primitive pg_lvm ocf:heartbeat:LVM \<o:p></o:p></p><p class=MsoPlainText> params volgrpname="vg_local-lv_pgsql" \<o:p></o:p></p><p class=MsoPlainText> op start interval="0" timeout="30" \<o:p></o:p></p><p class=MsoPlainText> op stop interval="0" timeout="30"<o:p></o:p></p><p class=MsoPlainText>primitive pg_vip ocf:heartbeat:IPaddr2 \<o:p></o:p></p><p class=MsoPlainText> params ip="x.x.x.x" iflabel="pcmkvip" \<o:p></o:p></p><p class=MsoPlainText> op monitor interval="5"<o:p></o:p></p><p class=MsoPlainText>group PGServer pg_lvm pg_fs pg_lsb pg_vip \<o:p></o:p></p><p class=MsoPlainText> meta target-role="Started"<o:p></o:p></p><p class=MsoPlainText>ms ms_drbd_pg drbd_pg \<o:p></o:p></p><p class=MsoPlainText> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true" target-role="Started"<o:p></o:p></p><p class=MsoPlainText>location master-prefer-node1 pg_vip 50: pcmk1<o:p></o:p></p><p class=MsoPlainText>colocation col_pg_drbd inf: PGServer ms_drbd_pg:Master<o:p></o:p></p><p class=MsoPlainText>order ord_pg inf: ms_drbd_pg:promote PGServer:start<o:p></o:p></p><p class=MsoPlainText>property $id="cib-bootstrap-options" \<o:p></o:p></p><p class=MsoPlainText> dc-version="1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14" \<o:p></o:p></p><p class=MsoPlainText> cluster-infrastructure="openais" \<o:p></o:p></p><p class=MsoPlainText> expected-quorum-votes="4" \<o:p></o:p></p><p class=MsoPlainText> stonith-enabled="false" \<o:p></o:p></p><p class=MsoPlainText> no-quorum-policy="ignore" \<o:p></o:p></p><p class=MsoPlainText> maintenance-mode="true" \<o:p></o:p></p><p class=MsoPlainText> last-lrm-refresh="1376753310"<o:p></o:p></p><p class=MsoPlainText>rsc_defaults $id="rsc-options" \<o:p></o:p></p><p class=MsoPlainText> resource-stickiness="100"<o:p></o:p></p><p class=MsoNormal><span style='mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-GB'>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span style='mso-fareast-language:EN-GB'>Michal Mistina<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>