Don't lose walreceiver start requests due to race condition in postmaster.

When a walreceiver dies, the startup process will notice that and send
a PMSIGNAL_START_WALRECEIVER signal to the postmaster, asking for a new
walreceiver to be launched.  There's a race condition, which at least
in HEAD is very easy to hit, whereby the postmaster might see that
signal before it processes the SIGCHLD from the walreceiver process.
In that situation, sigusr1_handler() just dropped the start request
on the floor, reasoning that it must be redundant.  Eventually, after
10 seconds (WALRCV_STARTUP_TIMEOUT), the startup process would make a
fresh request --- but that's a long time if the connection could have
been re-established almost immediately.

Fix it by setting a state flag inside the postmaster that we won't
clear until we do launch a walreceiver.  In cases where that results
in an extra walreceiver launch, it's up to the walreceiver to realize
it's unwanted and go away --- but we have, and need, that logic anyway
for the opposite race case.

I came across this through investigating unexpected delays in the
src/test/recovery TAP tests: it manifests there in test cases where
a master server is stopped and restarted while leaving streaming
slaves active.

This logic has been broken all along, so back-patch to all supported
branches.

Discussion: https://postgr.es/m/21344.1498494720@sss.pgh.pa.us

diff --git a/src/backend/postmaster/postmaster.c b/src/backend/postmaster/postmaster.c
index 95b2ce3c6be0213bd09b3225f93b283fdc0a8816..39ce9c17da41a6fe35020b8db75631693e886dba 100644 (file)
--- a/src/backend/postmaster/postmaster.c
+++ b/src/backend/postmaster/postmaster.c
@@ -315,6 +315,9 @@ static volatile sig_atomic_t start_autovac_launcher = false;
 /* the launcher needs to be signalled to communicate some condition */
 static volatile bool avlauncher_needs_signal = false;
 
+/* received START_WALRECEIVER signal */
+static volatile sig_atomic_t WalReceiverRequested = false;
+
 /*
  * State for assigning random salts and cancel keys.
  * Also, the global MyCancelKey passes the cancel key assigned to a given
@@ -385,6 +388,7 @@ static int  CountChildren(int target);
 static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
 static pid_t StartChildProcess(AuxProcType type);
 static void StartAutovacuumWorker(void);
+static void MaybeStartWalReceiver(void);
 static void InitPostmasterDeathWatchHandle(void);
 
 #ifdef EXEC_BACKEND
@@ -1529,6 +1533,10 @@ ServerLoop(void)
                kill(AutoVacPID, SIGUSR2);
        }
 
+       /* If we need to start a WAL receiver, try to do that now */
+       if (WalReceiverRequested)
+           MaybeStartWalReceiver();
+
 #ifdef HAVE_PTHREAD_IS_THREADED_NP
 
        /*
@@ -2636,7 +2644,8 @@ reaper(SIGNAL_ARGS)
        /*
         * Was it the wal receiver?  If exit status is zero (normal) or one
         * (FATAL exit), we assume everything is all right just like normal
-        * backends.
+        * backends.  (If we need a new wal receiver, we'll start one at the
+        * next iteration of the postmaster's main loop.)
         */
        if (pid == WalReceiverPID)
        {
@@ -4486,14 +4495,12 @@ sigusr1_handler(SIGNAL_ARGS)
        StartAutovacuumWorker();
    }
 
-   if (CheckPostmasterSignal(PMSIGNAL_START_WALRECEIVER) &&
-       WalReceiverPID == 0 &&
-       (pmState == PM_STARTUP || pmState == PM_RECOVERY ||
-        pmState == PM_HOT_STANDBY || pmState == PM_WAIT_READONLY) &&
-       Shutdown == NoShutdown)
+   if (CheckPostmasterSignal(PMSIGNAL_START_WALRECEIVER))
    {
        /* Startup Process wants us to start the walreceiver process. */
-       WalReceiverPID = StartWalReceiver();
+       /* Start immediately if possible, else remember request for later. */
+       WalReceiverRequested = true;
+       MaybeStartWalReceiver();
    }
 
    if (CheckPostmasterSignal(PMSIGNAL_ADVANCE_STATE_MACHINE) &&
@@ -4832,6 +4839,24 @@ StartAutovacuumWorker(void)
    }
 }
 
+/*
+ * MaybeStartWalReceiver
+ *     Start the WAL receiver process, if not running and our state allows.
+ */
+static void
+MaybeStartWalReceiver(void)
+{
+   if (WalReceiverPID == 0 &&
+       (pmState == PM_STARTUP || pmState == PM_RECOVERY ||
+        pmState == PM_HOT_STANDBY || pmState == PM_WAIT_READONLY) &&
+       Shutdown == NoShutdown)
+   {
+       WalReceiverPID = StartWalReceiver();
+       WalReceiverRequested = false;
+   }
+}
+
+
 /*
  * Create the opts file
  */