Reduce memory consumption for multi-statement query strings.

Previously, exec_simple_query always ran parse analysis, rewrite, and
planning in MessageContext, allowing all the data generated thereby
to persist until the end of processing of the whole query string.
That's fine for single-command strings, but if a client sends many
commands in a single simple-Query message, this strategy could result
in annoying memory bloat, as complained of by Andreas Seltenreich.

To fix, create a child context to do this work in, and reclaim it
after each command.  But we only do so for parsetrees that are not
last in their query string.  That avoids adding any memory management
overhead for the typical case of a single-command string.  Memory
allocated for the last parsetree would be freed immediately after
finishing the command string anyway.

Similarly, adjust extension.c's execute_sql_string() to reclaim memory
after each command.  In that usage, multi-command strings are the norm,
so it's a bit surprising that no one has yet complained of bloat ---
especially since the bloat extended to whatever data ProcessUtility
execution might leak.

Amit Langote, reviewed by Julien Rouhaud

Discussion: https://postgr.es/m/[email protected]

diff --git a/src/backend/commands/extension.c b/src/backend/commands/extension.c
index 59ca5cd5a9ff4f15be99daadcc10ec97d045997a..f7202cc9e7dc5ef51821bcb855d0c93db96c803c 100644 (file)
--- a/src/backend/commands/extension.c
+++ b/src/backend/commands/extension.c
@@ -717,9 +717,21 @@ execute_sql_string(const char *sql)
        foreach(lc1, raw_parsetree_list)
        {
                RawStmt    *parsetree = lfirst_node(RawStmt, lc1);
+               MemoryContext per_parsetree_context,
+                                       oldcontext;
                List       *stmt_list;
                ListCell   *lc2;
 
+               /*
+                * We do the work for each parsetree in a short-lived context, to
+                * limit the memory used when there are many commands in the string.
+                */
+               per_parsetree_context =
+                       AllocSetContextCreate(CurrentMemoryContext,
+                                                                 "execute_sql_string per-statement context",
+                                                                 ALLOCSET_DEFAULT_SIZES);
+               oldcontext = MemoryContextSwitchTo(per_parsetree_context);
+
                /* Be sure parser can see any DDL done so far */
                CommandCounterIncrement();
 
@@ -772,6 +784,10 @@ execute_sql_string(const char *sql)
 
                        PopActiveSnapshot();
                }
+
+               /* Clean up per-parsetree context. */
+               MemoryContextSwitchTo(oldcontext);
+               MemoryContextDelete(per_parsetree_context);
        }
 
        /* Be sure to advance the command counter after the last script command */

diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 44a59e1d4fb6fb7ffdc67c24f6e77b0824549bfa..ffd84d877c1cf8b7a4d588d4e8baeb84cd591f81 100644 (file)
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -1070,6 +1070,7 @@ exec_simple_query(const char *query_string)
                bool            snapshot_set = false;
                const char *commandTag;
                char            completionTag[COMPLETION_TAG_BUFSIZE];
+               MemoryContext per_parsetree_context = NULL;
                List       *querytree_list,
                                   *plantree_list;
                Portal          portal;
@@ -1132,10 +1133,25 @@ exec_simple_query(const char *query_string)
                /*
                 * OK to analyze, rewrite, and plan this query.
                 *
-                * Switch to appropriate context for constructing querytrees (again,
-                * these must outlive the execution context).
+                * Switch to appropriate context for constructing query and plan trees
+                * (these can't be in the transaction context, as that will get reset
+                * when the command is COMMIT/ROLLBACK).  If we have multiple
+                * parsetrees, we use a separate context for each one, so that we can
+                * free that memory before moving on to the next one.  But for the
+                * last (or only) parsetree, just use MessageContext, which will be
+                * reset shortly after completion anyway.  In event of an error, the
+                * per_parsetree_context will be deleted when MessageContext is reset.
                 */
-               oldcontext = MemoryContextSwitchTo(MessageContext);
+               if (lnext(parsetree_item) != NULL)
+               {
+                       per_parsetree_context =
+                               AllocSetContextCreate(MessageContext,
+                                                                         "per-parsetree message context",
+                                                                         ALLOCSET_DEFAULT_SIZES);
+                       oldcontext = MemoryContextSwitchTo(per_parsetree_context);
+               }
+               else
+                       oldcontext = MemoryContextSwitchTo(MessageContext);
 
                querytree_list = pg_analyze_and_rewrite(parsetree, query_string,
                                                                                                NULL, 0, NULL);
@@ -1160,8 +1176,8 @@ exec_simple_query(const char *query_string)
 
                /*
                 * We don't have to copy anything into the portal, because everything
-                * we are passing here is in MessageContext, which will outlive the
-                * portal anyway.
+                * we are passing here is in MessageContext or the
+                * per_parsetree_context, and so will outlive the portal anyway.
                 */
                PortalDefineQuery(portal,
                                                  NULL,
@@ -1263,6 +1279,10 @@ exec_simple_query(const char *query_string)
                 * aborted by error will not send an EndCommand report at all.)
                 */
                EndCommand(completionTag, dest);
+
+               /* Now we may drop the per-parsetree context, if one was created. */
+               if (per_parsetree_context)
+                       MemoryContextDelete(per_parsetree_context);
        }                                                       /* end loop over parsetrees */
 
        /*