Skip to content

Custom explain #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 548 commits into
base: master
Choose a base branch
from
Open

Custom explain #1

wants to merge 548 commits into from

Conversation

knizhnik
Copy link
Owner

Make it possible to extend EXPLAIN with custom extension.
I demonstrate it on Bloom extension just to report number of Bloom matches.
Not sure that it is rally useful information but it is used mostly as example.

@knizhnik knizhnik changed the base branch from resize_shared_buffers to master October 14, 2023 06:04
michaelpq and others added 29 commits November 27, 2023 09:40
The libpq code in charge of creating per-connection SSL objects was
prone to a race condition when loading the custom BIO methods needed by
my_SSL_set_fd().  As BIO methods are stored as a static variable, the
initialization of a connection could fail because it could be possible
to have one thread refer to my_bio_methods while it is being manipulated
by a second concurrent thread.

This error has been introduced by 8bb14cd, that has removed
ssl_config_mutex around the call of my_SSL_set_fd(), that itself sets
the custom BIO methods used in libpq.  Like previously, the BIO method
initialization is now protected by the existing ssl_config_mutex, itself
initialized earlier for WIN32.

While on it, document that my_bio_methods is protected by
ssl_config_mutex, as this can be easy to miss.

Reported-by: Willi Mann
Author: Willi Mann, Michael Paquier
Discussion: https://postgr.es/m/[email protected]
Backpatch-through: 12
This is preliminary patch.  It adds NOT NULL checking for the result of
pg_stat_statements_reset() function. It is needed for upcoming patch
"Track statement entry timestamp" that will change the result type of
this function to the timestamp of a reset performed.

Discussion: https://postgr.es/m/flat/72e80e7b160a6eb189df9ef6f068cce3765d37f8.camel%40moonset.ru
Author: Andrei Zubkov
Reviewed-by: Julien Rouhaud, Hayato Kuroda, Yuki Seino, Chengxi Sun
Reviewed-by: Anton Melnikov, Darren Rush, Michael Paquier, Sergei Kornilov
Reviewed-by: Alena Rybakina, Andrei Lepikhov
This patch adds 'stats_since' and 'minmax_stats_since' columns to the
pg_stat_statements view and pg_stat_statements() function.  The new min/max
reset mode for the pg_stat_stetments_reset() function is controlled by the
parameter minmax_only.

'stat_since' column is populated with the current timestamp when a new
statement is added to the pg_stat_statements hashtable.  It provides clean
information about statistics collection time intervals for each statement.
Besides it can be used by sampling solutions to detect situations when a
statement was evicted and stored again between samples.

Such a sampling solution could derive any pg_stat_statements statistic values
for an interval between two samples with the exception of all min/max
statistics. To address this issue this patch adds the ability to reset
min/max statistics independently of the statement reset using the new
minmax_only parameter of the pg_stat_statements_reset(userid oid, dbid oid,
queryid bigint, minmax_only boolean) function. The timestamp of such reset
is stored in the minmax_stats_since field for each statement.
pg_stat_statements_reset() function now returns the timestamp of a reset as the
result.

Discussion: https://postgr.es/m/flat/72e80e7b160a6eb189df9ef6f068cce3765d37f8.camel%40moonset.ru
Author: Andrei Zubkov
Reviewed-by: Julien Rouhaud, Hayato Kuroda, Yuki Seino, Chengxi Sun
Reviewed-by: Anton Melnikov, Darren Rush, Michael Paquier, Sergei Kornilov
Reviewed-by: Alena Rybakina, Andrei Lepikhov
52e4f0c introduced a bug in pgoutput in which missing values in tuples
were incorrectly filled in with NULL. The problem was the use of
CreateTupleDescCopy where CreateTupleDescCopyConstr was required, as the
former drops the constraints in the tuple description (specifically, the
default value constraint) on the floor.

The bug could result in incorrectness when a table replicated via
`REPLICA IDENTITY FULL` underwent a schema change that added a column
with a default value. The problem is that in such cases updates fill NULL
values in old tuples for missing columns for default values. Then on the
subscriber, we failed to find a matching tuple and missed updating the
required row.

Author: Nikhil Benesch
Reviewed-by: Hou Zhijie, Amit Kapila
Backpatch-through: 15
Discussion: http://postgr.es/m/CAPWqQZTEpZQamYsGMn6ZDRvVywwpVPiKH6OY4KSgA+NmeqFNzA@mail.gmail.com
XLogSetAsyncXactLSN(), called at asynchronous commit, would wake up
walwriter every time the LSN advances, but walwriter doesn't actually
do anything unless it has at least 'wal_writer_flush_after' full
blocks of WAL to write. Repeatedly waking up walwriter to do nothing
is a waste of CPU cycles in both walwriter and the backends doing the
wakeups. To fix, apply the same logic in XLogSetAsyncXactLSN() to
decide whether to wake up walwriter, as walwriter uses to determine if
it has any work to do.

In the passing, rename misleadingly named 'flushbytes' local variable
to 'flushblocks'.

Author: Andres Freund, Heikki Linnakangas
Discussion: https://www.postgresql.org/message-id/[email protected]
Fix a bug introduced by c1ec02b. It may happen that the executor
opens indexes on the result relation, but no rows end up being inserted.
Then the index_insert_cleanup still gets executed, but passes down NULL
to the AM callback. The AM callback may not expect this, as is the case
of brininsertcleanup, leading to a crash.

Fixed by only calling the cleanup callback if (ii_AmCache != NULL). This
way the AM can simply assume to only see a valid cache.

Reported-by: Richard Guo
Discussion: https://postgr.es/m/CAMbWs4-w9qC-o9hQox9UHvdVZAYTp8OrPQOKtwbvzWaRejTT=Q@mail.gmail.com
It fails to use the CONCURRENTLY keyword where it was necessary, so add
it.  This text was added to pg11 in commit 5efd604; backpatch to pg12.

Author: Nikolay Samokhvalov <[email protected]>
Discussion: https://postgr.es/m/CAM527d9iz6+=_c7EqSKaGzjqWvSeCeRVVvHZ1v3gDgjTtvgsbw@mail.gmail.com
As of commits dd04e95 and 1833f1a, tuplestore_donestoring(),
SPI_push(), SPI_pop(), SPI_push_conditional(),
SPI_pop_conditional(), and SPI_restore_connection() are no-op
macros provided for backwards compatibility.  This commit removes
these macros, so any uses in third-party code will need to be
removed, too.  Since these macros have been no-ops for a while,
such adjustments won't produce any behavior changes for all
currently-supported versions of PostgreSQL.

Author: Bharath Rupireddy
Discussion: https://postgr.es/m/CALj2ACVeO58JM5tK2Qa8QC-%3DkC8sdkJOTd4BFU%3DK8zs4gGYpjQ%40mail.gmail.com
00b4146 adjusted Bitmapset so that an empty set is always represented
as NULL.  This makes checking for empty sets far cheaper than it used
to be.

There were various places in the code where we'd call bms_membership()
to handle the 3 possible BMS_Membership values.  For the BMS_SINGLETON
case, we'd also call bms_singleton_member() to find the single set member.
This can now be done in a more optimal way by first checking if the set is
NULL and then not bothering with bms_membership() and simply call
bms_get_singleton_member() instead to find the single member.  This
function will return false if there are multiple members in the set.

Here we also tidy up some logic in examine_variable() for the single
member case.  There's now no need to call bms_is_member() as we've
already established that we're working with a singleton Bitmapset, so we
can just check if varRelid matches the singleton member.

Reviewed-by: Richard Guo
Discussion: https://postgr.es/m/CAApHDvqW+CxNPcY245GaWiuqkkqgTudtG2ncGvvSjGn2wdTZLA@mail.gmail.com
scram_SaltedPassword() could take a long time to compute when the number
of iterations used is large enough, and this code uses a tight loop to
compute a salted password.

Note that the same issue exists in libpq when using \password and a
large iteration number, but this cannot be interrupted.  A CFI in the
backend is useful for server-side computations, at least.

Backpatch down to 16, where the user-settable GUC scram_iterations has
been added.

Author: Bowen Shi
Reviewed-by: Aleksander Alekseev, Daniel Gustafsson
Discussion: https://postgr.es/m/CAM_vCueV6xfr08KczfaCEk5J_qeTZtgqN7+orkNLx=g+phE82Q@mail.gmail.com
Backpatch-through: 16
This routine is located in heapam_handler.c, not tableamapi.c.  Issue
noted while hacking the area for a different patch.

Reviewed-by: Richard Guo
Discussion: https://postgr.es/m/[email protected]
Some buildfarm members have been failing a test related to pg_stat_io,
as an effect of 23c8c0c that has switched pg_stat_reset_shared()
from being a no-op to reset all shared stats types.

This extra reset has the effect to make pg_stat_io's counters low enough
that little concurrent activity is enough to cause a failure.  Another
thing I have considered is to move this sequence at the end of
stats.sql, but there are other instabilities, one being pg_stat_wal.

Knowing that there are already tests for the reset of each individual
shared stats target, this test has limited value, so let's remove it to
minimize the number of resets done for each shared stats type.  This
should hopefully improve the stability of the whole.

Discussion: https://postgr.es/m/[email protected]
Add support for tab completion of WITH (...) options to CREATE VIEW,
and for the corresponding SET/RESET (...) options in ALTER VIEW.

Christoph Heiss, reviewed by Melih Mutlu, Vignesh C, Jim Jones,
Mikhail Gribkov, David Zhang, Shubham Khanna, and me.

Discussion: https://postgr.es/m/[email protected]
If the tuple being updated is not visible to the crosscheck snapshot,
we return TM_Updated but the assertions would not hold in that case.
Move them to before the cross-check.

Fixes bug #17893. Backpatch to all supported versions.

Author: Alexander Lakhin
Backpatch-through: 12
Discussion: https://www.postgresql.org/message-id/17893-35847009eec517b5%40postgresql.org
Some of the statements capturing stats reset timestamps have become
unnecessary after a9a8108, so let's remove them.

Author: Bharath Rupireddy
Discussion: https://postgr.es/m/CALj2ACUnvB_Yo=O1xApBa4CDqQpW-x=QM35GBN1MqVRAxAGXEg@mail.gmail.com
We should have done it this way all along, but we accidentally got
away with using the wrong BIO field up until OpenSSL 3.2.  There,
the library's BIO routines that we rely on use the "data" field
for their own purposes, and our conflicting use causes assorted
weird behaviors up to and including core dumps when SSL connections
are attempted.  Switch to using the approved field for the purpose,
i.e. app_data.

While at it, remove our configure probes for BIO_get_data as well
as the fallback implementation.  BIO_{get,set}_app_data have been
there since long before any OpenSSL version that we still support,
even in the back branches.

Also, update src/test/ssl/t/001_ssltests.pl to allow for a minor
change in an error message spelling that evidently came in with 3.2.

Tristan Partin and Bo Andreson.  Back-patch to all supported branches.

Discussion: https://postgr.es/m/CAN55FZ1eDDYsYaL7mv+oSLUij2h_u6hvD4Qmv-7PK7jkji0uyQ@mail.gmail.com
Assigning a precedence to a keyword that isn't a kind of expression
operator is rather dangerous, because it might mask grammar
ambiguities that we'd rather know about.  It's much safer to attach
explicit precedences to individual rules, which will affect the
behavior of only that one rule.  Moreover, when we do have to give
a precedence to a non-operator keyword, we should try to give it the
same precedence as IDENT, thereby reducing the risk of surprising
side-effects.

Apply this hard-won knowledge to SET (which I misassigned ages ago
in commit 2647ad6) and some SQL/JSON-related productions
(from commits 6ee3020, 71bfd15).

Patch HEAD only, since there's no evidence of actual bugs here.

Discussion: https://postgr.es/m/CADT4RqBPdbsZW7HS1jJP319TMRHs1hzUiP=iRJYR6UqgHCrgNQ@mail.gmail.com
We've had repeated bugs in the area of handling SLRU wraparound in the past,
some of which have caused data loss. Switching to an indexing system for SLRUs
that does not wrap around should allow us to get rid of a whole bunch
of problems and improve the overall reliability of the system.

This particular patch however only changes the indexing and doesn't address
the wraparound per se. This is going to be done in the following patches.

Author: Maxim Orlov, Aleksander Alekseev, Alexander Korotkov, Teodor Sigaev
Author: Nikita Glukhov, Pavel Borisov, Yura Sokolov
Reviewed-by: Jacob Champion, Heikki Linnakangas, Alexander Korotkov
Reviewed-by: Japin Li, Pavel Borisov, Tom Lane, Peter Eisentraut, Andres Freund
Reviewed-by: Andrey Borodin, Dilip Kumar, Aleksander Alekseev
Discussion: https://postgr.es/m/CACG%3DezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe%3DpyyjVWA%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPDOYBYrnCAeyndkBktO0WG2xSdYduTF0nxq%2BvfkmTF5Q%40mail.gmail.com
This avoids the wraparound in async.c and removes the corresponding code
complexity. The maximum amount of allocated SLRU pages for NOTIFY / LISTEN
queue is now determined by the max_notify_queue_pages GUC. The default
value is 1048576. It allows to consume up to 8 GB of disk space which is
exactly the limit we had previously.

Author: Maxim Orlov, Aleksander Alekseev, Alexander Korotkov, Teodor Sigaev
Author: Nikita Glukhov, Pavel Borisov, Yura Sokolov
Reviewed-by: Jacob Champion, Heikki Linnakangas, Alexander Korotkov
Reviewed-by: Japin Li, Pavel Borisov, Tom Lane, Peter Eisentraut, Andres Freund
Reviewed-by: Andrey Borodin, Dilip Kumar, Aleksander Alekseev
Discussion: https://postgr.es/m/CACG%3DezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe%3DpyyjVWA%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPDOYBYrnCAeyndkBktO0WG2xSdYduTF0nxq%2BvfkmTF5Q%40mail.gmail.com
Switch from using TransactionId to FullTransactionId in naming of 2PC files.
Transaction state file in the pg_twophase directory now have extra 8 bytes in
the name to address an epoch of a given xid.

Author: Maxim Orlov, Aleksander Alekseev, Alexander Korotkov, Teodor Sigaev
Author: Nikita Glukhov, Pavel Borisov, Yura Sokolov
Reviewed-by: Jacob Champion, Heikki Linnakangas, Alexander Korotkov
Reviewed-by: Japin Li, Pavel Borisov, Tom Lane, Peter Eisentraut, Andres Freund
Reviewed-by: Andrey Borodin, Dilip Kumar, Aleksander Alekseev
Discussion: https://postgr.es/m/CACG%3DezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe%3DpyyjVWA%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPDOYBYrnCAeyndkBktO0WG2xSdYduTF0nxq%2BvfkmTF5Q%40mail.gmail.com
4ed8f09 added 64-bit page numbering for SLRU.  This commit adds tests for
page numbers higher than 2^32.

Author: Maxim Orlov
Reviewed-by: Aleksander Alekseev, Alexander Korotkov
Discussion: https://postgr.es/m/CACG%3DezZe1NQSCnfHOr78AtAZxJZeCvxrts0ygrxYwe%3DpyyjVWA%40mail.gmail.com
Discussion: https://postgr.es/m/CAJ7c6TPDOYBYrnCAeyndkBktO0WG2xSdYduTF0nxq%2BvfkmTF5Q%40mail.gmail.com
For the trivial case of iovcnt == 1, kernels are measurably slower at
dealing with the more complex arguments of preadv/pwritev than the
equivalent plain old pread/pwrite.  The overheads are worth it for
iovcnt > 1, but for 1 let's just redirect to the cheaper calls.  While
we could leave it to callers to worry about that, we already have to
have our own pg_ wrappers for portability reasons so it seems
reasonable to centralize this knowledge there (thanks to Heikki for this
suggestion).  Try to avoid function call overheads by making them
inlinable, which might also allow the compiler to avoid the branch in
some cases.  For systems that don't have preadv and pwritev (currently:
Windows and [closed] Solaris), we might as well pull the replacement
functions up into the static inline functions too.

Reviewed-by: Heikki Linnakangas <[email protected]>
Discussion: https://postgr.es/m/CA+hUKGJkOiOCa+mag4BF+zHo7qo=o9CFheB8=g6uT5TUm2gkvA@mail.gmail.com
When there is a need to filter multiple tables with include and/or exclude
options it's quite possible to run into the limitations of the commandline.
This adds a --filter=FILENAME feature to pg_dump, pg_dumpall and pg_restore
which is used to supply a file containing object exclude/include commands
which work just like their commandline counterparts. The format of the file
is one command per row like:

    <command> <object> <objectpattern>

<command> can be "include" or "exclude", <object> can be table_data, index
table_data_and_children, database, extension, foreign_data, function, table
schema, table_and_children or trigger.

This patch has gone through many revisions and design changes over a long
period of time, the list of reviewers reflect reviewers of some version of
the patch, not necessarily the final version.

Patch by Pavel Stehule with some additional hacking by me.

Author: Pavel Stehule <[email protected]>
Reviewed-by: Justin Pryzby <[email protected]>
Reviewed-by: vignesh C <[email protected]>
Reviewed-by: Dean Rasheed <[email protected]>
Reviewed-by: Tomas Vondra <[email protected]>
Reviewed-by: Julien Rouhaud <[email protected]>
Reviewed-by: Erik Rijkers <[email protected]>
Discussion: https://postgr.es/m/CAFj8pRB10wvW0CC9Xq=1XDs=zCQxer3cbLcNZa+qiX4cUH-G_A@mail.gmail.com
Spotted while looking over changes for another patch.
The set_query_timer_restart() required an argument to define a value
to query_timer_restart, but none of the existing callers passes an
argument to this function.

This changes the function to set a value without an argument.

Backpatch through 16 where the background psql TAP functions were
refactored by 664d757.

Reviewed-by: Bharath Rupireddy, Tom Lane
Discussion: https://postgr.es/m/CAD21AoA0B6VKe_5A9nZi8i5umwSN-zJJuPVNht9DaOZ9SJumMA@mail.gmail.com
Backpatch-through: 16
Backpatch through 16 where this was introduced by 664d757.

Reviewed-by: Daniel Gustafsson
Backpatch-through: 16
Discussion: http://postgr.es/m/CAD21AoBXMEqDBLoDuAWVWoTLYB4aNsxx4oYNmyJJbhfq_vGQBQ@mail.gmail.com
We allow to upgrade the slot iff there is no pending WAL to be processed.
The test first disables the subscription to avoid unnecessary LOGs on the
subscriber and then stops the publisher node. It is quite possible that
just before the shutdown of the publisher, autovacuum generates some WAL
record that needs to be processed, so just disable the autovacuum for this
test.

Per buildfarm.

Author: Hayato Kuroda
Reviewed-by: Amit Kapila
Discussion: http://postgr.es/m/OS3PR01MB9882FED1F0060468FB01B9DAF583A@OS3PR01MB9882.jpnprd01.prod.outlook.com
Display the name of the foreign server for which the user mapping was
not found.

Author: Ian Lawrence Barwick <[email protected]>
Reviewed-by: Laurenz Albe <[email protected]>
Reviewed-by: Peter Eisentraut <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/CAB8KJ=jFzNaeyFtLcTZNOc6fd1+F93pGVLFa-wyt31wn7VNxqQ@mail.gmail.com
petere and others added 28 commits January 4, 2024 16:28
This adds a new ALTER TABLE subcommand ALTER COLUMN ... SET EXPRESSION
that changes the generation expression of a generated column.

The syntax is not standard but was adapted from other SQL
implementations.

This command causes a table rewrite, using the usual ALTER TABLE
mechanisms.  The implementation is similar to and makes use of some of
the infrastructure of the SET DATA TYPE subcommand (for example,
rebuilding constraints and indexes afterwards).  The new command
requires a new pass in AlterTablePass, and the ADD COLUMN pass had to
be moved earlier so that combinations of ADD COLUMN and SET EXPRESSION
can work.

Author: Amul Sul <[email protected]>
Discussion: https://www.postgresql.org/message-id/flat/CAAJ_b94yyJeGA-5M951_Lr+KfZokOp-2kXicpmEhi5FXhBeTog@mail.gmail.com
This provides the useful ability to declare a variable that is an array
of the type of some other variable or some table column.

Quan Zongliang, Pavel Stehule

Discussion: https://postgr.es/m/[email protected]
Many foreach loops only use the ListCell pointer to retrieve the
content of the cell, like so:

    ListCell   *lc;

    foreach(lc, mylist)
    {
        int         myint = lfirst_int(lc);

        ...
    }

This commit adds a few convenience macros that automatically
declare the loop variable and retrieve the current cell's contents.
This allows us to rewrite the previous loop like this:

    foreach_int(myint, mylist)
    {
        ...
    }

This commit also adjusts a few existing loops in order to add
coverage for the new/adjusted macros.  There is presently no plan
to bulk update all foreach loops, as that could introduce a
significant amount of back-patching pain.  Instead, these macros
are primarily intended for use in new code.

Author: Jelte Fennema-Nio
Reviewed-by: David Rowley, Alvaro Herrera, Vignesh C, Tom Lane
Discussion: https://postgr.es/m/CAGECzQSwXKnxGwW1_Q5JE%2B8Ja20kyAbhBHO04vVrQsLcDciwXA%40mail.gmail.com
If we have DECHIST statistics about the argument expression, use
the average number of distinct elements as the array length estimate.
(It'd be better to use the average total number of elements, but
that is not currently calculated by compute_array_stats(), and
it's unclear that it'd be worth extra effort to get.)

To do this, we have to change the signature of estimate_array_length
to pass the "root" pointer.  While at it, also change its result
type to "double".  That's probably not really necessary, but it
avoids any risk of overflow of the value extracted from DECHIST.
All existing callers are going to use the result in a "double"
calculation anyway.

Paul Jungwirth, reviewed by Jian He and myself

Discussion: https://postgr.es/m/CA+renyUnM2d+SmrxKpDuAdpiq6FOM=FByvi6aS6yi__qyf6j9A@mail.gmail.com
A typo has been introduced by 31966b1 when updating the state of a
local buffer when a temporary relation is extended, for the case of a
block included in the relation range extended, when it is already found
in the hash table holding the local buffers.  In this case, BM_VALID
should be cleared, but the buffer state was changed so as BM_VALID
remained while clearing the other flags.

As reported on the thread, it was possible to corrupt the state of the
local buffers on ENOSPC, but the states would be corrupted on any kind
of ERROR during the relation extend (like partial writes or some other
errno).

Reported-by: Alexander Lakhin
Author: Tender Wang
Reviewed-by: Richard Guo, Alexander Lakhin, Michael Paquier
Discussion: https://postgr.es/m/[email protected]
Backpatch-through: 16
Support referencing a composite-type variable in %TYPE.

Remove the undocumented, untested, and pretty useless ability
to have the subject of %TYPE be an (unqualified) type name.
You get the same result by just not writing %TYPE.

Add or adjust some test cases to improve code coverage here.

Discussion: https://postgr.es/m/[email protected]
When the SJE code handles the transfer of qual clauses from the removed
relation to the remaining one, it replaces the Vars of the removed
relation with the Vars of the remaining relation for each clause, and
then reintegrates these clauses into the appropriate restriction or join
clause lists, while attempting to avoid duplicates.

However, the code compares RestrictInfo->clause to determine if two
clauses are duplicates.  This is just flat wrong.  Two RestrictInfos
with the same clause can have different required_relids,
incompatible_relids, is_pushed_down, and so on.  This can cause qual
clauses to be mistakenly omitted, leading to wrong results.

This patch fixes it by comparing the entire RestrictInfos not just their
clauses ignoring 'rinfo_serial' field (otherwise almost all RestrictInfos will
be unique).  Making 'rinfo_serial' equal_ignore would break other code.  This
is why this commit implements our own comparison function for checking the
equality of RestrictInfos.

Reported-by: Zuming Jiang
Bug: #18261
Discussion: https://postgr.es/m/18261-2a75d748c928609b%40postgresql.org
Author: Richard Guo
During the calculations of the maximum for the number of buckets, take into
account that later we round that to the next power of 2.

Reported-by: Karen Talarico
Bug: #16925
Discussion: https://postgr.es/m/16925-ec96d83529d0d629%40postgresql.org
Author: Thomas Munro, Andrei Lepikhov, Alexander Korotkov
Reviewed-by: Alena Rybakina
Backpatch-through: 12
An array element equal to INT_MAX gave this code indigestion,
causing an infinite loop that surely ended in SIGSEGV.  We fixed
some nearby problems awhile ago (cf 757c518) but missed this.

Report and diagnosis by Alexander Lakhin (bug #18273); patch by me

Discussion: https://postgr.es/m/[email protected]
The test query in 5ef34a8 is running over the empty emp1 table giving the
same (empty) return both with and without the fix.  Add one row to that table
to make not just the test query plan, but also the test query result different.

Reported-by: Richard Guo
Bug: #18261
Discussion: https://postgr.es/m/CAMbWs49igjcszLgicb4D1N21_5iNDoxheJ7KFmAcs_z%3DLx6jhg%40mail.gmail.com
Since commit 599b33b, this function assumed that every RTE_RELATION
RangeTblEntry would have an associated RelOptInfo.  But that's not so:
we only build RelOptInfos for relations that are scanned by the query.
In particular the target of an INSERT won't have one, so that Vars
appearing in an INSERT ... RETURNING list will not have an associated
RelOptInfo.  This apparently wasn't a problem before commit f7816ae
taught examine_simple_variable() to drill down into CTEs containing
INSERT RETURNING, but it is now.

To fix, add a fallback code path that gets the userid to use directly
from the RTEPermissionInfo associated with the RTE.  (Sadly, we must
have two code paths, because not every RTE has a RTEPermissionInfo
either.)

Per report from Alexander Lakhin.  No back-patch, since the case is
apparently unreachable before f7816ae.

Discussion: https://postgr.es/m/[email protected]
This simplifies copying libpq/libpq-be-fe-helpers.h into extensions,
because some supported PostgreSQL versions lack the other header.

Discussion: https://postgr.es/m/[email protected]
This replaces dblink's blocking libpq calls, allowing cancellation and
allowing DROP DATABASE (of a database not involved in the query).  Apart
from explicit dblink_cancel_query() calls, dblink still doesn't cancel
the remote side.  The replacement for the blocking calls consists of
new, general-purpose query execution wrappers in the libpqsrv facility.
Out-of-tree extensions should adopt these.  Use them in postgres_fdw,
replacing a local implementation from which the libpqsrv implementation
derives.  This is a bug fix for dblink.  Code inspection identified the
bug at least thirteen years ago, but user complaints have not appeared.
Hence, no back-patch for now.

Discussion: https://postgr.es/m/[email protected]
When SJE uses RelOptInfo.unique_for_rels cache, it passes filtered quals to
innerrel_is_unique_ext().  That might lead to an invalid match to cache entries
made by previous non self-join checking calls.  Add UniqueRelInfo.self_join
flag to prevent such cases.  Also, fix that SJE should require a strict match
of outerrelids to make sure UniqueRelInfo.extra_clauses are valid.

Reported-by: Alexander Lakhin
Discussion: https://postgr.es/m/4788f781-31bd-9796-d7d6-588a751c8787%40gmail.com
The target relation for INSERT/UPDATE/DELETE/MERGE has a different behavior
than other relations in EvalPlanQual() and RETURNING clause.  This is why we
forbid target relation to be either source or target relation in SJE.
It's not clear if we could ever support this.

Reported-by: Alexander Lakhin
Discussion: https://postgr.es/m/b9e8f460-f9a6-0e9b-e8ba-60d59f0bc22c%40gmail.com
Given that now SJE doesn't work with result relation, turn a code dealing with
that into an assert that it shouldn't happen.
The note regarding character encoding form in "The Information Schema"
said that LATIN1 character repertoires only use one encoding form
LATIN1. This is not correct because LATIN1 has another encoding form
ISO-2022-JP-2. To fix this, replace LATIN1 with LATIN2, which is not
supported by ISO-2022-JP-2, thus it can be said that LATIN2 only uses
one encoding form.

Back-patch to supported branches.

Author: Tatsuo Ishii
Reviewed-by: Daniel Gustafsson
Discussion: https://postgr.es/m/flat/20240102.153925.1147403616414525145.t-ishii%40sranhm.sra.co.jp
Essentially this moves the non-interactive part of psql's "\password"
command into an exported client function. The password is not sent to the
server in cleartext because it is "encrypted" (in the case of scram and md5
it is actually hashed, but we have called these encrypted passwords for a
long time now) on the client side. This is good because it ensures the
cleartext password is never known by the server, and therefore won't end up
in logs, pg_stat displays, etc.

In other words, it exists for the same reason as PQencryptPasswordConn(), but
is more convenient as it both builds and runs the "ALTER USER" command for
you. PQchangePassword() uses PQencryptPasswordConn() to do the password
encryption. PQencryptPasswordConn() is passed a NULL for the algorithm
argument, hence encryption is done according to the server's
password_encryption setting.

Also modify the psql client to use the new function. That provides a builtin
test case. Ultimately drivers built on top of libpq should expose this
function and its use should be generally encouraged over doing ALTER USER
directly for password changes.

Author: Joe Conway
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/flat/b75955f7-e8cc-4bbd-817f-ef536bacbe93%40joeconway.com
Both lwlocknames.txt and wait_event_names.txt contain a list of all
the predefined LWLocks, i.e., those with predefined positions
within MainLWLockArray.  It is easy to miss one or the other,
especially since the list in wait_event_names.txt omits the "Lock"
suffix from all the LWLock wait events.  This commit adds a cross-
check of these lists to the script that generates lwlocknames.h.
If the lists do not match exactly, building will fail.

Suggested-by: Robert Haas
Reviewed-by: Robert Haas, Michael Paquier, Bertrand Drouvot
Discussion: https://postgr.es/m/20240102173120.GA1061678%40nathanxps13
The documentation for this parameter lists its type as boolean, but
it is actually an integer.  Furthermore, there is no mention of how
the value is interpreted when specified without units.  This commit
fixes these oversights in commit 174c480.

Co-authored-by: Hubert Depesz Lubaczewski
Discussion: https://postgr.es/m/ZZwkujFihO2uqKwp%40depesz.com
On Windows, cmd.exe is used to launch the postmaster process to ease its
redirection setup.  However, cmd.exe may execute other programs at
startup due to autorun configurations, which could influence the
postmaster startup.  This patch adds /D flag to the launcher cmd.exe
command line to disable autorun settings written in the registry.

This is arguably a bug, but no backpatch is done now out of caution.

Reported-by: Hayato Kuroda
Author: Kyotaro Horiguchi
Reviewed-by: Robert Haas, Michael Paquier
Discussion: https://postgr.es/m/[email protected]
Author: Michael Banck
Reviewed-By: Aleksander Alekseev, Michael Paquier, Peter Eisentraut
Discussion: https://postgr.es/m/[email protected]
knizhnik pushed a commit that referenced this pull request Jan 10, 2024
Default privileges are represented as NULL::aclitem[] in catalog ACL
columns, while revoking all privileges leaves an empty aclitem[].
These two cases used to produce identical output in psql meta-commands
like \dp.  Using something like "\pset null '(default)'" as a
workaround for spotting the difference did not work, because null
values were always displayed as empty strings by describe.c's
meta-commands.

This patch improves that with two changes:

1. Print "(none)" for empty privileges so that the user is able to
   distinguish them from default privileges, even without special
   workarounds.

2. Remove the special handling of null values in describe.c,
   so that "\pset null" is honored like everywhere else.
   (This affects all output from these commands, not only ACLs.)

The privileges shown by \dconfig+ and \ddp as well as the column
privileges shown by \dp are not affected by change #1, because the
respective aclitem[] is reset to NULL or deleted from the catalog
instead of leaving an empty array.

Erik Wienhold and Laurenz Albe

Discussion: https://postgr.es/m/[email protected]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.