Function for listing pg_wal/summaries directory

Hi,

While WAL summaries feature and some support functions have been added
in version 17, merely listing the contents of the pg_wal/summaries
directory is missing. As discussed in the pg_ls_archive_status()
discussion, it would be convenient to add pg_ls_summariesdir() that
lists the contents of the pg_wal/summaries directory with the pg_monitor
role.

This patch is based on a very recent master and does not include the
catversion bump.

Best,
Yushi

On Fri, Oct 04, 2024 at 11:32:08AM +0900, btogiwarayuushi wrote:
> While WAL summaries feature and some support functions have been added in
> version 17, merely listing the contents of the pg_wal/summaries directory is
> missing.

Could you explain why you feel the existing support functions are
insufficient?

--
nathan

On Fri, Oct 04, 2024 at 10:02:11AM -0500, Nathan Bossart wrote:
> Could you explain why you feel the existing support functions are
> insufficient?

Because it is not possible to outsource the scan of pg_wal/summaries/
to a different role, no?

On HEAD, one would require a full access to the data folder, as
pg_ls_waldir() does not recurse. Perhaps, rather than introducing a
new function, we should just extend pg_ls_dir_files() so as it can
optionally recurse, then use this option in the existing
pg_ls_waldir() to show stats for pg_wal/ and pg_wal/summary/? You'd
optinally need to print the relative path of the file in the output of
the function, meaning that any summary file in should be shown as
summaries/blah.summary. Just an idea for the bucket of ideas.

This would be unconsistent with the existing
pg_ls_archive_statusdir(), though, so using a new function may be just
better and simpler.
--
Michael

On Mon, Oct 07, 2024 at 10:07:10AM +0900, Michael Paquier wrote:
> On Fri, Oct 04, 2024 at 10:02:11AM -0500, Nathan Bossart wrote:
>> Could you explain why you feel the existing support functions are
>> insufficient?
>
> Because it is not possible to outsource the scan of pg_wal/summaries/
> to a different role, no?

I was under the impression that you could do this with
pg_available_wal_summaries() [0].

[0] https://www.postgresql.org/docs/current/functions-info.html#FUNCTIONS-INFO-WAL-SUMMARY

--
nathan

On 2024/10/07 23:35, Nathan Bossart wrote:
> On Mon, Oct 07, 2024 at 10:07:10AM +0900, Michael Paquier wrote:
>> On Fri, Oct 04, 2024 at 10:02:11AM -0500, Nathan Bossart wrote:
>>> Could you explain why you feel the existing support functions are
>>> insufficient?
>>
>> Because it is not possible to outsource the scan of pg_wal/summaries/
>> to a different role, no?
>
> I was under the impression that you could do this with
> pg_available_wal_summaries() [0].
>
> [0] https://www.postgresql.org/docs/current/functions-info.html#FUNCTIONS-INFO-WAL-SUMMARY

One benefit of supporting something like pg_ls_summariesdir() is that
it allows us to view the last modification time of each WAL summary file
and estimate when they'll be removed based on wal_summary_keep_time.

Of course, we could also extend the existing function to report
the last modification time if this use case is valid, though.

Regards,

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

On Tue, Oct 08, 2024 at 12:41:16PM +0900, Fujii Masao wrote:
> One benefit of supporting something like pg_ls_summariesdir() is that
> it allows us to view the last modification time of each WAL summary file
> and estimate when they'll be removed based on wal_summary_keep_time.
>
> Of course, we could also extend the existing function to report
> the last modification time if this use case is valid, though.

My argument is about knowing the size of each file, for monitoring of
disk space. The retention can be controlled by a GUC based on time,
and this function requires knowing about the file name format.
--
Michael

On Tue, Oct 08, 2024 at 01:19:52PM +0900, Michael Paquier wrote:
> On Tue, Oct 08, 2024 at 12:41:16PM +0900, Fujii Masao wrote:
>> One benefit of supporting something like pg_ls_summariesdir() is that
>> it allows us to view the last modification time of each WAL summary file
>> and estimate when they'll be removed based on wal_summary_keep_time.
>>
>> Of course, we could also extend the existing function to report
>> the last modification time if this use case is valid, though.
>
> My argument is about knowing the size of each file, for monitoring of
> disk space. The retention can be controlled by a GUC based on time,
> and this function requires knowing about the file name format.

Okay. I have no problem with adding something like pg_ls_summariesdir(),
but I guess I was hopeful we could just add any missing information to the
existing WAL summarization information functions. A new pg_ls_*dir()
function would indeed fit nicely with the existing suite of generic file
access functions.

The patch posted upthread looks reasonable to me, so I'll go commit it soon
unless there is any feedback. IMHO we should consider alphabetizing the
table in the docs [0], too.

[0] https://www.postgresql.org/docs/devel/functions-admin.html#FUNCTIONS-ADMIN-GENFILE

--
nathan

On 2024/10/08 23:36, Nathan Bossart wrote:
> On Tue, Oct 08, 2024 at 01:19:52PM +0900, Michael Paquier wrote:
>> On Tue, Oct 08, 2024 at 12:41:16PM +0900, Fujii Masao wrote:
>>> One benefit of supporting something like pg_ls_summariesdir() is that
>>> it allows us to view the last modification time of each WAL summary file
>>> and estimate when they'll be removed based on wal_summary_keep_time.
>>>
>>> Of course, we could also extend the existing function to report
>>> the last modification time if this use case is valid, though.
>>
>> My argument is about knowing the size of each file, for monitoring of
>> disk space. The retention can be controlled by a GUC based on time,
>> and this function requires knowing about the file name format.
>
> Okay. I have no problem with adding something like pg_ls_summariesdir(),
> but I guess I was hopeful we could just add any missing information to the
> existing WAL summarization information functions. A new pg_ls_*dir()
> function would indeed fit nicely with the existing suite of generic file
> access functions.
>
> The patch posted upthread looks reasonable to me, so I'll go commit it soon
> unless there is any feedback.

Thanks! The patch looks good to me, too.

> IMHO we should consider alphabetizing the
> table in the docs [0], too.

+1

Regards,

--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION

On Fri, Oct 11, 2024 at 11:09:30AM +0900, Fujii Masao wrote:
> On 2024/10/08 23:36, Nathan Bossart wrote:
>> The patch posted upthread looks reasonable to me, so I'll go commit it soon
>> unless there is any feedback.
>
> Thanks! The patch looks good to me, too.

Committed.

--
nathan