One HTML-only cron + one everything-but-HTML cron

[hugovk]

To help python/docsbuild-scripts#169.

Current situation

Right now the docs server is taking over 40 hours to build a full set of 3.12-3.14 docs, plus 12 translations each:

List of versions/languages

1. 3.14/zh-tw
2. 3.14/zh-cn
3. 3.14/uk
4. 3.14/tr
5. 3.14/pt-br
6. 3.14/pl
7. 3.14/ko
8. 3.14/ja
9. 3.14/it
10. 3.14/id
11. 3.14/fr
12. 3.14/es
13. 3.14/en
14. 3.13/zh-tw
15. 3.13/zh-cn
16. 3.13/uk
17. 3.13/tr
18. 3.13/pt-br
19. 3.13/pl
20. 3.13/ko
21. 3.13/ja
22. 3.13/it
23. 3.13/id
24. 3.13/fr
25. 3.13/es
26. 3.13/en
27. 3.12/zh-tw
28. 3.12/zh-cn
29. 3.12/uk
30. 3.12/tr
31. 3.12/pt-br
32. 3.12/pl
33. 3.12/ko
34. 3.12/ja
35. 3.12/it
36. 3.12/id
37. 3.12/fr
38. 3.12/es
39. 3.12/en

Nearly all these include HTML, plain text, PDF, Texinfo and EPUB (Ukrainian is HTML only). HTML-only is fast to build, about 3-4 minutes. The full set of artifacts is much slower to build, between 40 minutes and two hours, depending on the language, and is mostly due to building latex for PDFs.

What happens is:

A cron goes off at 7 minutes past the hour and starts a new full build loop.

• If there's a build running (= lockfile found), the new one exits and allows the running one to continue.
• If there's no build running, it creates its own lockfile and starts a new build.

For each language/version, we only do a build if the docs have changed since last time, or if the translation has changed since last time. This is good, there's no point rebuilding something that hasn't changed.

However, because the full loop takes over 40 hours, inevitably there have been docs or translation changes since the last time, and we get a full rebuild each time.

This results in long delays between docs being updated, not to mention the high server resources usage.

HTML vs. PDF

We have download stats for the HTML docs, but we don't have download numbers for the other artifacts to compare.

However, I'm certain the HTML is by far the most used, and there's the most benefit to getting fresh HTML up quickly.

An affordance of websites is being able to look up just the pages you need, on-demand. Compared with PDF, where you can download it once and use it as an offline reference. Maybe you'll re-download again later, but there's less benefit in updating often, as the one you usually consult is an old, offline copy.

Proposal

I suggest we have two cron jobs:

1. The current hourly job only builds HTML.

2. A new job builds everything else except HTML.

1. HTML only

When there are new changes, they will be built and uploaded much sooner. It will run much quicker.

It's more likely that on the next pass, some languages can be skipped because there's nothing to update this time round.

2. Everything but HTML

This will be much slower than the HTML-only job, and will take about the same as the current loop does now.

Maybe it'll be a bit quicker due to not needing to build HTML, but maybe a bit slower because we'll sometimes be using CPU to build HTML at the same time. However, the majority of the time is spent running a latex command on a single CPU, so it might not make much difference.

We also don't need to update the non-HTML as often, so its cron could be every few days?

[hugovk]

TODO

If we do this, build_docs.py already has --quick to build HTML-only:

https://github.com/python/docsbuild-scripts/blob/56d72d43e5759cc0ed600827b56e81d8310bcaca/build_docs.py#L525-L530

• Add an option to build everything-but-HTML. (Doc: Add make dist-no-html cpython#124383)
• Adjust the lockfile mechanism so we can have one HTML-only build running and also one everything-but-HTML running at the same time. (Add --select-output docsbuild-scripts#199)
• Adjust the docs cron at psf-salt adjust the existing and add a new cron.present entry: (Restore the HTML docsbuild cron job psf-salt#497)

Anything else?

[hugovk]

It's been mentioned previously, one downside of this sort of approach is:

1. New Python release is made
2. Fast HTML docs built
3. Reader goes to https://docs.python.org/3/download.html and clicks to download a PDF (or other artifact)
4. PDFs haven't built yet and user gets 404 for https://docs.python.org/x.y/archives/python-x.y.z-docs-pdf-a4.zip

This has been partially addressed by adding a 404 that says "The archive you're trying to download has not been built yet. Please try again later or consult the archives for earlier versions."

---

Perhaps we could also mitigate this by renaming the files so they only have x.y in the filename and not x.y.z?

So https://docs.python.org/x.y/archives/python-x.y-docs-pdf-a4.zip instead.

For example, on the day we release 3.13.0, instead of getting a 404 page for the 3.13.0 PDF, they get the 3.13.0rc2 PDF.

I think this is fine. There's usually not much that's changed, and the benefit is everyone gets the HTML and PDF files sooner (all the time, not just for releases).

[AA-Turner]

I seem to remember that the release process includes building the docs + PDF etc, or perhaps it used to. If this is still the case, can part of the release process be to upload a copy of that archive to the docs server as well as the release server?

@ned-deily did this I think for rc2? Is this something we can formalise?

A

[ned-deily]

I seem to remember that the release process includes building the docs + PDF etc, or perhaps it used to. If this is still the case, can part of the release process be to upload a copy of that archive to the docs server as well as the release server?

This subject is confusingly complex for various reasons (some due to attempts to provide compatibility with older end-of-life releases) so there's a good chance that some or all of what follows is wrong but, to the best of my knowledge, it works today like this.

The release process currently does produce a quick build of the untranslated docs (html and PDF et als) for all rc and final releases (but not alpha and beta releases) built from the source release git tag (i.e. v3.13.0rc2). As mostly discussed in PEP 101, the release process saves the unpacked html files of the docset for every release on the doc server (under /srv/docs.python.org/release/ and saves the downloadable files to the download server (under /srv/www.python.org/ftp/python/doc/). The archived html is linked to from the Python Documentation by Version web page and, importantly, the links in their Download these documents pages, for example Download Python 3.12.6 Documentation, link to these archived downloadable files. Links to these archived versions of the docs may also be used by the RM on the individual release pages, often for the link to the changelog. This is all completely independent of the cron doc builds under discussion here.

At the same time, the cron jobs do their things and (try to) produce their daily/3-hour builds on the docs server that are served under various python.org URLs. The actual file names of the download files do include the version number but are served under the branch-specific directories (/srv/docs.python.org/x.y/archives/) resulting in the URLs mentioned earlier (like https://docs.python.org/x.y/archives/python-x.y.z-docs-pdf-a4.zip). But this does result in a curious set of files in these x.y/archives directories because the file name changes following the first cron run after the release manager merges the release engineering branch back into the main repo. The effect is that, at some somewhat random time following a release, the download files built by the cron jobs will switch from ...-x.y.z-... to ...-x.y.z+1.. file names (or even to ...-x.y+1.0a0...) in the same directory and the older files are never deleted even though they imprecisely imply their source, i.e. 3.12/archives/python-3.12.5-docs-pdf-a4.zip neither reflects the state of the documentation at the time of the 3.12.5 release nor necessarily the final state of the 3.12 docs prior to the 3.12.6 release; it's just what existed in the 3.12 git branch the last time the cron started building (and successfully finished) before the 3.12.6 release was merged into the 3.12 branch.

A partial look at a docs server branch directory shows this:

$ ls 3.12/archive
[...]
-rw-rw-r-- 1 docsbuild docs  8361535 Jul 31 11:06 python-3.12.4-docs-html.tar.bz2
-rw-rw-r-- 1 docsbuild docs 13120842 Jul 31 11:07 python-3.12.4-docs-html.zip
-rw-rw-r-- 1 docsbuild docs 17910690 Jul 31 11:17 python-3.12.4-docs-pdf-a4.tar.bz2
-rw-rw-r-- 1 docsbuild docs 17851043 Jul 31 11:17 python-3.12.4-docs-pdf-a4.zip
-rw-rw-r-- 1 docsbuild docs 18067471 Jul 31 11:26 python-3.12.4-docs-pdf-letter.tar.bz2
-rw-rw-r-- 1 docsbuild docs 18010439 Jul 31 11:26 python-3.12.4-docs-pdf-letter.zip
-rw-rw-r-- 1 docsbuild docs  7541776 Jul 31 11:30 python-3.12.4-docs-texinfo.tar.bz2
-rw-rw-r-- 1 docsbuild docs  9727536 Jul 31 11:31 python-3.12.4-docs-texinfo.zip
-rw-rw-r-- 1 docsbuild docs  2878769 Jul 31 11:08 python-3.12.4-docs-text.tar.bz2
-rw-rw-r-- 1 docsbuild docs  4025748 Jul 31 11:08 python-3.12.4-docs-text.zip
-rw-rw-r-- 1 docsbuild docs  6653572 Jul 31 11:27 python-3.12.4-docs.epub
-rw-rw-r-- 1 docsbuild docs  8360373 Sep  7 13:51 python-3.12.5-docs-html.tar.bz2
-rw-rw-r-- 1 docsbuild docs 13126773 Sep  7 13:51 python-3.12.5-docs-html.zip
-rw-rw-r-- 1 docsbuild docs 18385513 Sep  7 14:05 python-3.12.5-docs-pdf-a4.tar.bz2
-rw-rw-r-- 1 docsbuild docs 18342026 Sep  7 14:05 python-3.12.5-docs-pdf-a4.zip
-rw-rw-r-- 1 docsbuild docs 18559979 Sep  7 14:17 python-3.12.5-docs-pdf-letter.tar.bz2
-rw-rw-r-- 1 docsbuild docs 18515957 Sep  7 14:17 python-3.12.5-docs-pdf-letter.zip
-rw-rw-r-- 1 docsbuild docs  7550334 Sep  7 14:22 python-3.12.5-docs-texinfo.tar.bz2
-rw-rw-r-- 1 docsbuild docs  9733259 Sep  7 14:22 python-3.12.5-docs-texinfo.zip
-rw-rw-r-- 1 docsbuild docs  2885597 Sep  7 13:53 python-3.12.5-docs-text.tar.bz2
-rw-rw-r-- 1 docsbuild docs  4028513 Sep  7 13:53 python-3.12.5-docs-text.zip
-rw-rw-r-- 1 docsbuild docs  6658516 Sep  7 14:19 python-3.12.5-docs.epub
-rw-rw-r-- 1 docsbuild docs  8359767 Sep 21 08:07 python-3.12.6-docs-html.tar.bz2
-rw-rw-r-- 1 docsbuild docs 13132489 Sep 21 08:07 python-3.12.6-docs-html.zip
-rw-rw-r-- 1 docsbuild docs 18447737 Sep 21 08:20 python-3.12.6-docs-pdf-a4.tar.bz2
-rw-rw-r-- 1 docsbuild docs 18401603 Sep 21 08:20 python-3.12.6-docs-pdf-a4.zip
-rw-rw-r-- 1 docsbuild docs 18613117 Sep 16 03:08 python-3.12.6-docs-pdf-letter.tar.bz2
-rw-rw-r-- 1 docsbuild docs 18566788 Sep 16 03:08 python-3.12.6-docs-pdf-letter.zip
-rw-rw-r-- 1 docsbuild docs  7592417 Sep 21 08:25 python-3.12.6-docs-texinfo.tar.bz2
-rw-rw-r-- 1 docsbuild docs  9777614 Sep 21 08:25 python-3.12.6-docs-texinfo.zip
-rw-rw-r-- 1 docsbuild docs  2885648 Sep 21 08:09 python-3.12.6-docs-text.tar.bz2
-rw-rw-r-- 1 docsbuild docs  4030381 Sep 21 08:09 python-3.12.6-docs-text.zip
-rw-rw-r-- 1 docsbuild docs  6661795 Sep 21 08:22 python-3.12.6-docs.epub
-rw-rw-r-- 1 docsbuild docs  7852747 May 23  2023 python-3.13.0a0-docs-html.tar.bz2
-rw-rw-r-- 1 docsbuild docs 12366266 May 23  2023 python-3.13.0a0-docs-html.zip
-rw-rw-r-- 1 docsbuild docs 16804233 May 23  2023 python-3.13.0a0-docs-pdf-a4.tar.bz2
-rw-rw-r-- 1 docsbuild docs 16756843 May 23  2023 python-3.13.0a0-docs-pdf-a4.zip
-rw-rw-r-- 1 docsbuild docs 16928897 May 23  2023 python-3.13.0a0-docs-pdf-letter.tar.bz2
-rw-rw-r-- 1 docsbuild docs 16882274 May 23  2023 python-3.13.0a0-docs-pdf-letter.zip
-rw-rw-r-- 1 docsbuild docs  7098538 May 23  2023 python-3.13.0a0-docs-texinfo.tar.bz2
-rw-rw-r-- 1 docsbuild docs  9239149 May 23  2023 python-3.13.0a0-docs-texinfo.zip
-rw-rw-r-- 1 docsbuild docs  2811736 May 23  2023 python-3.13.0a0-docs-text.tar.bz2
-rw-rw-r-- 1 docsbuild docs  3927852 May 23  2023 python-3.13.0a0-docs-text.zip
-rw-rw-r-- 1 docsbuild docs  6715932 May 23  2023 python-3.13.0a0-docs.epub

AFAICT, there is no reason to be keeping the older files in this directory and this would be solved/mitigated, along with the sync problem alluded to above, if the download file names were changed as suggested above by @hugovk. The trick, of course, is to eliminate or minimize any compatibility issues with user expectations and with the separate release archives produced by the RMs.

To the other point:

@ned-deily did this I think for rc2? Is this something we can formalise?

I believe that what I did for 3.13.0rc2 was to add a temporary link on the Python Documentation by Version web page to the rc2 documentation produced by the release process; normally that page does not include links to pre-release versions. Maybe there was something else on the release page, too.

[AA-Turner]

See python/cpython#124489 to alter the build process.

We'll need to consider what to do with the existing /archives/ files. It likely doesn't hurt to keep them, even though they are in effect random snapshots of the documentation at some point in time. If we do remove them, we can create redirects in the docs server configuration so that links don't break.

A

[AA-Turner]

In terms of splitting the build into HTML and non-HTML, we have a triumvirate of patches:

1. Add --select-output docsbuild-scripts#199 to allow choosing between HTML or non-HTML outputs in build_docs.py
2. Restore the HTML docsbuild cron job psf-salt#497 to restore the cron job for HTML-only builds
3. Doc: Run HTML and non-HTML daily builds separately cpython#124493 to split dist-html from everything else in the autobuild targets that docsbuild-scripts uses.

The first can be merged and the builds will carry on as they are now without any change. The second will start building HTML files twice and could potentially overwite itself when copying to /srv/... if a full job and HTML-only job finish at the same time. The CPython change should be merged third, and fixes this by splitting the non-HTML and HTML-only builds into disjoint sets.

A

[ned-deily]

See python/cpython#124489 to alter the build process.

Thanks, that looks good to me. The only potential issue I can think of is that there might be users/scripts out there that might be periodically expecting to directly download the current built artifacts using the old URL formats. I would guess that is not common and I don't think we've ever provided any guarantees about the URLs other than linking through the download.html pages. ...

We'll need to consider what to do with the existing /archives/ files. It likely doesn't hurt to keep them, even though they are in effect random snapshots of the documentation at some point in time. If we do remove them, we can create redirects in the docs server configuration so that links don't break.

... And the links that would break, besides the above-mentioned possible scripts, would be from previously downloaded copies of the html-format documentation (the only artifact where the download links appear?) and from embedded copies of the HTML documentation that are provided, for example, in the python.org macOS installer. Others? So, if we did add redirects from the x.y.z to the new x.y URLs for releases where we apply this change (and are still building docs), that should solve the problem for all of those cases, I think. If it would make the redirecting easier, we could perhaps do a one-time create x.y symlinks for EOL releases.

[AA-Turner]

If we want to add redirects, I've opened python/psf-salt#498 as a draft.

A