[ci] limit parallel windows compile jobs to 24

[lnihlen]

This is an experiment to see if we can prevent some of the compiler OOMs happening without unduly impacting the Windows build latency.

[lnihlen]

I want to roll this for a few days and see how it impacts things, please take a look.

[tstellar]

There is also LLVM_RAM_PER_COMPILE_JOB which might scale better if the machine sizes change, but I'm not sure if this works for Windows.

You may also want to use LLVM_RAM_PER_LINK_JOB/LLVM_PARALLEL_LINK_JOBS too.

[lnihlen]

I wasn't aware of the *_RAM_PER_COMPILE_JOB parameters, that's interesting and might be worth a try, although not sure it would help with compiles OOMing. I haven't seen link OOMs so I didn't want to limit that phase, even though in my own experience its the link that causes OOMs the most. But this is the MSVC toolchain, so it's a bit different of a situation, I suppose.

[Endilll]

One thing to be aware of is https://learn.microsoft.com/en-us/cpp/build/reference/cgthreads-code-generation-threads?view=msvc-170:

By default, cl.exe uses four threads, as if /cgthreads4 were specified.

As far as I can see, we don't override it anywhere. /cgthreads1 should reduce the thread contention (ninja already launches enough to put all the cores to work), and, hopefully, RAM usage.

[lnihlen]

I think the problem is this is all guesswork. I'm looking in to some of the performance monitoring steps suggested by others, to see if we can collect some actual data about what's going on with the Windows build servers. But, in the mean time, if builds are unreliable due to OOM, this may be a band-aid we can put on the problem until we have a more informed solution.

[Meinersbur]

@Endilll I think this PR is to avoid OOM errors (Compiling Flang requires >8GB RAM per .cpp). msvc being multi-threaded (i.e. NOT specifying /cgthreads1) still enables more parallelism that those 24 compile jobs. In my experience, /cgthreads does not change memory usage.

[lnihlen]

If compiling flang really does require >8 GB per .cpp file on MSVC, it probably makes sense to drop this to 16 threads. Although none of the analytics show memory pressure as a problem (see the chart I posted in the Discourse thread), I'm wondering if we're running slowly because we're swap thrashing, because reducing paralellism doesn't seem to negatively impact build times, at least based on my limited testing.

[Meinersbur]

I collected statistics some time ago: https://discourse.llvm.org/t/build-time-comparision/56024 (including for /cgthreds1). Usage may have grown since then.

According to the Discourse post, there is a 20 min wait where nothing happens. This matches the --timeout=1200 LIT parameter. That is, there may be a deadlock in some test that is eventually given up on by LIT.

[AaronBallman]

FWIW, I've also found I need to set LLVM_PARALLEL_LINK_JOBS to something low (like 2-4) unless building with lld. link.exe tends to be quite memory hungry as well, so maybe that's another option to experiment with here?

[AaronBallman]

LGTM as far as the experiment goes, in case you want to land it to see if it solves the OOM issues immediately. We can explore other options in a follow-up if desired.

[AaronBallman]

I collected statistics some time ago: https://discourse.llvm.org/t/build-time-comparision/56024 (including for /cgthreds1). Usage may have grown since then.

According to the Discourse post, there is a 20 min wait where nothing happens. This matches the --timeout=1200 LIT parameter. That is, there may be a deadlock in some test that is eventually given up on by LIT.

FWIW, I see this from a clangd test with some regularity:

timeout=1200 --time-tests C:/ws/src/build/tools/clang/tools/extra/include-cleaner/test C:/ws/src/build/tools/clang/tools/extra/pseudo/test C:/ws/src/build/tools/clang/tools/extra/clangd/test/../unittests C:/ws/src/build/tools/clang/tools/extra/clangd/test C:/ws/src/build/tools/clang/tools/extra/test"

from https://buildkite.com/llvm-project/github-pull-requests/builds/65934

[Endilll]

FWIW, I see this from a clangd test with some regularity:

timeout=1200 --time-tests C:/ws/src/build/tools/clang/tools/extra/include-cleaner/test C:/ws/src/build/tools/clang/tools/extra/pseudo/test C:/ws/src/build/tools/clang/tools/extra/clangd/test/../unittests C:/ws/src/build/tools/clang/tools/extra/clangd/test C:/ws/src/build/tools/clang/tools/extra/test"

from https://buildkite.com/llvm-project/github-pull-requests/builds/65934

This was fixed in #92888. Make sure your branches are up to date.

[AaronBallman]

FWIW, I see this from a clangd test with some regularity:

timeout=1200 --time-tests C:/ws/src/build/tools/clang/tools/extra/include-cleaner/test C:/ws/src/build/tools/clang/tools/extra/pseudo/test C:/ws/src/build/tools/clang/tools/extra/clangd/test/../unittests C:/ws/src/build/tools/clang/tools/extra/clangd/test C:/ws/src/build/tools/clang/tools/extra/test"

from https://buildkite.com/llvm-project/github-pull-requests/builds/65934

This was fixed in #92888. Make sure your branches are up to date.

Good to know, thank you! Yeah, it seems that PR was updated before the fix went in, so it was missing the fix.