Discussion: CI resources

[alanz]

At the moment CI is becoming a major bottleneck for landing PRs.

Given that HLS is the first point of contact for many people with Haskell, I believe it is critical that we test as many different configuration options as possible, preferably on a continuous basis.

This goes along with a second important point, it must be easy to make a release, ideally just polishing a change log and tagging, once a month.

We are hitting the limits of the free resources available to OSS projects.

The Haskell Foundation is now a thing, and there is funding for the GHC infrastructure. An initial discussion on IRC with @bgamari indicates that we may be able to host the HLS CI on the GHC gitlab infrastructure (keeping dev in GH, as now).

Is this something we should explore?

[jneira]

ci infra considerations apart, we should review our ci and try to cut jobs that will catch errors with almost no probabilities to be triggered
only test last minor versions of ghc like we did in ghcide f.e.
maybe @pepeiborra can help to see where we can drop more jobs (but not too much 😝)

[bgamari]

For what it's worth, I have put together a quick hack to act as a basis for discussion: https://gitlab.haskell.org/bgamari/haskell-language-server/-/merge_requests/1. At the moment it is running about 8 concurrent jobs, with each taking around 10 minutes (although this should be shorter once the cabal caches are warmed). GHC CI load fluctuates pretty significantly, but we have some capacity set aside for "short" jobs (e.g. not GHC builds, which generally take ~90 minutes). One area where we are currently lacking is Darwin, although this will improve soon as we bring up a new set of donated runners.

Note that GitLab can be used in a CI only mode for a GitHub repository (although I'll admit I've not used it in this way yet).

[bgamari]

Another thing that really ought to be considered regardless of where CI happens: Currently it appears that benchmarks are the largest contributor to CI build time. However, these really don't need to be run on every platform. The sorts of performance regressions that haskell-language-server will feel are almost certainly platform independent.

[jneira]

it t is too late but to take in account for future decisions: we lose a queue of free jobs merging ghcide in hls including its functional test suite and benchmarks.
maybe we should review what are we testing in both suites and remove redundant tests

[jneira]

Another thing that really ought to be considered regardless of where CI happens: Currently it appears that benchmarks are the largest contributor to CI build time. However, these really don't need to be run on every platform. The sorts of performance regressions that haskell-language-server will feel are almost certainly platform independent.

not sure about Linux and macos but performance in windows could deviate quite likely, even in hls, taking in account that we are quite close to ghc, no?

[jneira]

nvm, windows is the one that could deviate the most but it is the unique without benchmarks 🤔
maybe we could drop macos benchs

[pepeiborra]

A major problem with GitHub actions is that they have no native support to cancel redundant builds

https://github.community/t/github-actions-cancel-redundant-builds-not-solved/16025

Right now, 95% of the jobs in the action queue are redundant, since they refer to commits that are below the tip of their branch, or since the branch is out of date with master.

There is a bunch of action providers that claim to solve this problem, so I think we need to pick and adopt one ASAP

[pepeiborra]

Summary of actions I have taken during the last couple of days:

• Halve the number of Github action workflows from 33 to 17
• Streamline CircleCI jobs by removing the Cabal task and skipping testing that is already covered in the GitHub Actions
• Enable auto-cancel of redundant jobs in CircleCI and trigger only on PRs

The attempts to enable auto-cancel of redundant jobs in GitHub Actions have been unsuccessful. Since it is not a core feature, it must be scripted in an action provider. But those scripts need a GITHUB_TOKEN with write access in order to be able to cancel past jobs, and the issue is that PRs from forked repos do not get such write access, whereas PRs from local branches do. This can be changed at the org level /cc @jaspervdj but it raises security concerns.

It's likely that our Github CI will continue to be overwhelmed until we manage to find a solution for redundant builds

[pepeiborra]

Things left to do, as I have ran out of time to work on this myself:

1. Fix the GitHub Actions cache on Windows. It looks like we run over the limit of 5GB relatively easily, so keep this in mind
2. Find a solution for auto-canceling redundant builds or move away from GitHub Actions. Both CircleCI and Azure workflows support this as a core feature

[jneira]

Fix the GitHub Actions cache on Windows. It looks like we run over the limit of 5GB relatively easily, so keep this in mind

I am investigating how to fix windows cache, testing only last minor versions and restore macos builds (hopefully the latter will give us more time than the former will cost): master...jneira:test-last-versions
Alternatively we can only enable builds for macos.

[hazelweakly]

I like to joke that the most effective way to improve performance in CI is to either do less work or do less work. What I mean by that is you can either run less or run it less frequently. Doing both will be good for the long term.

I looked through all of the workflows and I have a few thoughts on how to accomplish both, listed in order of bang-for-buck in time:

1. Run less stuff. Currently everything runs on every single commit of every PR out there.

   • The first change I would suggest is to switch all of the benchmarking to manual trigger instead. People can trigger benchmarks when they want (ie when they have working code to benchmark) and it vastly reduces the amount of churn that goes through the cache.
   • The nix workflow can probably be ran much less often than it is.
   • The second would be to only build a few versions of the LSP until "ready". Maybe as little as just the latest, maybe the latest 3 major version, maybe just the latest on windows/macos and 3 versions on Linux? Lots of wiggle room there. There's a surprising amount of code that only ends up needing to be (initially) tested on a single platform in practice.

2. Every single workflow tests everything. Which doesn't scale well as you can tell :)

   There are 13 packages in cabal.project and in an ideal world each one has a separate set of tests, benchmarks, etc., and the minimal subset is selected to run every time. In a really ideal world, caching everything and letting cabal do the thing would give you this property; however, git + transient caches + VMs + etc... just makes it way too hard for a build system to reliably do change detection like that. Even teams using build systems like Bazel (designed for this exact purpose) end up having to do partial targets on CI systems.

   • Another great change to think about is only building/testing parts that you need to. The granularity doesn't have to be very high here, but the payoff can be great. Drop dependency on shake for install.hs #63 is a great example of a PR that needs nothing else to run in its CI. Plugins are another one that could be an easy win (if that makes sense for how PRs are usually scoped).

More extreme savings are possible by switching the build system to something like Bazel and investing in dedicated CI infrastructure that can git checkout into bare metal with no state cleared between runs. Git fetches get way faster, caches don't have to be shoved into/out of the cloud, and the machines are much faster. I don't really recommend this approach because it makes the tooling way less approachable for contributors and you're going to end up having to do all of the above anyway, so you might as well milk the mileage out of the low hanging fruit first before considering something like Bazel.