Update JIT compilation to use LLVM 19

[savannahostrowski]

The JIT currently uses LLVM 18. The final release of LLVM 19 is slated for September 17th. We should update to the latest version.

Linked PRs

• GH-123714: Update JIT compilation to use LLVM 19 #124093
• gh-125498: Update JIT builds to use LLVM 19 and use preserve_none #125499

[savannahostrowski]

I'm going to start poking around at this.

[ZeroIntensity]

I'm a little concerned about this. I was on a laptop a few weeks ago that had LLVM 17 installed, and I upgraded it to 18 to build the JIT -- it completely broke my system, and took several hours of debugging in a recovery shell to find the problem and fix it. Will we require LLVM 19, or just support it?

[brandtbucher]

Probably require, especially since the feature is optional and (still) experimental. We're also going to want to use some new features of 19, since it means we can get rid of some ugly hacks and will probably pave the way for easier performance improvements on our end (so it's not just updating our toolchain for the sake of updating).

BTW, multiple LLVM versions can typically be installed side-by-side, and our build process works correctly even if clang is an incompatible version but clang-18 exists (for example). Did just installing the new version break everything, or did you completely replace the existing 17 installation?

We should also hold off on this until LLVM 19 is available for all of the platforms we care about. IIRC, Homebrew support was way behind in the 18 cycle.

[brandtbucher]

Also, if your machine was macOS, installing an incompatible LLVM on your PATH will probably break stuff (which is why Homebrew doesn't do it by default). The build scripts know how to find a Homebrew install that isn't on the user's PATH though.

[ZeroIntensity]

Did just installing the new version break everything, or did you completely replace the existing 17 installation?

I'm not totally sure (I don't know much about package managers), probably the latter. FWIW, the machine was Arch.

[savannahostrowski]

@brandtbucher Yeah, I wouldn't want to merge the PR in until the installers catch up. It'll make the dev experience a lot more palatable than building LLVM from source 😅. I just kind of figured I'd get a head start trying to wrap my head around it since rc-4 should be pretty stable.

[picnixz]

I'm not totally sure (I don't know much about package managers), probably the latter. FWIW, the machine was Arch.

Assuming that Arch is Arch Linux for you, IIRC, it does not support having two different versions of LLVMs living together (but I don't know if it's still the case and if you can just build it from source and isolate it). On the other hand, upgrading to LLVM 18 can break some of your system packages (I remember that there were some issues with LLVM 16 which broke Steam games, but I'm not surprised that it can break a rolling-release).

[ZeroIntensity]

it does not support having two different versions of LLVMs living together

Yeah, I learned that the hard way. I'm not opposed to upgrading to LLVM 19 for the JIT, I just think we should keep compatibility in mind.

[brandtbucher]

Hm, okay, definitely something to keep in mind then (though as a general improvement for the future, I don’t see it as a blocker for this particular issue).

I wonder if we can do something like we do for Homebrew, where there’s some sort of standard install location that’s not on the user’s PATH. Another option could be to let a user pass a path to their LLVM installation via a configure arg instead of us trying to find it. Or we could attempt to support a range of versions once the JIT is stable, which could work but seems like it has the potential to massively increase the complexity of the build scripts (and our testing matrix).

See also GH-115869 for related discussion about removing the LLVM dependency.

[picnixz]

I wonder if we can do something like we do for Homebrew, where there’s some sort of standard install location that’s not on the user’s PATH

Should probably be possible according to https://llvm.org/docs/GettingStarted.html#getting-the-source-code-and-building-llvm. For other non-rolling release distributions, multiple versions can co-exist together (I have multiple LLVM versions on openSUSE Leap).