lld doesn't support x86-32 -fno-plt GD->LD, GD->LE, LD->LE optimizations

[rakslice]

$ clang++ --version
Alpine clang version 15.0.6
Target: i586-alpine-linux-musl
Thread model: posix
InstalledDir: /usr/bin
$ ld.lld --version
LLD 15.0.6 (compatible with GNU linkers)

I'm in the process of chasing the segfault in firefox in alpine 3.17 x86.

I've found that an object file mozglue/baseprofiler/Unified_cpp_mozglue_baseprofiler0.o built with clang++
(build command line)

/usr/bin/clang++ -Qunused-arguments -std=gnu++17 -o Unified_cpp_mozglue_baseprofiler0.o -c  -I/home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/dist/stl_wrappers -I/home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/dist/system_wrappers -include /home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -DNDEBUG=1 -DTRIMMED=1 -DIMPL_MFBT -DIMPL_MFBT -DMOZ_HAS_MOZGLUE -I/home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/mozglue/baseprofiler -I/home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/mozglue/baseprofiler -I/home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/mozglue/baseprofiler/core -I/home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/mozglue/linker -I/home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/dist/include -I/usr/include/nspr -I/home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/dist/include/nss -I/usr/include/libpng16 -I/usr/include/pixman-1 -DMOZILLA_CLIENT -include /home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/mozilla-config.h -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -Os -fomit-frame-pointer -O2 -fno-plt -fomit-frame-pointer -funwind-tables -Wall -Wbitfield-enum-conversion -Wdeprecated-this-capture -Wempty-body -Wformat-type-confusion -Wignored-qualifiers -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wno-error=tautological-type-limit-compare -Wunreachable-code -Wunreachable-code-return -Wunused-but-set-parameter -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wno-range-loop-analysis -Wc++2a-compat -Wenum-compare-conditional -Wenum-float-conversion -Wno-ambiguous-reversed-operator -Wno-error=deprecated -Wno-error=depecated-anon-enum-enum-conversion -Wno-error=deprecated-enum-enum-conversion -Wno-error=deprecated-pragma -Wno-error=deprecated-this-capture -Wno-error=deprecated-volatile -Wcomma -Wimplicit-fallthrough -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=free-nonheap-object -Wno-error=atomic-alignment -Wno-error=deprecated-builtins -Wformat -Wformat-security -Wno-psabi -Wthread-safety -Wno-unknown-warning-option -Wno-ignored-qualifiers -fno-strict-aliasing -ffp-contract=off  -MD -MP -MF .deps/Unified_cpp_mozglue_baseprofiler0.o.pp   Unified_cpp_mozglue_baseprofiler0.cpp

has TLS lea-call sequences like

 175:   f0 ff 46 28             lock incl 0x28(%esi)
 179:   8d 83 00 00 00 00       lea    0x0(%ebx),%eax
 17f:   ff 93 00 00 00 00       call   *0x0(%ebx)
 185:   89 c1                   mov    %eax,%ecx
 187: 89 a8 00 00 00 00 mov %ebp,0x0(%eax)

with relevant relocs

OFFSET   TYPE              VALUE
0000017b R_386_TLS_LDM     _ZN7mozilla12baseprofiler19TLSRegisteredThread17sRegisteredThreadE
00000181 R_386_GOT32       ___tls_get_addr

when this is linked with lld
(build command line)

/usr/bin/clang++ -Qunused-arguments -std=gnu++17 -o ../../dist/bin/firefox -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -Os -fomit-frame-pointer -O2 -fno-plt -fomit-frame-pointer -funwind-tables /home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/browser/app/firefox.list -lpthread -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -fstack-protector-strong -fstack-clash-protection -rdynamic -Wl,-rpath-link,/home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/dist/bin -Wl,-rpath-link,/usr/lib -pie -latomic

(resulting lld command line)

`"/usr/bin/ld.lld" -pie -export-dynamic -z now -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_i386 -export-dynamic -dynamic-linker /lib/ld-musl-i386.so.1 -o ../../dist/bin/firefox /usr/lib/Scrt1.o /usr/lib/crti.o /usr/bin/../lib/gcc/i586-alpine-linux-musl/12.2.1/crtbeginS.o -L/usr/bin/../lib/gcc/i586-alpine-linux-musl/12.2.1 -L/usr/bin/../lib/gcc/i586-alpine-linux-musl/12.2.1/../../../../i586-alpine-linux-musl/lib -L/lib -L/usr/lib /home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/browser/app/firefox.list -lpthread -z noexecstack -z text -z relro -Bsymbolic-functions --build-id=sha1 -rpath-link /home/rakslice/src/aports/community/firefox/src/firefox-108.0.1/obj/dist/bin -rpath-link /usr/lib -latomic -lssp_nonshared -lstdc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /usr/bin/../lib/gcc/i586-alpine-linux-musl/12.2.1/crtendS.o /usr/lib/crtn.o`

it generates broken output like

   444d5:       f0 ff 46 28             lock incl 0x28(%esi)
   444d9:       65 a1 00 00 00 00       mov    %gs:0x0,%eax
   444df:       90                      nop
   444e0:       8d 74 26 00             lea    0x0(%esi,%eiz,1),%esi
   444e4:       00 89 c1 89 a8 f4       add    %cl,-0xb57763f(%ecx)
   444ea:       ff                      (bad)
   444eb:       ff                      (bad)
   444ec:       ff 89 a8 f8 ff ff       decl   -0x758(%ecx)

this mov-nop-lea-no-op is the canned sequence from X86::relaxTlsLdToLe() in lld/ELF/Arch/X86.cpp https://github.com/llvm/llvm-project/blob/llvmorg-15.0.6/lld/ELF/Arch/X86.cpp#L462,

  // Convert
  //   leal foo(%reg),%eax
  //   call ___tls_get_addr
  // to
  //   movl %gs:0,%eax
  //   nop
  //   leal 0(%esi,1),%esi
  const uint8_t inst[] = {
      0x65, 0xa1, 0x00, 0x00, 0x00, 0x00, // movl %gs:0,%eax
      0x90,                               // nop
      0x8d, 0x74, 0x26, 0x00,             // leal 0(%esi,1),%esi
  };

which is basically the LD->LE transition right out of "ELF Handling for Thread-Local Storage" v0.21 (2013-08-22) starting on p.54 (https://lrita.github.io/images/posts/cplusplus/ELF_Handling_For_Thread-Local_Storage.pdf#page=54), edit: actually it's the second one, "The GNU variant" on the next page.

But that particular rewrite is clearly inappropriate; the call here in the GOT ___tls_get_addr case is 6 bytes rather than presumably 5 the canned sequence is expecting so it is a byte short, and leaves us with garbage output.

How lld got to this:

1. lld/ELF/Arch/X86.cpp:92: use expr R_TLSLD_GOTPLT for R_386_TLS_LDM
   https://github.com/llvm/llvm-project/blob/llvmorg-15.0.6/lld/ELF/Arch/X86.cpp#L92
2. lld/ELF/Relocations.cpp:1218: handling of the R_TLSLD_GOTPLT relocation figures that this LD->LE "relaxation" applies https://github.com/llvm/llvm-project/blob/llvmorg-15.0.6/lld/ELF/Relocations.cpp#L1218
3. lld/ELF/Arch/X86.cpp:462: as noted X86::relaxTlsLdToLe() applies canned code block presumably designed for a different call than the one we have https://github.com/llvm/llvm-project/blob/llvmorg-15.0.6/lld/ELF/Arch/X86.cpp#L462

Anyway, there you go. I don't know if this .o is ABI compliant, I don't know if this is a valid candidate for some other LD->LE transition, I don't know what a minimal example of TLS-using code that produces the lea-call with these relocs would look like, I'm just an end user chasing after a broken binary I got shipped.

[rakslice]

Perhaps I should mention just skipping the if (toExecRelax) { block at Relocations.cpp:1221 https://github.com/llvm/llvm-project/blob/llvmorg-15.0.6/lld/ELF/Relocations.cpp#L1221, forcing it not to attempt this LD->LE transition, gives me a working binary, but I don't know if it's correct to still have LD (or whatever this actually is) in the output executable.

[rakslice]

FYI the downstream issue is: https://gitlab.alpinelinux.org/alpine/aports/-/issues/14395

[MaskRay]

Your build uses gcc -fno-plt and the TLS GD/LD model uses call *___tls_get_addr@GOT(%edx). Clang doesn't do this.
I wonder why you use -fno-plt. It's usually a pessimization instead of an optimization (https://maskray.me/blog/2021-09-19-all-about-procedure-linkage-table#fno-plt).

I can add the support to lld x86-32. The x86-64 port does support GOTPCRELX __tls_get_addr calls.

[nekopsykose]

Your build uses gcc -fno-plt and the TLS GD/LD model uses call *___tls_get_addr@GOT(%edx). Clang doesn't do this.

the above build is with clang, no? (clang++ -fno-plt -fuse-ld=lld ...)

I wonder why you use -fno-plt. It's usually a pessimization instead of an optimization

i had added it to shave a subpercent off binary size- i didn't think it was actually broken on 32-bit x86. fooled myself again :)

[MaskRay]

@rakslice Try https://reviews.llvm.org/D140813

[rakslice]

@rakslice Try https://reviews.llvm.org/D140813

Looks good. The new output for this example is

   44495:       f0 ff 46 28             lock incl 0x28(%esi)
   44499:       65 a1 00 00 00 00       mov    %gs:0x0,%eax
   4449f:       8d b6 00 00 00 00       lea    0x0(%esi),%esi
   444a5:       89 c1                   mov    %eax,%ecx
   444a7:       89 a8 f4 ff ff ff       mov    %ebp,-0xc(%eax)

The resulting binary works.

[llvmbot]

@llvm/issue-subscribers-lld-elf

[MaskRay]

Your build uses gcc -fno-plt and the TLS GD/LD model uses call *___tls_get_addr@GOT(%edx). Clang doesn't do this.

the above build is with clang, no? (clang++ -fno-plt -fuse-ld=lld ...)

OK, I actually added -fno-plt for __tls_get_addr in 2019 but disabled it when x86 relax relocations were disabled to work around a GNU ld issue (fixed in binutils in 2016-06).
There was a difference between clang -fno-plt -S and clang -fno-plt -c. Fixed by 83cec14

I wonder why you use -fno-plt. It's usually a pessimization instead of an optimization

i had added it to shave a subpercent off binary size- i didn't think it was actually broken on 32-bit x86. fooled myself again :)

-fno-plt will usually increase the binary size :)