Hit "Inconsistent kallsyms data" for riscv64 debug config

[fbq]

gcc: 9.0
rustc: 1.54.0

Hit a "Inconsistent kallsyms data" in the checks:

https://github.com/Rust-for-Linux/linux/runs/3677382723?check_suite_focus=true

I added a simple debug output to show the diff between two ".map":

diff --git a/scripts/link-vmlinux.sh b/scripts/link-vmlinux.sh
index 36ef7b37fc5d..5eb7531aea04 100755
--- a/scripts/link-vmlinux.sh
+++ b/scripts/link-vmlinux.sh
@@ -440,6 +440,7 @@ if [ -n "${CONFIG_KALLSYMS}" ]; then
        if ! cmp -s System.map .tmp_System.map; then
                echo >&2 Inconsistent kallsyms data
                echo >&2 Try "make KALLSYMS_EXTRA_PASS=1" as a workaround
+               diff >&2 .tmp_System.map System.map
                exit 1
        fi
 fi

and result was

Inconsistent kallsyms data
Try make KALLSYMS_EXTRA_PASS=1 as a workaround
--- .tmp_System.map 2021-09-22 16:00:21.593766590 +0000
+++ System.map 2021-09-22 16:00:21.541766045 +0000
@@ -42127,8 +42127,8 @@
ffffffff8086fe08 R kallsyms_relative_base
ffffffff8086fe10 R kallsyms_num_syms
ffffffff8086fe18 R kallsyms_names
-ffffffff8098d310 R kallsyms_markers
-ffffffff8098d688 R kallsyms_token_table
+ffffffff8098d308 R kallsyms_markers
+ffffffff8098d680 R kallsyms_token_table
ffffffff8098dbe0 R kallsyms_token_index
ffffffff809b25e0 R __begin_sched_classes
ffffffff809b25e0 R idle_sched_class

Some analysis:

link-vmlinux.sh will first run kallsyms (script/kallsyms.c) twice to generate the tables for symbols of the kernel (for kernel itself to lookup), the program kallsyms will try to compress the tables with a simple algorithm, and the tables may be different between different runs (link-vmlinux.sh call it steps). Program kallsyms reads the symbol table of the kernel image, and generate a .S file, which eventually be compiled into an object file. The object file will be linked into the kernel image.

Currently the logic is that if the size of the object file stays the same in step 1 & 2, the compilation continues, otherwise, do an extra run for kallsyms. And in kallsyms data consistency checking, the script compares the address of every symbol (including kallsyms symbols) between the final kernel image and the result of the previous kallsyms step.

What we are hitting here is that the sizes of the object file of step 1 & 2 are the same (so no step 3), but the contents are different. Moreover, kallsyms_names is the table to store kernel symbols, and kallsyms_token_table is the table to store data for compression, so according to the .map diff above, the case we are hitting is the kernel symbols (compressed data) space shrinks 8bytes while the compress utility data space increases 8bytes.

The issue is very hard to reproduce since a small change in symbol table can result in different compressed results. Actually when I tried to randomize (by hand) some function names introduced in my PR, the problem was gone. FWIW, my original PR that hit the issue is as follow: https://github.com/fbq/linux-rust/tree/debug/kallsym-issue

[ojeda]

Thanks a lot for taking a look at this -- I also hit this yesterday with the sync PR (after the CI servers stopped timing out due to networking issues) when trying to reproduce locally the hang on QEMU that happened in the CI.

[ojeda]

In my case, the sizes 1 & 2 are different, and so there is a step 3, but the extra step does not help. There is a big difference in size between step 1 and 2 (856 bytes), which then gets reduced to a few bytes between 2 and 3 (16 bytes). Doing a 4th pass finds the fixed point, though.

So we have 2 issues: this one and the one you report above (that the check is too lax and does not notice that step 3 is required). I assume that a forced step 3 (i.e. KALLSYMS_EXTRA_PASS=1) helped you?

[ojeda]

One more case: the sizes increase quite a lot (rather than decrease, which is what I saw so far), requiring 5 passes.

[ojeda]

One more: there is no fixed point, it oscillates between two sizes. Great!

[fbq]

One more: there is no fixed point, it oscillates between two sizes. Great!

;-) While I was looking at this issue, I also found that I cannot prove that the process has a fix point.

[fbq]

So we have 2 issues: this one and the one you report above (that the check is too lax and does not notice that step 3 is required). I assume that a forced step 3 (i.e. KALLSYMS_EXTRA_PASS=1) helped you?

Right, I remembered that I did comment out the if statement in scripts/linux-vmlinux.sh, which in fact forced step 3, and the CI passed after that.

[nbdd0121]

One more: there is no fixed point, it oscillates between two sizes. Great!

I suppose RISC-V linker relaxation has something to do with this. What's the diff between the two?

[ojeda]

I suppose RISC-V linker relaxation has something to do with this. What's the diff between the two?

The first few got shifted by 8, everything else after that by 0x20 (alignment, I guess):

  ffffffff809afd80 R kallsyms_relative_base
  ffffffff809afd88 R kallsyms_num_syms
  ffffffff809afd90 R kallsyms_names
+ ffffffff80cc56c8 R kallsyms_markers
- ffffffff80cc56d0 R kallsyms_markers
+ ffffffff80cc6e08 R kallsyms_token_table
- ffffffff80cc6e10 R kallsyms_token_table
+ ffffffff80cc73e8 R kallsyms_token_index
- ffffffff80cc73f0 R kallsyms_token_index
+ ffffffff80ccd797 r trunc_msg
- ffffffff80ccd79f r trunc_msg
+ ffffffff80cde793 r resource_string.io_spec
+ ffffffff80cde79b r resource_string.mem_spec
+ ffffffff80cde7a3 r resource_string.bus_spec
- ffffffff80cde7b3 r resource_string.io_spec
- ffffffff80cde7bb r resource_string.mem_spec
- ffffffff80cde7c3 r resource_string.bus_spec
+ ffffffff80ce3dc0 R __begin_sched_classes
+ ffffffff80ce3dc0 R idle_sched_class
- ffffffff80ce3de0 R __begin_sched_classes
- ffffffff80ce3de0 R idle_sched_class
...
(everything from here on)

I will see tomorrow what are the contents.