compiletest: Support matching on non-json lines in compiler output

[petrochenkov]

and migrate most of remaining error-patterns to it.
Such diagnostics use a new diagnostic kind RAW.

Also emit an error for error-patterns that can be replaced with line annotations.
Also remove a number of conditions to check both line annotations and error-patterns in more cases.
Also respect //@ check-stdout when collecting "actual errors" for comparing against line annotations.
(A couple of tiny refactorings is also included.)

Continuation of #139760.

r? @jieyouxu

[rustbot]

The rustc-dev-guide subtree was changed. If this PR only touches the dev guide consider submitting a PR directly to rust-lang/rustc-dev-guide otherwise thank you for updating the dev guide with your changes.

cc @BoxyUwU, @jieyouxu, @Kobzol

Some changes occurred in src/tools/compiletest

cc @jieyouxu

[jieyouxu]

Thanks! I think RAW annotations are much nicer to use than //@ error-pattern. I noticed a few strange diffs, left some discussion for them.

[jieyouxu]

@rustbot author

[rustbot]

Reminder, once the PR becomes ready for a review, use @rustbot ready.

[petrochenkov]

@rustbot ready

[jieyouxu]

Thanks

[jieyouxu]

@bors r+ rollup

[bors]

📌 Commit f33c69e has been approved by jieyouxu

It is now in the queue for this repository.

[jieyouxu]

Actually, new/modified tests in a rollup might conflict, so
@bors rollup=never

[petrochenkov]

@bors r=jieyouxu

[bors]

📌 Commit 20d5cd3 has been approved by jieyouxu

It is now in the queue for this repository.

[bors]

⌛ Testing commit 20d5cd3 with merge 6317883...

[petrochenkov]

@bors r=jieyouxu

[bors]

📌 Commit 56d6b4e has been approved by jieyouxu

It is now in the queue for this repository.

[bors]

⌛ Testing commit 56d6b4e with merge 13e8790...

[bors]

☀️ Test successful - checks-actions
Approved by: jieyouxu
Pushing 13e8790 to master...

[github-actions]

What is this?

This is an experimental post-merge analysis report that shows differences in test outcomes between the merged PR and its parent PR.

Comparing ab62d56 (parent) -> 13e8790 (this PR)

Test differences

Show 2 test diffs

Stage 2

• [incremental] tests/incremental/issue-84252-global-alloc.rs: pass -> ignore (skipping test as target does not support all of the crate types ["cdylib"]) (J0)
• [incremental] tests/incremental/no_mangle.rs: pass -> ignore (skipping test as target does not support all of the crate types ["cdylib"]) (J0)

Job group index

• J0: dist-i586-gnu-i586-i686-musl, test-various

Test dashboard

Run

cargo run --manifest-path src/ci/citool/Cargo.toml -- \
    test-dashboard 13e87909494f2f31bce51bc106e2b629420d1235 --output-dir test-dashboard

And then open test-dashboard/index.html in your browser to see an overview of all executed tests.

Job duration changes

1. dist-apple-various: 5932.8s -> 7542.7s (27.1%)
2. dist-x86_64-apple: 8368.2s -> 10495.6s (25.4%)
3. dist-arm-linux: 4522.6s -> 5167.0s (14.2%)
4. x86_64-apple-1: 6557.5s -> 7321.4s (11.6%)
5. x86_64-apple-2: 4972.4s -> 5272.3s (6.0%)
6. dist-i686-linux: 5988.1s -> 5633.6s (-5.9%)
7. dist-x86_64-illumos: 5747.5s -> 5504.6s (-4.2%)
8. dist-various-1: 4413.4s -> 4595.3s (4.1%)
9. dist-aarch64-apple: 5283.1s -> 5080.3s (-3.8%)
10. i686-gnu-1: 8600.3s -> 8270.9s (-3.8%)

How to interpret the job duration changes?

Job durations can vary a lot, based on the actual runner instance
that executed the job, system noise, invalidated caches, etc. The table above is provided
mostly for t-infra members, for simpler debugging of potential CI slow-downs.

[rust-timer]

Finished benchmarking commit (13e8790): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.4%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.4%	[-0.4%, -0.4%]	1

Max RSS (memory usage)

Results (primary 0.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.6%	[0.4%, 1.3%]	13
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-1.0%	[-2.1%, -0.6%]	4
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[-2.1%, 1.3%]	17

Cycles

Results (primary 0.3%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.6%	[0.4%, 1.0%]	9
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.4%	[-0.5%, -0.4%]	3
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.3%	[-0.5%, 1.0%]	12

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 769.013s -> 769.178s (0.02%)
Artifact size: 365.42 MiB -> 365.41 MiB (-0.00%)