Skip to content

Error building main crate and error building its tests are not differentiated #558

@SNCPlay42

Description

@SNCPlay42

If the old compiler successfully built a crate, but failed to build its tests, and the new compiler failed to build the main crate, that should be reported as a regression (and vice-versa as a fix).

But both failing to build the main crate and failure to build the tests currently result in a BuildFail status (TestFail only comes from failure to run the tests):

crater/src/runner/test.rs

Lines 230 to 280 in 674e4c2

fn build<DB: WriteResults>(
ctx: &TaskCtx<DB>,
build_env: &Build,
local_packages_id: &HashSet<PackageId>,
) -> Fallible<()> {
run_cargo(
ctx,
build_env,
&["build", "--frozen", "--message-format=json"],
true,
local_packages_id,
)?;
run_cargo(
ctx,
build_env,
&["test", "--frozen", "--no-run", "--message-format=json"],
true,
local_packages_id,
)?;
Ok(())
}
fn test<DB: WriteResults>(ctx: &TaskCtx<DB>, build_env: &Build) -> Fallible<()> {
run_cargo(
ctx,
build_env,
&["test", "--frozen"],
false,
&HashSet::new(),
)
}
pub(super) fn test_build_and_test<DB: WriteResults>(
ctx: &TaskCtx<DB>,
build_env: &Build,
local_packages_id: &HashSet<PackageId>,
) -> Fallible<TestResult> {
let build_r = build(ctx, build_env, local_packages_id);
let test_r = if build_r.is_ok() {
Some(test(ctx, build_env))
} else {
None
};
Ok(match (build_r, test_r) {
(Err(err), None) => TestResult::BuildFail(failure_reason(&err)),
(Ok(_), Some(Err(err))) => TestResult::TestFail(failure_reason(&err)),
(Ok(_), Some(Ok(_))) => TestResult::TestPass,
(_, _) => unreachable!(),
})
}

Noticed in rust-lang/rust#80953 (in that case, the new compiler failed the same way as the old one, but if the new compiler did have the regression, it wouldn't have been noticed due to this).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions