misspelling a test name should probably cause failure

[oconnor663]

(Originally filed as rust-lang/cargo#3009.)

I'm worried that it's very easy to misspell a test's name, see the (zero tests) success output from cargo, and miss the fact that your test didn't run. Here's a couple examples from my project. First the correct spelling:

$ cargo test test_file
     Running target/debug/duct-0b3d1841f083e660

running 1 test
test test::test_file ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

   Doc-tests duct

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

And misspelled, so that no tests actually run:

$ cargo test tast_file
     Running target/debug/duct-0b3d1841f083e660

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

   Doc-tests duct

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

You can't see it here on GitHub, but the strongest detail in the terminal is that most of the green colored text (the ok's) is the same in both examples. Also because doc tests come after, and I have no doc tests in this project, the last three lines are also the same.

I can't think of a use case where you'd want the test runner to silently do nothing. Can we make running no tests an error?

[nagisa]

The test name isn’t matched exactly though. Are you suggesting that we should fail in case we match no tests? What about the use case of having multiple crates and running some wrapper test_all_crates ]test_name] script which only might match some tests from one crate?

[oconnor663]

Are you suggesting that we should fail in case we match no tests?

I think so, or at least I want to make sure we've considered it. It might be possible to support both use cases by adding a --matching-no-tests-is-fine-i-know-what-im-doing flag? Maybe the use case for a flag like that is too obscure to justify adding it, but that could also be an argument for not making it the default behavior :-D

From anecdotal experience using go test --run ..., it's pretty common to get confused with a spelling or capitalization mistake in the argument there. The Rust test framework is already better off in that it at least prints the number of tests it ran, but the green ok status tends to draw attention away from that. Another possible middle ground could be to make it print a warning, or a yellow ok or something, but not return a failing status? Humans don't usually care about the exit status in any case, though I'm not sure what script authors want from the output here.

For a similar example, fish and zsh both return errors by default when a glob doesn't match any files. That's not very friendly to scripts, but it optimizes for not surprising the human.

[Mark-Simulacrum]

Closing in favor of #31905.