Provide dedicated Spring Boot starter for JUnit 5

[odrotbohm]

When trying to port a Spring Boot project that has previously used JUnit 4 to JUnit 5 one needs to add quite a few additional dependencies to basically get back to the very same functionality that one got with the standard test starter. This is due to the fact that in JUnit 5, a lot of functionality that had been included in JUnit itself was extracted into extensions that now live in third party projects that require explicit inclusion.

Most notabily it are the following additional libraries:

• org.junit.jupiter:junit-jupiter-params for parameterized tests
• org.mockito:mockito-junit-jupiter for @Mock
• org.hamcrest:hamcrest-junit for assertThat(…)

Especially the latter is quite nasty as it in turn depends on JUnit 4.12, so that an explicit exclude is needed.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-test</artifactId>
  <scope>test</scope>
  <exclusions>
    <exclusion>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
    </exclusion>
  </exclusions>
</dependency>

<dependency>
  <groupId>org.junit.jupiter</groupId>
  <artifactId>junit-jupiter-engine</artifactId>
  <scope>test</scope>
</dependency>

<dependency>
  <groupId>org.junit.jupiter</groupId>
  <artifactId>junit-jupiter-params</artifactId>
  <scope>test</scope>
</dependency>

<!-- 
  As indicated by Marc below this could be replaced by
  hamcrest-core to avoid the exclusion
-->
<dependency>
  <groupId>org.hamcrest</groupId>
  <artifactId>hamcrest-junit</artifactId>
  <version>2.0.0.0</version>
  <scope>test</scope>
  <exclusions>
    <exclusion>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
    </exclusion>
  </exclusions>
</dependency>

<dependency>
  <groupId>org.mockito</groupId>
  <artifactId>mockito-junit-jupiter</artifactId>
  <scope>test</scope>
</dependency>

Be sure to read the note on that list below.

It would be cool if that could be reduced to only declare a spring-boot-starter-test-junit5 (name tbd) instead of spring-boot-starter-test.

[wilkinsona]

@olivergierke and I discussed this on Slack last week. A couple of thoughts from me from that discussion:

If we wanted to provide a starter for JUnit Jupiter, I guess we’d split some of spring-boot-starter-test out into spring-boot-starter-junit-4 and then have a spring-boot-starter-junit-jupiter that it could be swapped out for. That’d be similar to what we do for the embedded containers and spring-boot-starter-web

It’s a tricky one. An extra starter would probably have made your life easier but it adds complexity to Boot to solve a problem that’s really in JUnit. It’ll be fine when we flip everyone over to JUnit Jupiter, but I’m not sure things are ready for that just yet.

One additional thought:

The example above makes things look one dependency worse then they actually are. junit-jupiter-api is a transitive dependency of junit-jupiter-engine so need not be declared explicitly.

[wilkinsona]

I concluded with the following:

It’s probably worth opening an issue to see what the rest of the team thinks
I suspect they’ll be against it, but the point you made about the params dependency and the mockito dependency is a good one

[odrotbohm]

Thanks for the API JAR catch, I've removed that one. Regarding the naming, I thought that aligning with …-junit4 and …-junit5 is probably better than something …-jupiter as that's leaking some internal code name into the user's projects. I.e. users a re rather thinking in "I want to upgrade to JUnit 5" rather than "I want to migrate to the Jupiter engine".

[snicoll]

While I understand there is slightly more work than expected to use JUnit Jupiter, I am not keen to branch our current starter (or any starter for that matter). I agree with @wilkinsona that this adds complexity here for a problem that's in JUnit.

[odrotbohm]

… that this adds complexity here for a problem that's in JUnit.

Hm, isn't basically every Spring Boot starter an attempt to solve that problem for some third party library? I.e. what else than an attempt to solve exactly that problem are starters in the first place?

How do you mean branch? Wouldn't it be a new starter (i.e. a single additional POM file) containing just a slightly different setup of JUnit libraries? It's not that new major versions of JUnit pop up every other week, is it?

If you think it's too complex, fine. I just thought it felt weird to have to declare 5 explicit dependencies instead of 1, one of which even needs an explict version number if I wanted to use the current version of the most commonly used Java testing framework. That's quite at odds with the rest of the Spring Boot dependency management experience.

[snicoll]

How do you mean branch?

What I mean is having a "v4" and "v5" starter of the same feature. Regardless of how we name the new one, it'll become irrelevant once we decide to switch the current starter to JUnit 5. That's an odd situation I'd like to avoid if we can.

[wilkinsona]

I just thought it felt weird to have to declare 5 explicit dependencies instead of 1, one of which even needs an explict version number if I wanted to use the current version of the most commonly used Java testing framework.

It may be the current version as in the latest, but it isn't the most widely used, even for greenfield development. I'm pretty sure that honour still belongs to JUnit 4.

That's quite at odds with the rest of the Spring Boot dependency management experience.

Our goal is to make common use cases as easy as possible without preventing more complex or less typical scenarios. In this situation the common use case is to use JUnit 4 and the less typical scenario is the use of JUnit 5. As you've demonstrated above, using JUnit 5 is possible. It's more complex than you'd like but that's almost entirely due to JUnit 5's structure and the rest of the ecosystem arguably not being JUnit 5-ready just yet.

The only way that I can think of to simplify the situation for those who want to use JUnit 5 is to introduce another starter. That could either be two separate top-level starters, or it could be a lower-level starter that's swapped in much as we do with spring-boot-starter-undertow. No matter which option we choose, we've added complexity for users who are happily using JUnit 4 today. I think the question becomes one of balance. Do we think it's worth making things more complex for those who are happy with the status quo to make things easier for those who want to use JUnit 5? My opinion is that we haven't yet reached the tipping point where the extra complexity is justified.

[odrotbohm]

I'd argue this is a chicken and egg problem. If we only provide spring-boot-starter-test, that's of course what people use. If there was a starter working with JUnit 5, wouldn't that significantly increase the chance that projects that get started on Boot 2.x would use just that? How do we even measure "JUnit 5-ready"? Currently there's not even a way for us to track that, is there?

It feels like the team is not eager to add support for JUnit 5 right now. Let's use this ticket to see whether other parties are interested and gather further feedback.

[wilkinsona]

How do we even measure "JUnit 5-ready"? Currently there's not even a way for us to track that, is there?

I think there is. It's things like IDE support (largely there now, but will require adoption of new IDE versions) and build system support (likewise), in addition to things like Hamcrest not assuming the use of JUnit 4.

[asarkar]

I’ve come across the same problem (Boot lacking OOTB JUnit 5 support), with one difference. I used AssertJ, not Hamcrest, so I think not automatically bringing in Hamcrest and leaving that choice to the user might be a good thing. The way I see it, neither Hamcrest nor AssertJ are necessary to write the tests; in fact, JUnit 5 has some new assertions by itself

[marcphilipp]

Regarding Hamcrest: I consider hamcrest-junit an experiment that never quite took off. Thus, depending on hamcrest-core (which includes MatcherAssert.assertThat()) should be enough and simpler because it does not depend on JUnit.

[filiphr]

It may be the current version as in the latest, but it isn't the most widely used, even for greenfield development. I'm pretty sure that honour still belongs to JUnit 4.

I am not entirely certain that the honour belongs to JUnit 4. All recent projects that I have recently started are on JUnit 5 (with all the hoops one needs to do to support that in Spring Boot).

I would agree with @olivergierke that having a spring-boot-starter-junit5-test (name not really important) would make it really easy for people to use Spring Boot and JUnit 5. At least we won't need to add mockito, params and the engine as a dependency. There are already 3 different starters for web (tomcat, jetty and undertow), so why not do the same here.

I think there is. It's things like IDE support (largely there now, but will require adoption of new IDE versions) and build system support (likewise), in addition to things like Hamcrest not assuming the use of JUnit 4.

How can you measure JUnit 5 adoption for Spring Boot? The IDE support is already there and I think that users are already on the new IDE versions. IntelliJ has support since 2016.2 which is more than 2 years now and eclipse since Oxygen 1.a which is more than a year now. Since the 2.22.0 version of the maven-surefire-plugin there is ootb support for JUnit 5 in maven, and Gradle has native support since 4.6.

What exactly is the complexity for the Spring Boot team for this?

[philwebb]

I'm personally not keen to add an additional JUnit 5 starter because I think at some point (probably 2.2) we'll be looking to upgrade the existing test starter to JUnit 5 (probably whilst also trying to support JUnit 4 based tests). Once we do that, it would be nice if we didn't also have the additional job of deprecating a spring-boot-starter-jupiter dependency.

Looking again at the original issues that @olivergierke raised, he wants to add the following:

junit-jupiter-params

I'm quite surprised this one is a module at all, given that it appears to be very commonly used. I wonder if the JUnit team have considered publishing a POM themselves that pulls in junit-jupiter-api, junit-jupiter-engine and junit-jupiter-params. It seems like this trio is a supper common combination (perhaps @sbrannen knows)

org.mockito:mockito-junit-jupiter

This jar is super light so we could add it to the existing starter and exclude junit-jupiter-api.

org.hamcrest:hamcrest-junit

As @marcphilipp mentioned above, using MatcherAssert from hamcrest-core is a better option here and I don't think we will add a hamcrest-junit managed dependency. I also tend to recommend AssertJ these days if asked.

[philwebb]

I've raised junit-team/junit5#1629. If that's accepted and we pull in mockito-junit-jupiter most of the pain would go away.