Unit testing framework
Download as PPT, PDF0 likes938 views
The document discusses various unit testing frameworks for .NET such as MSTest v2, NUnit, and xUnit.net. It provides information on the key features of each framework, how to install and set them up, how to write tests, and comparisons between the frameworks. The document recommends NUnit as it has the richest set of assertions, supports data-driven testing well, and has full documentation available, which would be important for a new development team to learn to use the framework effectively.
![• Excellent availability of learning resources
Due to it's popularity and active development, plenty of
learning resources (such as detailed documentation and
various tutorials) exist for learning NUnit.
• Widely used
NUnit is one of the most popular testing frameworks
for .NET. Other than giving a certain sense of security in
the continuation of the project, it also means that there
are a lot of third-party resources, guides and tutorials
available for NUnit.
• The constraint-based Assert model[1].
Nunit 3.0](https://image.slidesharecdn.com/unittestingframework-181218194433/85/Unit-testing-framework-6-320.jpg)
![xUnit.net
• Single object instance per test method: It allows complete isolation of test methods that
allow developers to independently run tests in any order.
• No Setup & Teardown attributes: These methods are decorated with attributes named
SetUp and TearDown respectively. The down side is that it unnecessarily creates confusion
and make developers to hunt around the presence and absence of such methods. It is
more productive to write code set up and tear down code within test methods itself.
Hence, in xUnit.Net pink slip is given to these unproductive attributes.
• Reduced set of attributes: For example, unlike nUnit it doesn't require [TextFixture] &
[TestMethod] attributes to declare a test while it requires only [Fact] attribute decorated
on test method.
• xUnit’s terminology adheres TDD closely (Fact, Theory)
• Automation: xUnit has great ability for test automation and can smoothly work in
conjunction with other testing framework. It is highly extensible and open for
customization.
• Lack of documentation
• Supported and used by Microsoft itself(?)](https://image.slidesharecdn.com/unittestingframework-181218194433/85/Unit-testing-framework-7-320.jpg)
![Data driven testing
MSTest v2 NUnit xUnit.net
Inline data
DataRow,
DynamicData TestCase Attribute [Theory]
Separate DataSource TestCaseSource Attribute Xunit.Sdk.DataAttribute
Objectivity.Test.Automation
+ + +](https://image.slidesharecdn.com/unittestingframework-181218194433/85/Unit-testing-framework-9-320.jpg)
![NUnit 3.x MSTest 15.x xUnit.net 2.x Comments
[Test] [TestMethod] [Fact] Marks a test method.
[TestFixture] [TestClass] n/a
xUnit.net does not require an attribute for a test class; it looks for
all test methods in all public (exported) classes in the assembly.
Assert.That
[ExpectedException]
Assert.Throws
xUnit.net has done away with the ExpectedException attribute in
favor of Assert.Throws.Record.Exception Record.Exception
[SetUp] [TestInitialize] Constructor
We believe that use of [SetUp] is generally bad. However, you
can implement a parameterless constructor as a direct
replacement. See
[TearDown] [TestCleanup] IDisposable.Dispose
We believe that use of [TearDown] is generally bad. However,
you can implement IDisposable.Dispose as a direct replacement.
[OneTimeSetUp] [ClassInitialize] IClassFixture<T>
To get per-class fixture setup, implement IClassFixture<T> on
your test class.
[OneTimeTearDown] [ClassCleanup] IClassFixture<T>
To get per-class fixture teardown, implement IClassFixture<T> on
your test class.
n/a n/a ICollectionFixture<T>
To get per-collection fixture setup and teardown,
implement ICollectionFixture<T> on your test collection.
[Ignore("reason")] [Ignore] [Fact(Skip="reason")]
Set the Skip parameter on the [Fact] attribute to temporarily skip a
test.
[Property] [TestProperty] [Trait] Set arbitrary metadata on a test
Attributes](https://image.slidesharecdn.com/unittestingframework-181218194433/85/Unit-testing-framework-11-320.jpg)
![Performance of frameworks
These were timed using benchmark solutions of 10,000 tests. Visual Studio 2017 15.5 Preview 2 + NUnit
Adapter v3.9.0, published on 11 Oct 2017 + xUnit v2.3.1 published on 27 Oct 2017. [15]
Test Framework Test Discovery (secs) Test Execution (secs)
xUnit.net 6.4 68
NUnit 5.5 16.7
MSTest v2 5.2 11.74](https://image.slidesharecdn.com/unittestingframework-181218194433/85/Unit-testing-framework-15-320.jpg)
Unit testing framework
- 2. Agenda
- 3. …with Mocks Mock Library In Isolation Unit Testing Replace Dependencies… Dependency Injection Framework Automatic Unit Testing Framework mbUnit nUnit xUnit.Net Mstest v2 By hand Spring.Net Castle/Windsor Unity StructureMap nMock EasyMock Rhino.Mocks TypeMock Continuous Integration CrouiseControl.Net VS Team System GitLab CI Microsoft Azure etc.
- 4. Microsoft Test Framework "MSTest V2” - Open Source Unit Testing framework evolved from the MSTest framework - URL: https://github.com/Microsoft/testfx xUnit.net - Open Source Unit Testing framework written by original inventor of Nunit v2. URL: https://xunit.github.io/ Nunit – Open Source Unit Testing for .NET initialy ported from JUnit. URL: http://www.unit.org/ Unit Testing Frameworks for .NET
- 5. MSTest v2 • MSTest v2 is fully integrated with Visual Studios and works natively without the need for any plugins. • MSTest v2 is better suited for only using Microsoft technologies rather than mixed technology environments.
- 6. • Excellent availability of learning resources Due to it's popularity and active development, plenty of learning resources (such as detailed documentation and various tutorials) exist for learning NUnit. • Widely used NUnit is one of the most popular testing frameworks for .NET. Other than giving a certain sense of security in the continuation of the project, it also means that there are a lot of third-party resources, guides and tutorials available for NUnit. • The constraint-based Assert model[1]. Nunit 3.0
- 7. xUnit.net • Single object instance per test method: It allows complete isolation of test methods that allow developers to independently run tests in any order. • No Setup & Teardown attributes: These methods are decorated with attributes named SetUp and TearDown respectively. The down side is that it unnecessarily creates confusion and make developers to hunt around the presence and absence of such methods. It is more productive to write code set up and tear down code within test methods itself. Hence, in xUnit.Net pink slip is given to these unproductive attributes. • Reduced set of attributes: For example, unlike nUnit it doesn't require [TextFixture] & [TestMethod] attributes to declare a test while it requires only [Fact] attribute decorated on test method. • xUnit’s terminology adheres TDD closely (Fact, Theory) • Automation: xUnit has great ability for test automation and can smoothly work in conjunction with other testing framework. It is highly extensible and open for customization. • Lack of documentation • Supported and used by Microsoft itself(?)
- 8. Installation MSTest v2 NUnit x.Unit.net Full MSTest v2install via NuGet. Full NUnit install via NuGet. Full x.Unit.net install via NuGet. n/a NUnitLite install via NuGet. n/a n/a Zip and/or MSI file download. n/a n/a Combined Approach n/a
- 9. Data driven testing MSTest v2 NUnit xUnit.net Inline data DataRow, DynamicData TestCase Attribute [Theory] Separate DataSource TestCaseSource Attribute Xunit.Sdk.DataAttribute Objectivity.Test.Automation + + +
- 10. Writing tests MSTest v2 NUnit x.Unit.net Assumptions Assert.Inconclusive Assumptions Xunit.SkippableFact Multiple Asserts + + n/a Warnings n/a Warn class and the Assert.Warn n/a Constraints CollectionAssert Members Collection Constraints n/a TestCaseData n/a TestCaseData n/a TestFixtureData n/a TestFixtureData n/a TestContext n/a TestContext n/a n/a Static Properties : CurrentContext n/a n/a Out n/a n/a Error n/a n/a Progress n/a n/a TestParameters n/a n/a Static Methods : Write n/a n/a WriteLine n/a n/a AddFormatter (3.2+) n/a n/a AddTestAttachment (3.7+) n/a n/a Properties of the CurrentContext : Test n/a n/a Result n/a n/a TestDirectory n/a n/a WorkDirectory n/a n/a Random n/a AssertionHelper n/a NUnit.StaticExpect n/a ListMapper n/a ListMapper n/a
- 11. NUnit 3.x MSTest 15.x xUnit.net 2.x Comments [Test] [TestMethod] [Fact] Marks a test method. [TestFixture] [TestClass] n/a xUnit.net does not require an attribute for a test class; it looks for all test methods in all public (exported) classes in the assembly. Assert.That [ExpectedException] Assert.Throws xUnit.net has done away with the ExpectedException attribute in favor of Assert.Throws.Record.Exception Record.Exception [SetUp] [TestInitialize] Constructor We believe that use of [SetUp] is generally bad. However, you can implement a parameterless constructor as a direct replacement. See [TearDown] [TestCleanup] IDisposable.Dispose We believe that use of [TearDown] is generally bad. However, you can implement IDisposable.Dispose as a direct replacement. [OneTimeSetUp] [ClassInitialize] IClassFixture<T> To get per-class fixture setup, implement IClassFixture<T> on your test class. [OneTimeTearDown] [ClassCleanup] IClassFixture<T> To get per-class fixture teardown, implement IClassFixture<T> on your test class. n/a n/a ICollectionFixture<T> To get per-collection fixture setup and teardown, implement ICollectionFixture<T> on your test collection. [Ignore("reason")] [Ignore] [Fact(Skip="reason")] Set the Skip parameter on the [Fact] attribute to temporarily skip a test. [Property] [TestProperty] [Trait] Set arbitrary metadata on a test Attributes
- 12. NUnit 3.x (Constraint) MSTest 15.x xUnit.net 2.x Comments Is.EqualTo AreEqual Equal MSTest and xUnit.net support generic versions of this method Is.Not.EqualTo AreNotEqual NotEqual MSTest and xUnit.net support generic versions of this method Is.Not.SameAs AreNotSame NotSame Is.SameAs AreSame Same Does.Contain Contains Contains Does.Not.Contain DoesNotContain DoesNotContain Throws.Nothing n/a DoesNotThrow Ensures that the code does not throw any exceptions n/a Fail n/a xUnit.net alternative: Assert.True(false, "message") Is.GreaterThan n/a n/a xUnit.net alternative: Assert.True(x > y) Is.InRange n/a InRange Ensures that a value is in a given inclusive range Is.AssignableFrom n/a IsAssignableFrom Is.Empty n/a Empty Is.False IsFalse False Is.InstanceOf<T> IsInstanceOfType IsType<T> Is.NaN n/a n/a xUnit.net alternative: Assert.True(double.IsNaN(x)) Is.Not.AssignableFrom<T> n/a n/a xUnit.net alternative: Assert.False(obj is Type) Is.Not.Empty n/a NotEmpty Is.Not.InstanceOf<T> IsNotInstanceOfType IsNotType<T> Is.Not.Null IsNotNull NotNull Is.Null IsNull Null Is.True IsTrue True Is.LessThan n/a n/a xUnit.net alternative: Assert.True(x < y) Is.Not.InRange n/a NotInRange Ensures that a value is not in a given inclusive range Throws.TypeOf<T> n/a Throws<T> Ensures that the code throws an exact exception Asserts
- 13. ExtendingMSTest v2 NUnit x.Unit.net Framework Extensibility for Trait Attributes Custom Attributes n/a + Load-Time Interfaces n/a + IFixtureBuilder n/a + ITestBuilder n/a + ISimpleTestBuilder n/a + IParameterDataSource n/a + IImplyFixture n/a + IApplyToTest n/a Extensibility for Custom Test Execution Execution-Time Interfaces Test method extensibility + IApplyToContext n/a + ICommandWrapper n/a n/a Custom Constraints n/a Framework Extensibility for Custom AssertionsCustom Asserts Assert extensibility. n/a Engine Extensibility Project Loaders n/a n/a Result Writers n/a n/a Framework Drivers n/a n/a Event Listeners n/a Framework Extensibility for Custom Test Data Source n/a n/a
- 14. Logging and reporting MSTest v2 NUnit xUnit.net Output + Text Output from Tests Spec + Logging log4net log4net log4net Nlog Nlog Nlog ReportUnit reportunit reportunit n/a Allure Test Report + + + Test Results in XML n/a + + Performance n/a n/a xunit.performance Nbench Nbench Nbench Record n/a NUnit Video Recorder n/a
- 15. Performance of frameworks These were timed using benchmark solutions of 10,000 tests. Visual Studio 2017 15.5 Preview 2 + NUnit Adapter v3.9.0, published on 11 Oct 2017 + xUnit v2.3.1 published on 27 Oct 2017. [15] Test Framework Test Discovery (secs) Test Execution (secs) xUnit.net 6.4 68 NUnit 5.5 16.7 MSTest v2 5.2 11.74
- 16. Other parameters MSTest v2 NUnit xUnit.net Testing against multiple frameworks <TargetFrameworks> -- framework command <TargetFrameworks> Xamarin n/a + + Execute tests in parallel + + + IntelliTest built-in + +
- 17. Compare Param Max MSTest v2 NUnit 3.0 xUnit.net Easy to learn 5 3 5 2 Support data driven tests 4 4 4 4 Rich set of assertions 3 2 3 2 Extending 3 3 3 3 Fast 2 2 2 1 Sum - 14 17 12
- 18. ChosenChosen
- 19. Conclusion All frameworks have their pros and cons, their detractors and supporters. MSTest v2, Nunit 3.0 and xUnit.net have a few(few) distinctions. But only NUnit 3.0 has full documentation. Whereas our team have been learning yet, we need a lot of information about using new for us framework. So according to the requirements I have chosen NUnit 3.0 as a unit testing framework for our educational project.
- 20. 1. https://www.slant.co/topics/543/~best-unit-testing-frameworks-for-net 2. https://github.com/nunit/docs/wiki/NUnit-Documentation 3. https://github.com/Microsoft/testfx 4. https://xunit.github.io/docs/comparisons.html 5. https://msdn.microsoft.com/en-us/library/ms243147.aspx?f=255&MSPPError=-2147 6. https://docs.microsoft.com/en-us/visualstudio/test/unit-test-your-code?view=vs-2017 7. https://damsteen.nl/blog/2016/06/04/comparing-dotnet-testing-frameworks 8. https://www.automatetheplanet.com/nunit-cheat-sheet 9. https://www.automatetheplanet.com/mstest-cheat-sheet/ 10.https://redmonk.com/fryan/2018/03/26/a-look-at-unit-testing-frameworks/ 11.https://dzone.com/articles/using-xunit-mstest-or-nunit-to-test-net-core-libra 12.https://mitra.computa.asia/articles/msdn-evolving-visual-studio-test-platform- part-3-net-core-convergence-and-cross-plat 13.https://dev.to/franndotexe/parallel-test-execution-within-the-context-of- mstestv2-nunit3-and-vstestconsole-3f39 14.https://www.meziantou.net/2018/03/01/mstest-v2-execute-tests-in-parallel 15.https://blogs.msdn.microsoft.com/visualstudio/2017/11/16/test-experience- improvements 16.https://improveandrepeat.com/2018/03/xunit-net-cheat-sheet-for-nunit-users/ Sources
- 21. Questions?
- 23. Test frameworks vs runners vs libs • Test Runners: xUnit.net, NUnitLite Runner, nunit-gui, xunit.console, nunit3-console, MSBuild, Visual Studio Devices, $TestDriven.NET, $Meissa • Testing Frameworks. • Assertion Libraries: Fluent Assertions
- 24. Supplementing, not replacing, MSTest with another testing framework
- 25. MSTest vs MSTest v2 (duration tests)
- 26. Company and Community make contribution to NUnit
- 27. Company and Community make contribution to xUnit
- 28. GitHub
- 29. Nuget
- 30. Test Discovery And Execution in NUnit
- 31. Two different major types of unit tests in xUnit.net
Editor's Notes
- #12: CollectionAssert Members has the same elements as Nunit but it is uncategorized
- #14: The XUnit Assert class works differently than the one in NUnit. XUnit will throw an exception on an assertion failure, whereas NUnit reports an error to the test execution context.
- #18: Parallelism in Test Frameworks Parallelism in Runners Parallelism via Configuration https://github.com/nunit/docs/wiki/Parallelizable-Attribute