Game Programming 06 - Automated Testing
4 likes3,940 views
The document outlines the essentials of automated testing in game programming, focusing on unit testing frameworks like NUnit for .NET and tooling for Unity. It discusses concepts such as test-driven development (TDD), practical examples of unit tests, and limitations of testing, including integration and embedded system challenges. Additionally, it includes references to testing in C++ with Google Test and Unreal Engine 4's automation testing methods.
![NUnit Test Method Design
• [Test] attribute
• Return void
• No parameters
8 / 49](https://image.slidesharecdn.com/gameprogramming-automatedtesting-141027132043-conversion-gate01/85/Game-Programming-06-Automated-Testing-8-320.jpg)
![NUnit Test Example
C#
9 / 49
namespace LevelEditor.Tests
{
using LevelEditor.Model;
using NUnit.Framework;
public class MapTest
{
[Test]
public void TestMapConstructor()
{
// Arrange.
const int Width = 32;
const int Height = 16;
// Act.
Map map = new Map(Width, Height);
// Assert.
Assert.AreEqual(Width, map.Width);
Assert.AreEqual(Height, map.Height);
}
}
}](https://image.slidesharecdn.com/gameprogramming-automatedtesting-141027132043-conversion-gate01/85/Game-Programming-06-Automated-Testing-9-320.jpg)
![Expected Exceptions
C#
11 / 49
[Test]
[ExpectedException(typeof(ArgumentOutOfRangeException))]
public void TestNegativeWidth()
{
Map map = new Map(-10, 20);
}](https://image.slidesharecdn.com/gameprogramming-automatedtesting-141027132043-conversion-gate01/85/Game-Programming-06-Automated-Testing-11-320.jpg)
![SetUp and TearDown
C#
12 / 49
public class MapTest
{
private const int Height = 16;
private const int Width = 32;
private Map map;
[SetUp]
public void SetUp()
{
this.map = new Map(Width, Height);
}
[Test]
public void TestTileIndexer()
{
// Arrange.
const int X = 1;
const int Y = 2;
// Act.
var mapTile = new MapTile(X, Y, "Desert");
this.map[X, Y] = mapTile;
// Assert.
Assert.AreEqual(mapTile, this.map[X, Y]);
}
}](https://image.slidesharecdn.com/gameprogramming-automatedtesting-141027132043-conversion-gate01/85/Game-Programming-06-Automated-Testing-12-320.jpg)
Game Programming 06 - Automated Testing
- 1. Game Programming Automated Testing in Games Nick Prühs
- 2. Objectives • To learn how to properly set up automated testing for your games • To get an overview of common unit testing frameworks and tools 2 / 49
- 3. Unit Testing • Method by which individual units of source code are tested to determine if they are fit for use • Unit of source code is the smallest testable part of an application (e.g. method) • Created by programmers during the development process • Usually split up into three parts: ▪ Arrange ▪ Act ▪ Assert 3 / 49
- 4. Unit Testing • Ideally, each test case is independent from the others • Substitutes such as mocks can be used to assist testing a module in isolation (e.g. database, mails) • Can be implemented as part of automated builds 4 / 49
- 5. Unit Testing with NUnit • Unit Testing framework for all .NET languages • Initially ported from JUnit • Written entirely in C# • Stand-alone tools and R# integration 5 / 49
- 6. Setting up NUnit 1. Add new Class Library project to the solution. 2. Add reference to bin/framework/nunit.framework.dll. 6 / 49
- 7. NUnit Test Class Design • Public • Default constructor 7 / 49
- 8. NUnit Test Method Design • [Test] attribute • Return void • No parameters 8 / 49
- 9. NUnit Test Example C# 9 / 49 namespace LevelEditor.Tests { using LevelEditor.Model; using NUnit.Framework; public class MapTest { [Test] public void TestMapConstructor() { // Arrange. const int Width = 32; const int Height = 16; // Act. Map map = new Map(Width, Height); // Assert. Assert.AreEqual(Width, map.Width); Assert.AreEqual(Height, map.Height); } } }
- 10. NUnit Assertions • AreEqual, AreNotEqual • AreSame, AreNotSame • IsTrue, IsFalse • Greater, GreaterOrEqual • Less, LessOrEqual • IsEmpty, IsNotEmpty • IsNull, IsNotNull • Contains • Fail, Inconclusive 10 / 49
- 11. Expected Exceptions C# 11 / 49 [Test] [ExpectedException(typeof(ArgumentOutOfRangeException))] public void TestNegativeWidth() { Map map = new Map(-10, 20); }
- 12. SetUp and TearDown C# 12 / 49 public class MapTest { private const int Height = 16; private const int Width = 32; private Map map; [SetUp] public void SetUp() { this.map = new Map(Width, Height); } [Test] public void TestTileIndexer() { // Arrange. const int X = 1; const int Y = 2; // Act. var mapTile = new MapTile(X, Y, "Desert"); this.map[X, Y] = mapTile; // Assert. Assert.AreEqual(mapTile, this.map[X, Y]); } }
- 13. Hint Override Equals in types whose objects you need to compare! 13 / 49
- 14. NUnit Test Tool 14 / 49
- 15. NUnit Console Console Output 15 / 49 D:DevRepositoriesSAE-ToolDevelopmentVendorNUnit-2.6.3bin>nunit-console-x86.exe ......SourceLevelEditor.TestsbinDebugLevelEditor.Tests.dll NUnit-Console version 2.6.3.13283 Copyright (C) 2002-2012 Charlie Poole. Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov. Copyright (C) 2000-2002 Philip Craig. All Rights Reserved. Runtime Environment - OS Version: Microsoft Windows NT 6.2.9200.0 CLR Version: 2.0.50727.7905 ( Net 3.5 ) ProcessModel: Default DomainUsage: Single Execution Runtime: net-3.5 ... Tests run: 3, Errors: 0, Failures: 0, Inconclusive: 0, Time: 0.046059608766156 seconds Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0
- 16. NUnit & Visual Studio 16 / 49
- 17. NUnit & Visual Studio 17 / 49
- 18. Advantages of Unit Testing ✓ Finds problems early ▪ Test Driven Development ✓ Facilitates changes ▪ Can be run before each commit or build ▪ In combination with source control, can identify the revision (and originator) that broke the code 18 / 49
- 19. Limits of Unit Testing • Won’t catch every error in the program • Won’t catch integration errors • Combinatorial problem ▪ Every boolean decision statement requires at least two tests • Can’t test non-deterministic or concurrency problems 19 / 49
- 20. Limits of Unit Testing • Setting up realistic and useful tests is a challenge • Test case failures need to be reviewed daily and addressed immediately • Embedded system software presents a unique challenge ▪ Software is being developed on a different platform than the one it will eventually run on 20 / 49
- 21. Test Driven Development 1. Write an (initially failing) automated test case that defines a desired improvement or new function. 2. Produce the minimum amount of code required to pass that test. 3. Refactor the new code to acceptable standards. 21 / 49
- 22. Advantages of TDD ✓ Client-first development ✓ Taking small steps ✓ All written code is covered by at least one test ✓ Can lead to more modularized code 22 / 49
- 23. Limits of TDD • Support of the entire team is essential • Test are typically created by the developer who is writing the code being tested, and may therefore share the same blind spots with the code • Maintenance overhead 23 / 49
- 24. Unity Test Tools • Released by Unity in December 2013 • Partly integrated since Unity 5.3 • Completely open-source • Available on the Asset Store and Bitbucket • Based on NUnit 24 / 49
- 25. Unity Test Tools • Unit tests are discovered using reflections • Can be run automatically on recompile 26 / 49
- 26. Unity Test Tools 27 / 49 Unity NUnit Test Runner Window
- 27. Unity Test Tools • Integration tests allow testing integration of components, game objects and assets • Each test suite is a separate scene containing a game object with a TestRunner attached • Each test is a separate game object with a TestComponent attached ▪ Everything beneath that object in the hierarchy is considered to belong to that test • The CallTesting behaviour can modify the test result without having to actually write additional code 28 / 49
- 28. Unity Test Tools When you run the tests, the following steps are performed, in order: 1. Play mode is enabled. 2. The first or next test object is activated. 3. Wait until the test has finished (or a timeout has occurred). 4. The current active test gets deactivated. 5. If there are more tests in the queue, go to step 2. 6. Report results and finish test run. 29 / 49
- 29. Unity Test Tools 30 / 49Unity Integration Test Scene
- 30. Unity Test Tools • Assertions check invariants – conditions you expect to be always true • You can specify when to check these conditions • If any assertion fails, an exception is thrown • You can automatically cause the game in that case by enabling Error pause in the Unity console window 31 / 49
- 31. Unity Test Tools 32 / 49 Unity Assertion Component
- 32. Unity Test Tools Just like with NUnit, Unity integration tests can be run from command line: "C:Program Files (x86)UnityEditorUnity.exe“ –batchmode -projectPath D:TempUnityTest -executeMethod UnityTest.Batch.RunIntegrationTests -testscenes=NpruehsScene -targetPlatform=StandaloneWindows -resultsFileDirectory=D:TempResults 33 / 49
- 33. NSubstitute • Creates substitutes for interfaces • Saves you from having to use stubs, mocks, spies, test doubles 34 / 49 public interface ICalculator { int Add(int a, int b); }
- 34. NSubstitute • Creates substitutes for interfaces • Saves you from having to use stubs, mocks, spies, test doubles 35 / 49 ICalculator calculator = Substitute.For<ICalculator>();
- 35. NSubstitute • Allows you set up return values to method calls. • Great for mocking database connections or email plugins. 36 / 49 calculator.Add(1, 2).Returns(3);
- 36. 37 / 49
- 37. Unit Testing in C++ • googletest is a platform-independent framework for writing C++ tests • Used for a variety of Google projects, including Chromium and Google Protocol Buffers • googlemock allows you to write and mock C++ mock classes 38 / 49
- 38. Unit Testing in C++ 1. Create a new Win32 console application. 2. Add the projects you want to test to the additional include directories. 3. Add the gtest root folder and gtest-a.b.c/include folder to the additional include directories. 4. In your test source file, #include "src/gtest-all.cc“. 5. Provide a main entry point like this: 39 / 49 GTEST_API_ int main(int argc, char **argv) { testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); }
- 39. Unit Testing in C++ Test method signatures are automatically generated by the TEST macro: 40 / 49 TEST(TestSuiteName, TestName) { // Arrange. // Act. // Assert. }
- 40. Unit Testing in C++ Assertions are made using macros as well. 41 / 49 TEST(TestArithmetic, TestAdd) { // Arrange. auto i = 3; auto j = 4; // Act. auto k = i + j; // Assert. EXPECT_EQ(7, k); }
- 41. Unit Testing in C++ Run the tests by just starting the console application… 42 / 49
- 42. Unit Testing in C++ … or by using the UI of gtest-gbar. 43 / 49
- 43. Unit Testing in C++ googlemock provides macros for mocking methods. 44 / 49 #pragma once #include "gmock/gmock.h" class ICalculator { public: virtual int Add(int x, int y) = 0; }; class MockCalculator : public ICalculator { public: MOCK_METHOD2(Add, int(int x, int y)); };
- 44. Unit Testing in C++ Just like NSubstitute, it allows you to specify results: 45 / 49 #include "src/gtest-all.cc" #include "src/gmock-all.cc" #include "MockCalculator.h" using ::testing::Return; TEST(TestArithmetic, TestCalculator) { // Arrange. MockCalculator calculator; EXPECT_CALL(calculator, Add(3, 4)) .WillRepeatedly(Return(7)); // Act. auto sum = calculator.Add(3, 4); // Assert. EXPECT_EQ(7, sum); }
- 45. Unit Testing in UE 4 Unreal Engine 4 defines unit tests with macros, too: 46 / 49 IMPLEMENT_SIMPLE_AUTOMATION_TEST(FSetResTest, "Windows.SetResolution", ATF_Game) bool FSetResTest::RunTest(const FString& Parameters) { FString MapName = TEXT("AutomationTest"); FEngineAutomationTestUtilities::LoadMap(MapName); int32 ResX = GSystemSettings.ResX; int32 ResY = GSystemSettings.ResY; FString RestoreResolutionString = FString::Printf(TEXT("setres %dx%d"), ResX, ResY); ADD_LATENT_AUTOMATION_COMMAND(FEngineWaitLatentCommand(2.0f)); ADD_LATENT_AUTOMATION_COMMAND(FExecStringLatentCommand(TEXT("setres 640x480"))); ADD_LATENT_AUTOMATION_COMMAND(FEngineWaitLatentCommand(2.0f)); ADD_LATENT_AUTOMATION_COMMAND(FExecStringLatentCommand(RestoreResolutionString)); return true; }
- 46. Unit Testing in UE 4 Tests can be run from the Unreal Frontend: 47 / 49
- 47. References • Wikipedia. Unit testing. http://en.wikipedia.org/wiki/Unit_testing, April 25, 2017. • Poole. NUnit Quick Start. http://www.nunit.org/index.php?p=quickStart&r=2.6.4, 2015. • Wikipedia. Test-driven development. http://en.wikipedia.org/wiki/Test-driven_development, April 21, 2017. • Petersen. Unity Test Tools Released. http://blogs.unity3d.com/2013/12/18/unity-test-tools- released/, December 18, 2013. • Unity Technologies. Unity Test Tools. https://bitbucket.org/Unity-Technologies/unitytesttools, March 9, 2017. • NSubstitute. NSubstitute. http://nsubstitute.github.io/help/getting-started/, May 2017. • Google. Google C++ Testing Framework. https://code.google.com/p/googletest/, May 19, 2017. • Google. Google Test UI. https://code.google.com/p/gtest-gbar/, May 2, 2017. • Epic Games. Unreal Engine Automation Technical Guide. https://docs.unrealengine.com/latest/INT/Programming/Automation/TechnicalGuide/, May 2017. • Epic Games. Unreal Engine Automation User Guide. https://docs.unrealengine.com/latest/INT/Programming/Automation/UserGuide/, May 2017. 48 / 49
- 49. 5 Minute Review Session • What are unit tests? • What are the main advantages of unit testing? • What are the most important limits of unit tests? • Explain the process of Test Driven Development! • What are the upsides and downsides of TDD? • What are integration tests? • How do you test functionality that depends on external communication (e.g. database, mails)?