Mutation Testing using Mutpy Module in Python

Prerequisite: Mutation Testing

Mutpy is a Mutation testing tool in Python that generated mutants and computes a mutation score. It supports standard unittest module, generates YAML/HTML reports and has colorful output. It applies mutation on AST level.

Installation:

This module does not come built-in with Python. To install it type the below command in the terminal.

pip install mutpy

Here, we will use the Mutpy library to execute test cases written for a simple program.

Now, we perform Mutation Testing for a program that checks if a number is prime or not. 

Code: This code is save as isPrime.py

# define a function 
def isPrime(num):

    if num > 1:
      
       # check for factors
       for i in range(2,num):
           if (num % i) == 0:
               return False

       else:
           return True
           
    # if input number is less than
    # or equal to 1, it is not prime
    else:
       return False;

Now, we need to write test cases for the above program using Pytest or Unittest library. Test cases should be written with an approach to kill all the mutants, i.e. test cases should be effective enough to give a good mutation score. The test cases are written using the Unittest library in the file shown below. 

Test cases are written using self.assertEqual() that is a test assertion generally used to determine if a test case has passed or failed. We have written three test functions below to check the three types of input: 

1. Input is prime
2. Input is non-prime
3. Input is invalid

Note: The function name and the test file name should always start with the word 'test'.

Code: This code is save as test_isPrime.py.

# import required libraries
from unittest import TestCase
from isPrime import isPrime

# define a class
class CalculatorTest(TestCase):
    
    # test case for checking non prime nums
    def test_nonprime(self):
        self.assertEqual(isPrime(12),False)
    
    # test case to check prime nums
    def test_prime(self):
        self.assertEqual(isPrime(19),True)

    # test case to check invalid input
    def test_invalid(self):
        self.assertEqual(isPrime(-1),False)

To execute these test cases, we need to create two separate files isPrime.py and test_isPrime.py in a single folder and run the following command in the command prompt: 

mut.py --target isPrime --unit-test test_isPrime -m --runner pytest

In the above command, we have to specify three things: 

• Target: the target file on which test cases will be run, which in our case is isPrime.py
• Unit-test: the file which has the unit tests that have to be executed, i.e. test_isPrime.py in our case.
• Runner: pytest or unittest

The output will be a set of mutants along with details like mutation score, the number of mutations killed, survived, etc. 

[*] Start mutation process:
   - targets: isPrime
   - tests: test_isPrime
[*] 3 tests passed:
   - test_isPrime [3.07469 s]
[*] Start mutants generation and execution:
   - [#   1] AOR isPrime:
--------------------------------------------------------------------------------
   3:
   4:     if num > 1:
   5:
   6:         for i in range(2, num):
-  7:             if num % i == 0:
+  7:             if num * i == 0:
   8:                 return False
   9:         else:
  10:
  11:             return True
--------------------------------------------------------------------------------
[1.45151 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_nonprime
   - [#   2] COI isPrime:
--------------------------------------------------------------------------------
   1:
   2: def isPrime(num):
   3:
-  4:     if num > 1:
+  4:     if not (num > 1):
   5:
   6:         for i in range(2, num):
   7:             if num % i == 0:
   8:                 return False
--------------------------------------------------------------------------------
[1.25723 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_invalid
   - [#   3] COI isPrime:
--------------------------------------------------------------------------------
   3:
   4:     if num > 1:
   5:
   6:         for i in range(2, num):
-  7:             if num % i == 0:
+  7:             if not (num % i == 0):
   8:                 return False
   9:         else:
  10:
  11:             return True
--------------------------------------------------------------------------------
[1.28817 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_prime
   - [#   4] CRP isPrime:
--------------------------------------------------------------------------------
   1:
   2: def isPrime(num):
   3:
-  4:     if num > 1:
+  4:     if num > 2:
   5:
   6:         for i in range(2, num):
   7:             if num % i == 0:
   8:                 return False
--------------------------------------------------------------------------------
[1.23510 s] survived
   - [#   5] CRP isPrime:
--------------------------------------------------------------------------------
   2: def isPrime(num):
   3:
   4:     if num > 1:
   5:
-  6:         for i in range(2, num):
+  6:         for i in range(3, num):
   7:             if num % i == 0:
   8:                 return False
   9:         else:
  10:
--------------------------------------------------------------------------------
[1.20360 s] survived
   - [#   6] CRP isPrime:
--------------------------------------------------------------------------------
   3:
   4:     if num > 1:
   5:
   6:         for i in range(2, num):
-  7:             if num % i == 0:
+  7:             if num % i == 1:
   8:                 return False
   9:         else:
  10:
  11:             return True
--------------------------------------------------------------------------------
[1.23499 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_prime
   - [#   7] ROR isPrime:
--------------------------------------------------------------------------------
   1:
   2: def isPrime(num):
   3:
-  4:     if num > 1:
+  4:     if num < 1:
   5:
   6:         for i in range(2, num):
   7:             if num % i == 0:
   8:                 return False
--------------------------------------------------------------------------------
[1.24164 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_invalid
   - [#   8] ROR isPrime:
--------------------------------------------------------------------------------
   1:
   2: def isPrime(num):
   3:
-  4:     if num > 1:
+  4:     if num >= 1:
   5:
   6:         for i in range(2, num):
   7:             if num % i == 0:
   8:                 return False
--------------------------------------------------------------------------------
[1.21934 s] survived
   - [#   9] ROR isPrime:
--------------------------------------------------------------------------------
   3:
   4:     if num > 1:
   5:
   6:         for i in range(2, num):
-  7:             if num % i == 0:
+  7:             if num % i != 0:
   8:                 return False
   9:         else:
  10:
  11:             return True
--------------------------------------------------------------------------------
[1.32597 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_prime
[*] Mutation score [14.91747 s]: 66.7%
   - all: 9
   - killed: 6 (66.7%)
   - survived: 3 (33.3%)
   - incompetent: 0 (0.0%)
   - timeout: 0 (

We can see from the output above, 6 mutants have been killed and only 3 mutant was able to survive. Also, a mutation score of 66.7% was achieved. We can further improve this mutation score by analyzing the mutants that survived the test cases and writing new or modifying test cases to kill the mutants that survived.