SlideShare a Scribd company logo

Unit testing data with marbles - Jane Stewart Adams, Leif Walsh

PyData, profile picture
PyData

The document discusses the importance of unit testing data assumptions, emphasizing the need to encode assumptions in testable forms, test incoming data, and report when assumptions do not hold. It highlights the complexities of working with evolving data and the potential challenges in manual data verification. By using a systematic approach to testing, we can automate the verification of data quality and make informed decisions based on reliable data outputs.

Technology
1 of 128
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
UNIT TESTING DATAUNIT TESTING DATA WITH MARBLESWITH MARBLES JANE ADAMS & LEIF WALSHJANE ADAMS & LEIF WALSH
ONCE UPON A TIME...ONCE UPON A TIME...
ONCE UPON A TIME...ONCE UPON A TIME... CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS.
ONCE UPON A TIME...ONCE UPON A TIME... CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS. THEY WERETHEY WERE ALOTALOTOLDER THAN THEIR PARENTS.OLDER THAN THEIR PARENTS.
ONCE UPON A TIME...ONCE UPON A TIME... CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS. THEY WERETHEY WERE ALOTALOTOLDER THAN THEIR PARENTS.OLDER THAN THEIR PARENTS. THEY WERE A LOT OLDER THANTHEY WERE A LOT OLDER THAN EVERYONEEVERYONE..
EVERYONE THAT WORKS WITH DATA HAS STORIES LIKEEVERYONE THAT WORKS WITH DATA HAS STORIES LIKE THISTHIS
HI, I'M JANEHI, I'M JANE
WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS?
WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS? 1. Children are born after their parents
WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS? 1. Children are born after their parents 2. People can't live forever
WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA?
WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct
WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data
WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data Records are unique
WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data Records are unique Measurements are precise
WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data Records are unique Measurements are precise (this is a non-exhaustive list)
WHY DOES THIS MATTER?WHY DOES THIS MATTER?
WE DON'T JUST HAVE DATA TO HAVE IT.WE DON'T JUST HAVE DATA TO HAVE IT.
WE DON'T JUST HAVE DATA TO HAVE IT.WE DON'T JUST HAVE DATA TO HAVE IT. WE USE DATA TO MAKE DECISIONS.WE USE DATA TO MAKE DECISIONS.
WE SHOULD BE EXPLICIT ABOUT OUR ASSUMPTIONS.WE SHOULD BE EXPLICIT ABOUT OUR ASSUMPTIONS.
WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE?
WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing
WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent
WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent Manually checking data is inconsistent and error-prone
WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent Manually checking data is inconsistent and error-prone We're working with alotof data
WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent Manually checking data is inconsistent and error-prone We're working with alotof data We're working with a lot of differentkindsof data
I'M LEIFI'M LEIF
WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO?
WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form
WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form Test those assumptions on incoming data
WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form Test those assumptions on incoming data Report when our assumptions don't hold
WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form Test those assumptions on incoming data Report when our assumptions don't hold Report allof the assumptions that don't hold
"Whatifwewroteunittestsfordata likewewriteunittestsforcode?"
unittestunittest
HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM?
HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Encode our assumptions in testable form
HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Test those assumptions on incoming data Encode our assumptions in testable form
HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Report when our assumptions don't hold Encode our assumptions in testable form Test those assumptions on incoming data
HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Report allof the assumptions that don't hold Encode our assumptions in testable form Test those assumptions on incoming data Report when our assumptions don't hold
tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
How long was the trip? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
How far was the trip? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
Internal metadata tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
Who took the trip? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
??? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
Load the data class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
Pick some thresholds class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
For each threshold class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
Find trips longer than the threshold class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
Assert none exist class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
$ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US?
WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US? New data are automatically tested for long trips
WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US? New data are automatically tested for long trips Don't have to remember howto test for long trips
WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US? New data are automatically tested for long trips Don't have to remember howto test for long trips Can easily run this test over historical data
TEST WRITING INTERLUDE...TEST WRITING INTERLUDE...
TEST WRITING INTERLUDE...TEST WRITING INTERLUDE...
WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT!
WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data
WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down
WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down 3. We've made them executable
WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down 3. We've made them executable 4. We've automated them
WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down 3. We've made them executable 4. We've automated them ⭐⭐
MONTHS PASS...MONTHS PASS...
$ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
Her: "Is there a way to see local variables in my unittest output?" $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
Her: "Is there a way to see local variables in my unittest output?" Him: "I think pytest does that..." $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES
PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here?
PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure?
PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it?
PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it? Have we seen this failure before? When?
PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it? Have we seen this failure before? When? CONTEXT IS EXPENSIVE TO RECOVERCONTEXT IS EXPENSIVE TO RECOVER
THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA
THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA 1. Assumptions aren't black-and-white
THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA 1. Assumptions aren't black-and-white 2. Failures are usually introduced by someone else
THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA 1. Assumptions aren't black-and-white 2. Failures are usually introduced by someone else 3. Different tests require different follow-up
Unit testing data with marbles - Jane Stewart Adams, Leif Walsh
ANATOMY OF A MARBLES FAILURE MESSAGEANATOMY OF A MARBLES FAILURE MESSAGE
$ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
What is this test doing? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
What is this test doing? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
Why is it here? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
What am I supposed to do about this failure? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
How bad is it? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
Can we add more context? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS
SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper))
SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper)) self.assertGreater(x, lower) self.assertGreater(upper, x)
SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper)) self.assertGreater(x, lower) self.assertGreater(upper, x) self.assertTrue(all(a < b for a, b in zip([lower, x], [x, upper])))
SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper)) self.assertGreater(x, lower) self.assertGreater(upper, x) self.assertTrue(all(a < b for a, b in zip([lower, x], [x, upper]))) self.assertBetween(x, lower, upper)
marbles.mixinsmarbles.mixins
marbles.mixinsmarbles.mixins from marbles.mixins import mixins   class TripDistanceTestCase(BikeshareTestCase, mixins.BetweenMixins):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_unreasonable_distances(self): for distance in self.data['distance_meters']: self.assertBetween(distance, 100, 42195)
Unit testing data with marbles - Jane Stewart Adams, Leif Walsh
CUSTOM ASSERTIONSCUSTOM ASSERTIONS
CUSTOM ASSERTIONSCUSTOM ASSERTIONS self.assertTrue(long_trips.empty)
CUSTOM ASSERTIONSCUSTOM ASSERTIONS self.assertTrue(long_trips.empty) self.assertEqual(len(long_trips), 0)
CUSTOM ASSERTIONSCUSTOM ASSERTIONS self.assertTrue(long_trips.empty) self.assertEqual(len(long_trips), 0) class DataFrameMixins(object):   def assertDataFrameEmpty(self, df, msg=None): self.assertTrue(df.empty, msg=msg)
DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED?
DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? What is this test doing? Why is it here?
DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? What am I supposed to do about this failure? What is this test doing? Why is it here?
DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? How bad is it? What is this test doing? Why is it here? What am I supposed to do about this failure?
DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? Have we seen this failure before? When? What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it?
ASSERTION LOGGINGASSERTION LOGGING
ASSERTION LOGGINGASSERTION LOGGING import marbles.core from marbles.core import log   class TripDistanceTestCase(BikeshareTestCase): ...   if __name__ == '__main__': log.logger.configure(logfile='marbles.log') marbles.core.main()
{ "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
Which test was running? { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
What did we assert? { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
Local variables { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
Which data were we testing? { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
Other information about the assertion { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
More (not pictured) { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
HISTORICAL FAILURESHISTORICAL FAILURES
HISTORICAL FAILURESHISTORICAL FAILURES "Have we seen this kind of problem before?"
HISTORICAL FAILURESHISTORICAL FAILURES "Have we seen this kind of problem before?"
AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS
AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS df = df.pivot_table( index=['month'], columns=['severity'], values='anomalies', aggfunc=sum) df.describe()
AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS df = df.pivot_table( index=['month'], columns=['severity'], values='anomalies', aggfunc=sum) df.describe()
CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO
CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO $ python -m unittest F ====================================================================== FAIL: test_return_code (docs.examples.getting_started.ResponseTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/git/marbles/docs/examples/getting_started.py", line 43, in test_return_code 201 AssertionError: 409 != 201   ---------------------------------------------------------------------- Ran 1 test in 0.000s
CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO $ python -m marbles F ====================================================================== FAIL: test_return_code (docs.examples.getting_started.ResponseTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: 409 != 201   Source (/home/leif/git/marbles/docs/examples/getting_started.py): 40 res = requests.put(endpoint, data=data) > 41 self.assertEqual( 42 res.status_code, 43 201 44 ) Locals: endpoint = 'http://example.com/api/v1/resource' data = {'id': 1, 'name': 'Little Bobby Tables'} res = <docs.examples.getting_started.Response object at 0x7fae97e78978>     ---------------------------------------------------------------------- Ran 1 test in 0.001s
TWO STEPS TO MARBLESTWO STEPS TO MARBLES $ pip install marbles $ python -m marbles test_module.py
GITHUBGITHUB github.com/twosigma/marbles
DOCUMENTATIONDOCUMENTATION marbles.readthedocs.io
DOCUMENTATIONDOCUMENTATION marbles.readthedocs.io
CONTRIBUTING AND GETTING HELPCONTRIBUTING AND GETTING HELP github.com/twosigma/marbles/issues
✨ READ BETTER TEST FAILURES ✨✨ READ BETTER TEST FAILURES ✨ & github.com/twosigma/marbles marbles.readthedocs.io @thejunglejane @leifwalsh

More Related Content

PDF
Coding Culture
Sven Peters
 
PDF
UoB Lecture v1 2015_02_10jj
Joe Jones
 
KEY
Data for Business Journalism, NICAR 2012
Chris Taggart
 
PDF
Culture, But Not as You Know It - Sven Peters
Atlassian
 
PPT
The Age of Context: How it Will Change your Life & Work | Robert Scoble, Rack...
Dealmaker Media
 
PDF
Using Big Data to Reveal Consumer Values and Inform Storytelling
Ravi Iyer
 
PPTX
Graph based facet selection
Nicholai (Haogang) Yu
 
PDF
Ric Rodriguez — SEO Mythbusting
Semrush
 
Coding Culture
Sven Peters
 
UoB Lecture v1 2015_02_10jj
Joe Jones
 
Data for Business Journalism, NICAR 2012
Chris Taggart
 
Culture, But Not as You Know It - Sven Peters
Atlassian
 
The Age of Context: How it Will Change your Life & Work | Robert Scoble, Rack...
Dealmaker Media
 
Using Big Data to Reveal Consumer Values and Inform Storytelling
Ravi Iyer
 
Graph based facet selection
Nicholai (Haogang) Yu
 
Ric Rodriguez — SEO Mythbusting
Semrush
 

Similar to Unit testing data with marbles - Jane Stewart Adams, Leif Walsh (6)

PDF
Music 4.5: Robert Kaye, Founder, Metabrainz
MME 4.5 / Music 4.5 / 2Pears
 
PDF
Coding Culture - Sven Peters - Codemotion Milan 2016
Codemotion
 
PDF
Jeff jonas big data new physics
MIT Forum of Israel
 
PDF
Josiah Fisk - The message and the meta-message: How to assess and control the...
Plain Talk 2015
 
PDF
Blockchain for the people: Implementing blockchain technology for a secure an...
Marko Kovic
 
PDF
Success Factors in Web & Mobile Product Development: Why We Say 'No' and Why ...
Dolphin Micro, Inc.
 
Music 4.5: Robert Kaye, Founder, Metabrainz
MME 4.5 / Music 4.5 / 2Pears
 
Coding Culture - Sven Peters - Codemotion Milan 2016
Codemotion
 
Jeff jonas big data new physics
MIT Forum of Israel
 
Josiah Fisk - The message and the meta-message: How to assess and control the...
Plain Talk 2015
 
Blockchain for the people: Implementing blockchain technology for a secure an...
Marko Kovic
 
Success Factors in Web & Mobile Product Development: Why We Say 'No' and Why ...
Dolphin Micro, Inc.
 
Ad

More from PyData (20)

PDF
Michal Mucha: Build and Deploy an End-to-end Streaming NLP Insight System | P...
PyData
 
PDF
The TileDB Array Data Storage Manager - Stavros Papadopoulos, Jake Bolewski
PyData
 
PDF
Using Embeddings to Understand the Variance and Evolution of Data Science... ...
PyData
 
PDF
Deploying Data Science for Distribution of The New York Times - Anne Bauer
PyData
 
PPTX
Graph Analytics - From the Whiteboard to Your Toolbox - Sam Lerma
PyData
 
PPTX
Do Your Homework! Writing tests for Data Science and Stochastic Code - David ...
PyData
 
PDF
RESTful Machine Learning with Flask and TensorFlow Serving - Carlo Mazzaferro
PyData
 
PDF
Mining dockless bikeshare and dockless scootershare trip data - Stefanie Brod...
PyData
 
PDF
Avoiding Bad Database Surprises: Simulation and Scalability - Steven Lott
PyData
 
PDF
Words in Space - Rebecca Bilbro
PyData
 
PDF
End-to-End Machine learning pipelines for Python driven organizations - Nick ...
PyData
 
PPTX
Pydata beautiful soup - Monica Puerto
PyData
 
PDF
1D Convolutional Neural Networks for Time Series Modeling - Nathan Janos, Jef...
PyData
 
PPTX
Extending Pandas with Custom Types - Will Ayd
PyData
 
PDF
Measuring Model Fairness - Stephen Hoover
PyData
 
PDF
What's the Science in Data Science? - Skipper Seabold
PyData
 
PDF
Applying Statistical Modeling and Machine Learning to Perform Time-Series For...
PyData
 
PDF
Solving very simple substitution ciphers algorithmically - Stephen Enright-Ward
PyData
 
PDF
The Face of Nanomaterials: Insightful Classification Using Deep Learning - An...
PyData
 
PDF
Deprecating the state machine: building conversational AI with the Rasa stack...
PyData
 
Michal Mucha: Build and Deploy an End-to-end Streaming NLP Insight System | P...
PyData
 
The TileDB Array Data Storage Manager - Stavros Papadopoulos, Jake Bolewski
PyData
 
Using Embeddings to Understand the Variance and Evolution of Data Science... ...
PyData
 
Deploying Data Science for Distribution of The New York Times - Anne Bauer
PyData
 
Graph Analytics - From the Whiteboard to Your Toolbox - Sam Lerma
PyData
 
Do Your Homework! Writing tests for Data Science and Stochastic Code - David ...
PyData
 
RESTful Machine Learning with Flask and TensorFlow Serving - Carlo Mazzaferro
PyData
 
Mining dockless bikeshare and dockless scootershare trip data - Stefanie Brod...
PyData
 
Avoiding Bad Database Surprises: Simulation and Scalability - Steven Lott
PyData
 
Words in Space - Rebecca Bilbro
PyData
 
End-to-End Machine learning pipelines for Python driven organizations - Nick ...
PyData
 
Pydata beautiful soup - Monica Puerto
PyData
 
1D Convolutional Neural Networks for Time Series Modeling - Nathan Janos, Jef...
PyData
 
Extending Pandas with Custom Types - Will Ayd
PyData
 
Measuring Model Fairness - Stephen Hoover
PyData
 
What's the Science in Data Science? - Skipper Seabold
PyData
 
Applying Statistical Modeling and Machine Learning to Perform Time-Series For...
PyData
 
Solving very simple substitution ciphers algorithmically - Stephen Enright-Ward
PyData
 
The Face of Nanomaterials: Insightful Classification Using Deep Learning - An...
PyData
 
Deprecating the state machine: building conversational AI with the Rasa stack...
PyData
 
Ad

Recently uploaded (20)

PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
August Patch Tuesday
Ivanti
 
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PPTX
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
PDF
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PPTX
Machine Learning_overview_presentation.pptx
kondavamsidharreddy2
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PPTX
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
August Patch Tuesday
Ivanti
 
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
1. Introduction to Computer Programming.pptx
bonakiduza1024
 
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
Approach and Philosophy of On baking technology
30anthuong17
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Machine Learning_overview_presentation.pptx
kondavamsidharreddy2
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 

Unit testing data with marbles - Jane Stewart Adams, Leif Walsh

  • 1. UNIT TESTING DATAUNIT TESTING DATA WITH MARBLESWITH MARBLES JANE ADAMS & LEIF WALSHJANE ADAMS & LEIF WALSH
  • 2. ONCE UPON A TIME...ONCE UPON A TIME...
  • 3. ONCE UPON A TIME...ONCE UPON A TIME... CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS.
  • 4. ONCE UPON A TIME...ONCE UPON A TIME... CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS. THEY WERETHEY WERE ALOTALOTOLDER THAN THEIR PARENTS.OLDER THAN THEIR PARENTS.
  • 5. ONCE UPON A TIME...ONCE UPON A TIME... CHILDREN WERE OLDER THAN THEIR PARENTS.CHILDREN WERE OLDER THAN THEIR PARENTS. THEY WERETHEY WERE ALOTALOTOLDER THAN THEIR PARENTS.OLDER THAN THEIR PARENTS. THEY WERE A LOT OLDER THANTHEY WERE A LOT OLDER THAN EVERYONEEVERYONE..
  • 6. EVERYONE THAT WORKS WITH DATA HAS STORIES LIKEEVERYONE THAT WORKS WITH DATA HAS STORIES LIKE THISTHIS
  • 7. HI, I'M JANEHI, I'M JANE
  • 8. WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS?
  • 9. WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS? 1. Children are born after their parents
  • 10. WHAT WERE MY ASSUMPTIONS?WHAT WERE MY ASSUMPTIONS? 1. Children are born after their parents 2. People can't live forever
  • 11. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA?
  • 12. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct
  • 13. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data
  • 14. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data Records are unique
  • 15. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data Records are unique Measurements are precise
  • 16. WHAT ELSE DO WE ASSUME ABOUT DATA?WHAT ELSE DO WE ASSUME ABOUT DATA? Values are correct We're not missing any data Records are unique Measurements are precise (this is a non-exhaustive list)
  • 17. WHY DOES THIS MATTER?WHY DOES THIS MATTER?
  • 18. WE DON'T JUST HAVE DATA TO HAVE IT.WE DON'T JUST HAVE DATA TO HAVE IT.
  • 19. WE DON'T JUST HAVE DATA TO HAVE IT.WE DON'T JUST HAVE DATA TO HAVE IT. WE USE DATA TO MAKE DECISIONS.WE USE DATA TO MAKE DECISIONS.
  • 20. WE SHOULD BE EXPLICIT ABOUT OUR ASSUMPTIONS.WE SHOULD BE EXPLICIT ABOUT OUR ASSUMPTIONS.
  • 21. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE?
  • 22. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing
  • 23. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent
  • 24. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent Manually checking data is inconsistent and error-prone
  • 25. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent Manually checking data is inconsistent and error-prone We're working with alotof data
  • 26. WHAT ARE THE IMPORTANT PROBLEMS HERE?WHAT ARE THE IMPORTANT PROBLEMS HERE? Data are always changing Some changes are loud while others are silent Manually checking data is inconsistent and error-prone We're working with alotof data We're working with a lot of differentkindsof data
  • 28. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO?
  • 29. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form
  • 30. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form Test those assumptions on incoming data
  • 31. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form Test those assumptions on incoming data Report when our assumptions don't hold
  • 32. WHAT DO WE WANT TO DO?WHAT DO WE WANT TO DO? Encode our assumptions in testable form Test those assumptions on incoming data Report when our assumptions don't hold Report allof the assumptions that don't hold
  • 35. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM?
  • 36. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Encode our assumptions in testable form
  • 37. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Test those assumptions on incoming data Encode our assumptions in testable form
  • 38. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Report when our assumptions don't hold Encode our assumptions in testable form Test those assumptions on incoming data
  • 39. HOW DOES UNITTEST SOLVE OUR PROBLEM?HOW DOES UNITTEST SOLVE OUR PROBLEM? Report allof the assumptions that don't hold Encode our assumptions in testable form Test those assumptions on incoming data Report when our assumptions don't hold
  • 40. tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  • 41. How long was the trip? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  • 42. How far was the trip? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  • 43. Internal metadata tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  • 44. Who took the trip? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  • 45. ??? tripduration 0 days 00:18:09 starttime 2018-08-01 00:00:09.341000-04:00 stoptime 2018-08-01 00:18:18.889000-04:00 start station id 31 start station name Seaport Hotel - Congress St at Seaport Ln start station latitude 42.3488 start station longitude -71.0417 end station id 190 end station name Nashua Street at Red Auerbach Way end station latitude 42.3657 end station longitude -71.0643 bikeid 1026 usertype Subscriber birth year 1969 gender 0 Hubway Bike Share Dataset
  • 46. class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  • 47. Load the data class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  • 48. Pick some thresholds class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  • 49. For each threshold class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  • 50. Find trips longer than the threshold class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  • 51. Assert none exist class TripDistanceTestCase(unittest.TestCase):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_long_trips(self): thresholds = [ ('marathon', 42195), ('10km', 10000) ]   for severity, threshold in thresholds: with self.subTest(severity=severity): long_trips = self.data[ self.data['distance_meters'] > threshold] self.assertTrue(long_trips.empty)
  • 52. $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
  • 53. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US?
  • 54. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US? New data are automatically tested for long trips
  • 55. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US? New data are automatically tested for long trips Don't have to remember howto test for long trips
  • 56. WHAT DOES THIS GIVE US?WHAT DOES THIS GIVE US? New data are automatically tested for long trips Don't have to remember howto test for long trips Can easily run this test over historical data
  • 57. TEST WRITING INTERLUDE...TEST WRITING INTERLUDE...
  • 58. TEST WRITING INTERLUDE...TEST WRITING INTERLUDE...
  • 59. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT!
  • 60. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data
  • 61. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down
  • 62. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down 3. We've made them executable
  • 63. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down 3. We've made them executable 4. We've automated them
  • 64. WE'RE IN A PRETTY GOOD SPOT!WE'RE IN A PRETTY GOOD SPOT! 1. We've thought through our assumptions about the data 2. We've made them explicit by writing them down 3. We've made them executable 4. We've automated them ⭐⭐
  • 66. $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
  • 67. Her: "Is there a way to see local variables in my unittest output?" $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
  • 68. Her: "Is there a way to see local variables in my unittest output?" Him: "I think pytest does that..." $ python -m unittest test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/test_bikeshare.py", line 107, in test_for_long_trips self.assertTrue(long_trips.empty) AssertionError: False is not true   ---------------------------------------------------------------------- Ran 1 test in 0.000s   FAILED (failures=1)
  • 69. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES
  • 70. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here?
  • 71. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure?
  • 72. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it?
  • 73. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it? Have we seen this failure before? When?
  • 74. PUT YOURSELF IN THE TEST CONSUMER'S SHOESPUT YOURSELF IN THE TEST CONSUMER'S SHOES What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it? Have we seen this failure before? When? CONTEXT IS EXPENSIVE TO RECOVERCONTEXT IS EXPENSIVE TO RECOVER
  • 75. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA
  • 76. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA 1. Assumptions aren't black-and-white
  • 77. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA 1. Assumptions aren't black-and-white 2. Failures are usually introduced by someone else
  • 78. THIS ISN'T UNIQUE TO DATA,THIS ISN'T UNIQUE TO DATA, BUT IT'S ESPECIALLY HARD WITH DATABUT IT'S ESPECIALLY HARD WITH DATA 1. Assumptions aren't black-and-white 2. Failures are usually introduced by someone else 3. Different tests require different follow-up
  • 80. ANATOMY OF A MARBLES FAILURE MESSAGEANATOMY OF A MARBLES FAILURE MESSAGE
  • 81. $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  • 82. What is this test doing? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  • 83. What is this test doing? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  • 84. Why is it here? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  • 85. What am I supposed to do about this failure? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  • 86. How bad is it? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  • 87. Can we add more context? $ python -m marbles test_bikeshare.py ====================================================================== FAIL: test_for_long_trips (test_bikeshare.TripDistanceTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: False is not true   Source (/home/leif/test_bikeshare.py): 151 self.data['distance_meters'] > threshold] > 152 self.assertTrue(long_trips.empty, note=note) 153 Locals: severity = 'marathon' threshold = 42195 long_trips = start station latitude stop station latitude ... 27955 42.366277 0.0 ... Note: There appear to be some trips in the data that are longer than a marathon! If these are legitimate trips, consider contacting the local news station about a human-interest story. If these do not appear to be legitimate trips, contact the bike share mechanics to have affected bikes identified and repaired.
  • 90. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper)) self.assertGreater(x, lower) self.assertGreater(upper, x)
  • 91. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper)) self.assertGreater(x, lower) self.assertGreater(upper, x) self.assertTrue(all(a < b for a, b in zip([lower, x], [x, upper])))
  • 92. SEMANTIC ASSERTIONSSEMANTIC ASSERTIONS self.assertTrue((lower < x) and (x < upper)) self.assertGreater(x, lower) self.assertGreater(upper, x) self.assertTrue(all(a < b for a, b in zip([lower, x], [x, upper]))) self.assertBetween(x, lower, upper)
  • 94. marbles.mixinsmarbles.mixins from marbles.mixins import mixins   class TripDistanceTestCase(BikeshareTestCase, mixins.BetweenMixins):   def setUp(self): self.data = ...   def tearDown(self): delattr(self, 'data')   def test_for_unreasonable_distances(self): for distance in self.data['distance_meters']: self.assertBetween(distance, 100, 42195)
  • 99. CUSTOM ASSERTIONSCUSTOM ASSERTIONS self.assertTrue(long_trips.empty) self.assertEqual(len(long_trips), 0) class DataFrameMixins(object):   def assertDataFrameEmpty(self, df, msg=None): self.assertTrue(df.empty, msg=msg)
  • 100. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED?
  • 101. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? What is this test doing? Why is it here?
  • 102. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? What am I supposed to do about this failure? What is this test doing? Why is it here?
  • 103. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? How bad is it? What is this test doing? Why is it here? What am I supposed to do about this failure?
  • 104. DOES MARBLES RECOVER THE CONTEXT WE WANTED?DOES MARBLES RECOVER THE CONTEXT WE WANTED? Have we seen this failure before? When? What is this test doing? Why is it here? What am I supposed to do about this failure? How bad is it?
  • 106. ASSERTION LOGGINGASSERTION LOGGING import marbles.core from marbles.core import log   class TripDistanceTestCase(BikeshareTestCase): ...   if __name__ == '__main__': log.logger.configure(logfile='marbles.log') marbles.core.main()
  • 107. { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  • 108. Which test was running? { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  • 109. What did we assert? { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  • 110. Local variables { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  • 111. Which data were we testing? { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  • 112. Other information about the assertion { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  • 113. More (not pictured) { "case": "test_for_long_trips (test_bikeshare.TripDistanceTestCase)", "test_case": "TripDistanceTestCase", "test_method": "test_for_long_trips", "assertion": "assertTrue", ... "locals": [ { "key": "severity", "value": "marathon" }, { "key": "threshold", "value": "42195" }, { "key": "long_trips", "value": " start station latitude stop station latitude ... 27955 42.366277 0.0 ... " } ], "month": "2016-07-01", "severity": "marathon", "anomalies": "1", "result": "fail" }
  • 115. HISTORICAL FAILURESHISTORICAL FAILURES "Have we seen this kind of problem before?"
  • 116. HISTORICAL FAILURESHISTORICAL FAILURES "Have we seen this kind of problem before?"
  • 117. AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS
  • 118. AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS df = df.pivot_table( index=['month'], columns=['severity'], values='anomalies', aggfunc=sum) df.describe()
  • 119. AGGREGATE DATASET HEALTH METRICSAGGREGATE DATASET HEALTH METRICS df = df.pivot_table( index=['month'], columns=['severity'], values='anomalies', aggfunc=sum) df.describe()
  • 120. CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO
  • 121. CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO $ python -m unittest F ====================================================================== FAIL: test_return_code (docs.examples.getting_started.ResponseTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/leif/git/marbles/docs/examples/getting_started.py", line 43, in test_return_code 201 AssertionError: 409 != 201   ---------------------------------------------------------------------- Ran 1 test in 0.000s
  • 122. CONTEXT IS GOOD FOR SOFTWARE TESTS, TOOCONTEXT IS GOOD FOR SOFTWARE TESTS, TOO $ python -m marbles F ====================================================================== FAIL: test_return_code (docs.examples.getting_started.ResponseTestCase) ---------------------------------------------------------------------- marbles.core.marbles.ContextualAssertionError: 409 != 201   Source (/home/leif/git/marbles/docs/examples/getting_started.py): 40 res = requests.put(endpoint, data=data) > 41 self.assertEqual( 42 res.status_code, 43 201 44 ) Locals: endpoint = 'http://example.com/api/v1/resource' data = {'id': 1, 'name': 'Little Bobby Tables'} res = <docs.examples.getting_started.Response object at 0x7fae97e78978>     ---------------------------------------------------------------------- Ran 1 test in 0.001s
  • 123. TWO STEPS TO MARBLESTWO STEPS TO MARBLES $ pip install marbles $ python -m marbles test_module.py
  • 127. CONTRIBUTING AND GETTING HELPCONTRIBUTING AND GETTING HELP github.com/twosigma/marbles/issues
  • 128. ✨ READ BETTER TEST FAILURES ✨✨ READ BETTER TEST FAILURES ✨ & github.com/twosigma/marbles marbles.readthedocs.io @thejunglejane @leifwalsh