Chapter 12. Gathering Insights with Data Mining
Let us finish Part III of the book with a discussion of perhaps the most exciting and promising area for future exploration for the practitioners of end-to-end tracing. Distributed tracing data provides a treasure trove of information about our distributed systems. I have already shown that even inspecting a single trace can be an exceptionally insightful exercise that often helps engineering teams understand performance issues and identify the root causes. However, even with low probability of sampling, software systems operating at internet-scale can record millions or even billions of traces per day. Even if every engineer in the company looks at a few traces each day, it is going to add up to a tiny fraction of all the data collected by the end-to-end tracing backend. It is a shame to let the rest of this data go to waste, when we can build data mining tools to process all of it; create...
