Actor Concurrency Bugs: A Comprehensive Study on Symptoms, Root Causes, API Usages, and Differences

Editor's Notes

• #2: Hello everybody. My name is Raffi Khatchadourian, and I am an assistant professor at CUNY Hunter College. Today, I will present joint work with my colleagues Mehdi Bagherzadeh, Nicholas Fireman, and Anas Shawesh from Oakland University on a study of the characteristics of real-world actor bugs. Unfortunately, my colleagues were not able to attend today’s event, but I glad to have the opportunity to present the work in their place.
• #3: Nowadays, concurrency is everywhere: from hospitals, to stock exchanges, to power grids. And there are two major programming models behind this concurrency: multithreaded concurrency and message passing concurrency.
• #4: As a quick recap on actors, the Actor Model is a mathematical model of concurrent computation where actors are the universal primities of for concurrent computation. Actors respond to messages and can subsequently make local decisions, create more actors to fulfill a response, as well as coordinate with existing (child) actors. Actors have local private states and can affect the state of other actors only through message passing, foregoing the need of locking mechanisms for synchronization. Actors’ lives are also managed through life cycles, which are typically managed hierarchically.
• #5: Despite the dominance of multithreaded concurrency, message passing and its realization in actors is becoming increasingly popular. For example, PayPal uses actors to serve more than a billion financial transactions per day.
• #6: Unlike multithreaded concurrency, in which threads communicate using shared memory and locks, actors communicate using asynchronous message. This makes actors less prone to some of the standard multithreaded bugs such as low-level data races and deadlocks. However, actors come with their own bugs that are not only different but also largely unknown.
• #7: Knowing the common characteristics of bugs that involve actor-based concurrency can help focus current and future tools, inform new and current programming language and API designs, assist practitioners with the best practices of using actor concurrency, as well as guide educators in effectively teaching concurrency mechanisms and models to students.
• #8: To understand the real-world actor bugs and their characteristics, we formulate 4 research questions: 1. What are the symptoms of actor bugs? 2. What are the root cause of these bugs? 3. What APIs do these bugs use? 4. How different these bugs are when they come from different sources?
• #9: To collect the set of our actor bugs, we mined popular code open-source code repositories, namely, Stack Overflow Q&A posts and GitHub repositories.
• #10: We construct a set of real-world bugs from both Stack Overflow and GitHub. Stack Overflow is the most common question and answer website with more than 18 million questions and 4 million developers. Stack Overflow gives us insight about the bugs for which developers ask questions about and receive help in fixing. Similarly, GitHub is the most common code repository for open-source software with more than 100 million projects and 24 million developers. GitHub gives us insights about the bugs that developers leave in their code but find and fix later.
• #11: There are several industrial-strength actor frameworks and languages. Among these, we focus on Akka because it is growing faster than others. For example, the number of Akka questions in SO is 48 times the number of questions related to Microsoft’s Orleans.
• #12: To collect Akka bugs in Stack Overflow, we identify Akka questions that include code and manually analyze these questions to see if they discuss bugs and their fixes. In total, we analyzed near 4 thousand questions, answers, and comments, finding 130 bugs and their fixes. Multiple authors participated in this analysis and iterated and refined until they all agreed on the bugs.
• #13: To collect Akka bugs in GitHub, we identify mature Akka projects and their commits and manually analyze these commits to see if they discuss bugs and their fixes. In total, we analyzed more than 3 thousand commits, messages, original and modified code snippets, issues, and pull requests, finding 56 bugs and their fixes. Again, multiple authors participated in this analysis and iterated and refined until they all agree.
• #14: Our final dataset includes 186 actor bugs. The size of our dataset is in line with the previous work. For example, Hedden and Zhao study 126 actor bugs. Because of their complementary characteristics, we include both run-time and compile-time bugs in our dataset. 89% of our bugs are run-time bugs whereas 11% are compile-time.
• #15: To understand the characteristics of our bugs, we follow these next steps.
• #16: For our Stack Overflow bugs, we manually analyze text and code snippets in their questions, all their answers, and comments to understand the symptoms, root causes, API usages, and differences of these bugs. Similarly, for our GitHub bugs, we manually analyses the messages and code snippets in their commits, issues, and pull requests. Similar to data collection, multiple authors participate in these analyses and iterate and refine until they all agree.
• #17: Now, we can answer our research questions. Research question 1. What are the symptoms of actor bugs?
• #18: We find that the symptoms of actor bugs can be classified into 5 categories: Error, Unexpected Behavior, Incorrect Messaging, Incorrect Termination, and Incorrect Exceptions. Among these, Error is the most common, with more than a third of the symptoms, and Incorrect Exceptions is the least common, with only 2%. A symptom denotes the impacts of a bug when it occurs. For example, the impact of an Incorrect Exceptions bug is that an intended actor does not throw, catch, or handle an expected exception.
• #19: To better understand our symptoms, we present examples of bugs with these symptoms. To illustrate, this code, which is written in Scala Akka, shows a buggy actor program with an Incorrect Exceptions symptom. The code declares the application app and the actor MyActor with a constructor this. In app, the developer intends to create the actor act, of type MyActor, as a child of app and catch the exception that its creation throws. The invocation of actorOf on context creates act as a child of app. However, this program is buggy with the impact that app cannot catch the exception that act throws. This is because, in Akka, an exception that is thrown by an actor can be caught and handled only by its parent which is an actor itself. Here, app is an application and not an actor and thus cannot be the parent of act and cannot catch its exception.
• #20: We show the relation of our symptoms with the findings of the previous work. An example relation is that 3 of our symptoms overlap with all of Bianchi et al.’s symptoms. Bianchi et al. survey the existing actor testing tools and classify 3 symptoms that these tools use to identify buggy executions. In contrast, 2 of our symptoms, Incorrect Messaging and Incorrect Exceptions, are new and cannot be found in the previous work. These 2 new symptoms account for more than one fourth of our bugs.
• #21: Finally, we discuss the implications of our symptoms. An example implication is that current actor bug finding tools use Crash, Deadlock, and Assertion Violation as symptoms to identify buggy executions and find bugs. Our symptoms overlap with the symptoms that these tools use. However, we also find symptoms that are new and cannot be found in the previous work. Current and future tools can use our new symptoms to extend the set of buggy executions that they consider and find more bugs. *** END However, we also find new symptoms that these works do not use. Current and future bug finding tools can use our new Incorrect Messaging, and Incorrect Exceptions symptoms to extend the set of buggy executions that they consider and the bugs they find. *** END
• #22: Research question 2. What are the root causes of actor bugs?
• #23: We find that the root causes of our actor bugs can be classified into 10 categories ranging from Logic to Messaging Patterns to Untyped Communications. Among these, Logic is the most common root cause, with about one fifth of the root causes, and Untyped Communication is the least common with only 1%. A root causes specifies the main reason for the occurrence of a bug. For example, the reason for an Explicit Life Cycle management bug is that Akka requires the life cycles of actors to be managed explicitly by developers. This means any mismanagements of actor life cycles can cause bugs.
• #24: To illustrate, this code shows a buggy actor program with an Explicit Life Cycle Management root cause. The code declares the actor Parent and the actor Child. In Parent, the developer intends to send the message “doCleanup” to the child c, wait for its cleanup and then stop and terminate c and itself. In Akka, the operator ? sends a message with a future result that can be waited on. postStop is invoked by Akka when an actor terminates. However, this program is buggy with the impact that c does not perform any cleanup. The root cause of this bug is that during the lifecycle of actors in Akka a parent terminates only after all its children terminate and not before. Therefore, when postStop() of Parent is invoked, c is already stopped and cannot receive and process the cleanup message.
• #25: An example relation for our root causes is that 6 of our root causes overlap with Hedden and Zhao’s root causes. Hedden and Zhao study 126 Akka actor bugs and classify their root causes. In contrast, 4 of our root causes, which are API Confusion, Model Confusion, Misnaming, and Misconfiguration are new and cannot be found in the previous work. These 4 new root causes account for about one third of our bugs. *** END To further understand our root causes, we show their relations with the root causes of actor bugs in previous work. In contrast, 2 of our symptoms, that is Incorrect Messaging and Incorrect Exceptions are new and cannot be found in previous work. These new symptoms account for about 28% of our bugs, One example relation is that 6 of our root causes overlap with Hedden and Zhao’s actor bug root causes. Hedden and Zhao study 126 Akka actor bugs in 12 projects and classify their root causes. In contrast, 4 of our root causes are new and cannot be found in the previous work. These root causes are the root causes for 31.9% *** END
• #26: An example implication for our root causes is that current actor bug testing tools use patterns of root causes to find bugs that match these patterns. Our root causes overlap with the root causes that these tools use. However, we also find root causes that are new and cannot be found in the previous work. Current and future tools can use our new root causes to extend the set of patterns that they consider and find more bugs. *** END In addition to examples and relations, we also discuss the implications of our root causes. According to Bianchi et al.’s survey existing actor bug finding tools use Crash, Deadlock, and Assertion Violation symptoms—as testing oracles—to identify buggy executions and bugs. These However, our new Incorrect Messaging, and Incorrect Exceptions symptoms, that are symptoms for 27.5% of our bugs, can be used to extend the set of Buggy executions that these tools consider and thus the bugs they may find.
• #27: Research question 3. What APIs are involved in actor bugs?
• #28: We find that that our actor bugs mostly use 8 Akka API packages ranging from the package actor to serialization, to routing. Among these, the package actor is the most common, with 81% usage, and routing is the least common with only 0.2%. We also find that 97% of API usages are for only 3% of Akka packages that are specific to untyped and local actors.
• #29: An example implication for our API usages is that, current bug finding tools mine API usage patterns and find bugs that violate these patterns. One problem with these tools is that their mining processes do not scale to large code bases. However, we find that most API usages by actor bugs belong to a few packages for local and untyped actors. Current and future tools can use this to direct and thus scale their mining processes to large code bases.
• #30: Research question 4. How different are the characteristics of Stack Overflow and GitHub bugs.
• #31: We find that symptoms, root causes and API usages are significantly different among our Stack Overflow and GitHub bugs. For example, the Incorrect Messaging symptom is about 4 times more common for Stack Overflow bugs than GH bugs.
• #32: An example implication for our differences is that existing bug finding tools use characteristics of the bugs to build models that can predict bugs. However, similar models do not exist for actor bugs. Current and future tools can use our bug characteristics to build models that can accurately and separately predict both StackOverflow and GitHub actor bugs.
• #33: Two previous works are the most related to our work. First, Hedden and Zhao study 126 Akka actor bugs in 12 projects and classify their root causes. To compare, 4 of our root causes that account for about one third of of our bugs are new and cannot be found in this work. Second, Torres Lopez et al. study 23 actor bugs from 11 previous works and classify their root causes. To compare, 9 of our root causes that account for 85% of our bugs are new and cannot be found in this work. In addition, these works only study the root causes of actor bugs and not their other characteristics that we study.
• #34: Using tags, keywords, maturity criteria, and manual analyses to identify bugs and their characteristics is a threat. To mitigate this threat we closely follow the best practices of the previous work. Use of Stack Overflow and GitHub to find bugs is another threat. However, the large number of question/answers in Stack Overflow and projects in GitHub could help mitigate this threat. The complexity of concurrency bugs and its impact on their understanding, reporting, finding, and fixing is another threat. To mitigate this threat we closely follow the best practices of the previous work and study only bugs that have been found and fixed. *** END
• #35: To summarize, in this work, (1) we construct a set of real-world actor bugs from Stack Overflow and GitHub, (2) identify and classify their symptoms, root causes, API usages, and differences, (3) present examples of bugs with these symptoms and root causes, (4) study the relation of our findings with the findings of the previous work, and (5) discuss the implications of our findings.
• #36: All the data that we use in this study is publicly available at this URL. Thank you everybody and now I am ready to answer your questions.