Miryung Kim

An abundance of data in science, engineering, national security, and health care has led to the emerging field of big data analytics. To process massive quantities of data, developers leverage data-intensive scalable computing (DISC) systems in the cloud, such as Google's MapReduce, Hadoop, and Apache Spark. While DISC systems help to address the scalability challenges of big data analytics, they also introduce new challenges in debugging. In this talk, first, I will first describe interactive, real-time debugging primitives that we designed for the next generation data-intensive scalable cloud computing platform, Apache Spark. Second, I will briefly describe data provenance and optimized incremental computation capabilities that we built within Apache Spark to effectively and efficiently support debugging. Third, I will describe automated debugging in DISC called BIGSIFT that combines insights from automated fault isolation in software engineering and data provenance in database systems to find a minimum set of failure-inducing inputs. Compared to state-of-the-art approaches, our approach improves fault localizability by several orders-of-magnitude, and improves performance by up to 66×, at which BIGSIFT is able to localize faults faster than the original job running time.