A Community-Wide Infrastructure for Empirical Research in Architecture- Based Software Maintenance

REINFORCE workshop attendees. Prof. Sam Malek (far left), Prof. Nenad Medvidović (center, with glasses), and Associate Project Scientist Dr. Joshua Garcia (second from right).Over the past two decades, software architecture research has yielded many different tools and techniques for understanding the architectures of large software systems. However, the work in the area is still characterized by one-off approaches and evaluations on limited and/or proprietary datasets. More extensive empirical research and technology transfer are impeded by myriad disjoint research and development environments, lack of a shared research infrastructure, high initial costs associated with developing robust tools, and a lack of datasets needed to conduct empirical research in this domain. The resulting specialized solutions inhibit further advances in software architecture research, preventing systematic synthesis and empirical validation of novel or previously existing techniques. As a result, researchers conducting empirical studies and practitioners in need of cutting-edge tools must typically re-invent or re-implement research infrastructure (e.g., a seemingly promising technique for reverse engineering existing software systems’ architectures) from scratch. In doing so, they tend to repeat each other’s efforts, which wastes time and resources—especially if the previous solutions prove to be ineffective in the desired context—and repeats old mistakes instead of moving the field forward. As a result, the opportunities for potential breakthroughs in this domain are regularly missed and the field is replete with solutions that do not work as advertised beyond very limited scenarios and/or are not interoperable.

To address these challenges impeding empirical research in the domain of architecture-based software maintenance, a Software Architecture INfrastructure (SAIN) is needed that (1) provides existing tools and techniques often used in that area and domain of study to prevent researchers from wasting time trying to “reinvent the wheel” and makes those tools accessible and reusable; (2) contains a repository of architectural baselines, benchmarks, and datasets; and (3) serves as a means of integrating these tools in a manner that eases replication and reproduction of experiments, or construction of new ones based on existing tools, techniques, baselines, benchmarks, and datasets.

To meet the need for SAIN and understand the requirements of the targeted research community, four researchers have taken on the task of eliciting requirements from the software-engineering research community and running workshops based on the theme of designing and constructing SAIN. These researchers include: ISR Prof. Nenad Medvidović, a Professor in the Computer Science Department at the University of Southern California; ISR Prof. Sam Malek, an Associate Professor in the Informatics Department at the University of California, Irvine; Dr. Joshua Garcia, an Associate Project Scientist at ISR; and Prof. Mehdi Mirakhorli, an Assistant Professor at Rochester Institute of Technology, Department of Software Engineering.

Two workshops have been organized so far in support of designing and constructing SAIN. The first workshop on infRastructurEs and InstrumeNts For sOftware aRchiteCturE (REINFORCE), was held on January 11-12, 2017 at the University of Southern California, in Los Angeles. REINFORCE was structured as a mix of presentations by distinguished researchers and software architects from major companies, as well as “working sessions” intended to solicit the participants’ feedback on the aforementioned objectives of SAIN. Dr. Garcia gave a presentation introducing the original vision of SAIN and the challenges it hopes to address.

Each participant was asked to give a 10-minute lightning talk about architecture- based software engineering and the requirements the participant would like SAIN to satisfy. ISR Prof. Crista Lopes gave a lightning talk about her experience as one of the main architects of OpenSimulator, an open-source server platform for hosting virtual environments.

As a follow-up to REINFORCE, the First International Workshop on Establishing the Community-Wide Infrastructure for Architecture-Based Software Engineering (ECASE) was held at the 39th International Conference on Software Engineering (ICSE) in Buenos Aires, Argentina on May 22, 2017. In addition to the continued focus on requirements elicitation, planning, and design of SAIN, ECASE featured refereed paper presentations on novel tools and techniques, or useful benchmarks or datasets, that support empirical research on architecturebased maintenance.

Two keynotes were given at ECASE. Jane Cleland-Huang, a Professor in the Department of Computer Science and Engineering at the University of Notre Dame, gave a keynote presentation on lessons learned by the traceability community in their efforts to build a community- wide research infrastructure. Rick Kazman, Professor in the Information Technology Management Department at the University of Hawaii, gave a presentation on reproducibility problems in the field of software engineering and ideas about how to improve that discipline.

In the next phase of SAIN design and construction, Medvidović, Malek, Garcia and Mirakhorli will work to build: a means of gathering and organizing requirements online, including online surveys; a website showing the requirements the community have converged on so far; and an initial SAIN prototype. This prototype will serve as a pilot study for SAIN. The project will conclude with another workshop to assess the state of the infrastructure and how to move forward with it as a community.

The ECASE 2017 Proceedings is available on the IEEE Xplore and ACM Digital Libraries. For additional information, contact Dr. Joshua Garcia or visit the ECASE website

This article appeared in ISR Connector issue: