End Users as Developers in the World of Smart Things

The Internet of Things (IoT) refers to the emerging concept that everyday objects and environments become enriched with computing and networking capabilities in order to be more responsive to our needs. Specialized sensors (e.g. motion, temperature, video) monitor and detect specific conditions of interest and then stream data about them over the Internet for further processing, including triggering further actions in the physical environment through devices that can automatically control lighting, temperature, door locks, etc. By combining different specialized devices, our environments can become more responsive and “smart.”

Internet of People

Several technical, human, and social challenges are introduced by the emerging IoT infrastructure. Scalability, connectivity, and compatibility are some of the technical challenges that need to be addressed as the amount and types of interconnected devices increase. As for human and social challenges, questions arise about how users will prefer to interact with and be able to make effective use of these technologies. In this emerging context, users will need to perform an important role in order to tailor their own technologies to their specific needs, preferences, and contexts of use, as well as then understand automated behaviors. From security settings to custom functionality, end users will be left in charge of the task of paving “the last mile” and fine-tuning software and hardware to their needs. This will happen not only during the initial setup of a new device or system, but throughout the entire time that the technology is being used because users’ preferences, needs, and contexts change over time. For instance, in a family household, routines change due to the yearly school schedule, temperature preferences can change with the seasons or because you get sick, holidays and trips are planned, emergencies happen, etc. If this kind of technology is to support the numerous and varied situations we face in our everyday lives, it will need to support ways of continuous adaptation and modification, largely to be performed by the end users somehow.

End-User Development for the Internet of Things

End-User Development (EUD) is an approach to supporting end users (including everyday consumers) in tailoring their technologies on their own. In general, EUD has comprised techniques such as parameterization, programming by demonstration, scripting, and visual programming. Through these means, end users can customize the software they use to varying degrees. Recently, researchers have been looking into how EUD techniques, tools, and methods can be applied in IoT contexts in order to provide end users with tools to tailor their smart environments. Research carried out by Prof. David Redmiles and his team goes in this direction and tries to understand, at a fundamental level, what is different about EUD for the IoT compared to the other domains where EUD has been tried, what are the major challenges in this area, and how can these challenges be addressed in the design of tools for this context.

A Human-Centric Computing Framing of EUD for the IoT

Redmiles has always been interested in the human and social aspects of software engineering and development, including how end users can provide feedback to developers and how EUD can enhance collaboration among collaborators. One way to look at EUD is as a collaboration between developers and end users through the technology in use. Both parties perform certain development tasks that align to each other and improve the final product. In this perspective, technology development tends to be more inclusive and participatory with the end users having an active role in the development process.

From December 2016 through November 2017, Redmiles hosted Bruno Azevedo Chagas as a visiting researcher from Pontifical Catholic University of Rio de Janeiro, Brazil. Chagas is a Ph.D. student advised there by Professor Clarisse de Souza, an expert in semiotics and a novel software technique called semiotic engineering. Chagas has been researching the topic of EUD for the IoT, taking a human-centric computing perspective. In his Ph.D. research, he is applying semiotic engineering in this domain, which fits within the perspective that interacting with computer technologies is a communication between developers and users.

Initially, they have proposed a framing of EUD for the IoT where interaction in this context is viewed as a group communication scenario where users and developers “talk” to each other indirectly through different system interfaces (Fig. 1).

Breakdowns in EUD for the IoT

By experimenting with some commercially available IoT devices (e.g. the Echo Dot, Philips Hue, and Flic Button, among others), it became clear that disruptions in the interaction process are inevitable and happen whenever a mismatch in expectations is faced and/or mutual understanding is lost. This situation parallels what happens in regular “live” group conversations between people. In regular conversations, people can usually detect and restore these situations by employing repair strategies, such as “Excuse me,” “Could you repeat please,” “What do you mean?” and so on. However, when interacting with technology through interfaces, such repairs are limited to the mechanisms that systems developers have anticipated and implemented. Chagas and Redmiles call these disruptions breakdowns to designate the precise moment where the user will become motivated, or at least intrigued, to perform a modification in the system. Therefore, EUD (for the IoT) would be the activity of performing a repair in the technology as a consequence of a breakdown.

Currently, Chagas, Redmiles, and de Souza are trying to gain knowledge about breakdowns in this context: their kinds, causes, how people react to them, and people’s approaches to repairing them. They have just completed a diary study with several participants who were each given a set of 5 commercially available IoT devices for smart homes: a smart plug, a smart motion sensor, a smart wireless physical button, a smart light bulb, and a voice assistant. In order to mimic a heterogeneous IoT environment, devices were all picked from different manufacturers and functional compatibility between them was provided by the free online service IFTTT – If This Then That, which is an EUD tool that allows users to build and use small programs called “applets” using a simplified trigger-action language. This setting enables all devices to be combined with each other and, adding other online services and apps which can also be used with IFTTT (e.g. weather forecast websites, social networks, calendars, etc.), a reasonably large number of functions and variations can be created. Participants were asked to report regularly over four weeks and were interviewed at the end of the study. In so doing, data was accumulated about the typical breakdowns that happen over time and the co-evolution of end users’ behavior and fixes to the breakdowns. Stay tuned as analysis of the data is on-going.

For more information, visit Redmiles’s Collaboration Research in Action, Design, and Learning (CRADL) group website and de Souza’s Semiotic Engineering Research Group website, or contact Professor Redmiles and  Chagas.

Research partially funded by two Brazilian government agencies, CAPES (Grant No. 88887-122734/2016-00) and CNPq (Grant No. 307043/2013-4).

[1] B. A. Chagas, D. F. Redmiles, and C. S. de Souza, “End-user development for the Internet of Things OR How can a (smart) light bulb be so complicated?,” in 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2017, pp. 273–277.