Korean Educational Exchange – Crowd Design Research

For Summer and Fall 2017, ISR faculty member André van der Hoek and his research lab (Software Design and Collaboration Laboratory) hosted undergraduate students from two Korean universities – Ajou University and Kookmin University – as part of the UCI Undergraduate Research Opportunities Program (UROP) International Summer Undergraduate Research Fellowship (I-SURF) program. Originally slated to be a three-month summer visit, the research project was such a success that two students stayed on for an additional three months this Fall quarter to further the research.

The International Summer Undergraduate Research Fellowship program brings outstanding international undergraduates to UCI to both experience and contribute to cutting-edge research. van der Hoek saw in this program an opportunity to expand on his ongoing research into crowdsourcing and design in the context of a large NSF grant he shares with his collaborator and former PostDoc Thomas LaToza at George Mason University.

In previous work, with his former M.S. student Consuelo Lopez (who, at this time, is a product manager at Flybondi – a new Argentinian airline – developing its reservation system), van der Hoek demonstrated that a crowd of workers from Amazon Mechanical Turk can rapidly create a broad range of alternative solutions to small design problems. For each such design problem, termed a decision point, Lopez and van der Hoek found that the collection of alternatives produced by different workers not only is highly diverse, but also contains innovative and high-quality solutions. Compared to what an individual software designer could produce in the same time, a crowd of workers offers a rich variety of alternatives, with out-of-the-box solutions and, at times, great amounts of detail.

The challenge that van der Hoek asked the Korean students (Sanghun (“Jess”) Kim, Seung Sup (“Sup”) Kim, Dae Gweon (“Alex”) Ko, and Juhyeok (“Dan”) Seong) to address is the step that comes before the generation of the alternative solutions: is it possible to take a requirements document and use the crowd to turn it into a set of decision points that, together, form the core of the design problem underneath the requirements document? Taken at face value, this seems like a daunting challenge. Even experts typically will spend weeks, if not months, on a task like this.

van der Hoek and the visiting Korean students’ approach, however, seeks to leverage the wisdom of the crowd in addressing this problem. They specifically are exploring a two-stage process in which a first crowd identifies ‘important words’ in the requirements document and a second crowd designs, iteratively improves, and eventually ranks decision points for different pairs of words. The idea is that, by using the masses, pairs of words that are identified by multiple workers tend to be more important and that, by using iteration over multiple workers, initially, rough decision points in draft form can be refined to be high quality.

To test these ideas, the Korean students have implemented a pair of tools that they are now readying for an actual experiment with Amazon Mechanical Turk workers. Under the supervision of Ayushi Rastogi, a post doctoral researcher in van der Hoek’s group, a detailed experiment has been designed and will be executed in January. The Korean students will have returned home by then, but the cooperation and exchange has been so successful that they are committed to seeing the experiment through, actually collecting crowd work, analyzing the results, and producing a paper.

During the stay, Prof. Seok-Won Lee from Kookmin University visited to check on the team, as well as other student teams working with other professors at UCI. His visit happened to take place during the same time that van der Hoek’s long-term collaborator, Dr. Marian Petre from The Open University in the U.K. visited to work with van der Hoek on outlining their next book on software design. Having everyone here together led to – in the good tradition of ISR! – designs being discussed, preliminary demos being shown, next steps being debated, and fun being had.

Overall, van der Hoek could not be happier with the exchange: “I have enjoyed it tremendously and am 100% ready to repeat next year. The I-SURF program has brought us fantastic students eager to learn and participate, engaging deeply with the research, contributing key ideas, and producing really important tools helping the research along. Thank you Jess, Sup, Alex, and Dan!”

For more information, contact Prof. van der Hoek.