DUO_Master Thesis, TU Delft

Enhancing Daily Digital Communication Through Instinctive Action Tendencies of Emotions

As human beings, communication is core to who we are as a species (Samovar, L., Porter, R., & McDaniel, E., 2009). Communication is also the main format of human interactions. Personally, I find it really fascinating to explore new formats of connecting people and enriching digital communications.

Our daily communication consists of two main parts, verbal communication that mainly in charge of information and nonverbal communication that mainly in charge of emotion. However, currently our main mean of daily digital communication is still through text messages or phone calls, even with the newest smart gadgets, such as smart watches. Body language which accounts for 55% of our daily communication is not effectively in use for digital communication on a daily basis.

For my Master thesis, I combined all the theories and techniques I have learnt through my study at TU Delft and internship, to research and try to design a product system enables users to communicate their emotions by leveraging their body action tendencies that emerge corresponding to different emotions. The aim is to explore humane and novel applications of existing sensor technologies.

Mentor: Arnold P.O.S. Vermeeren (Assistant Professor at TU Delft), Aadjan van der Helm (Lecturer/Researcher at TU Delft)

Affiliation: idStudioLab, TU Delft

Personal Project

09/2015 - 03/2016

Responsive image

DUO is a concept IoT system which enables the user to communicate their emotions by leveraging their body action tendencies that emerge corresponding to different emotions. For example, when you feel angry, you want to punch or squeeze stuff. DUO reduces users’ dependence on having to encode their mental state into screen-based messages so they can be better engaging in their daily routine while establishing empathy with their multiple relatives and intimates in a dynamic, straightforward way.

In its first stage of implementation, the DUO system consists of two components, the physical interactive product as a physical extension of a mobile phone to translate body actions into dynamic visualisations, and the mobile application that serves as an universal tool which can be accessed both by users who have possession of the physical product and users without the product but who are willing to keep track of others’ emotions. Instead of only using the touch screen on the front surface of our mobile phones, the proposed physical embodiment creates potential to utilise the remaining surface area of our mobile phones as a physical interface to communicate with body actions. The DUO system takes short time to encode emotion messages instinctively while we are engaging in our daily routine, the actions along with their emotions are dynamic, real-time reflected. Users can even perceive the emotion intensity through the animated visualisations reflected from the sender’s actions. Through this way, users are also able to review their past emotions in a dynamic way through simulating past emotion actions, so as to improve their emotion self-awareness and management.

The full version of the thesis report is available for review here, with appendix here

Responsive image

. . .

Project Background and Motivation

Responsive image

Looking into the history, it happened often that new inventions of human communication caused social innovation along with human behavior change on social interactions (Tomasello, M., 2010).
We are now seeing the start of a dynamic medium. To a large extent, people today are still using this medium merely to emulate and extend static representations from the era of paper (Bret Victor, 2014). Andy Goodman, Design Director at Fjord Design, also brought up the term Zero UI. He argues, “a large amount of design work these days is visual. That makes sense, since the most essential products we interact with have screens. But as the Internet of things surrounds us with devices that can hear our words, anticipate our needs, and sense our gestures, the future interaction will turn to haptic, automated, and ambient interfaces. He calls it Zero UI (What Is Zero UI?(And Why Is It Crucial To The Future Of Design?, 2015).“

However, the main format of daily digital communications are still text messages and phone calls, like what it was like 20 years ago. With the emerging smart gadgets, we always seem to fall into the trap of automating everything, even so far as attempting to automatically interpret and communicate our emotions. Is technology getting too advanced nowadays? Somewhere along the way, we seem to have lost the autonomy and control we are supposed to enjoy as human beings.
Based on this background and motivation, the project brief was formulated:

. . .

Main Features of the Final Design

Emotion actions
The biggest privilege of DUO system is that it is able to communicate various emotions (at this stage of the project, Anger, Apprehension, Loneliness, Anticipation and Ecstasy) with instinctive body actions through a physical product regarding its format and design language.

Physical Feedback of Communication
As an IoT system of daily communications. Apart from empowering users to pass through their emotion states straightforward, the system also provides unobtrusive and comforting feedback to establish emotion self-awareness of the sender. As shown in Figure 44, referring back to Figure 7, actions with feedback on the physical product establish emotion self-awareness of the sender and contribute to a better communication circle.

Dynamic visualization
The intensity of the user’s emotion actions is translated by the physical product into a dynamically visualized animation in the mobile application. His/her contacts are able to view the real-time generated visualization through their mobile phones and keep track of the user’s mental statues

History review
The history review and management function in the mobile application allows the user to review their emotions in the past weeks in a visual organized way. In this way , it is possible for them to review the intensity and the dynamic visualization of their emotions afterwards. It is also possible for them to review recent updates on other connected users’ emotions.

Dynamic empathy
A unique feature of DUO system is that it enables the user to review and simulate their past emotions in a dynamic way by repeating the actions with the physical product to the same degree as their past actions. Through this way, they are empowered to empathize with the emotion experienced during that moment and better manage their emotions in the future.

Real-time notification based on preference
The application provides users with the freedom to receive real-time notifications by adjusting the scale of the emotion intensity. Once two users have adapted to each other’s emotion action behaviors, they are able to follow each other’s emotion changes based on personal preference and with more flexibility.

Responsive image

. . .

Technology Implementaion

The physical embodiment for the product has been proposed with the consideration of implementation of current technologies. By applying existing technologies to enhance the digital communication experience of users, a new meaning of interaction with more humane consideration can be achieved.

Responsive image

. . .

The DUO Application

The mobile application of DUO serves as the portal for the recipients to access users’ dynamic generated visualizations of the emotion actions and to keep track of their emotion status in real time on demand. It also allows the user to keep track of multiple people’s emotion status by providing them with real-time visualization summaries and notifications. The recent history of the emotions and their animated visualizations are stored in the application for review by either the user or their recipients with duly applied protocols and controls for privacy. Simple functions of emotion management are provided. The uniqueness of using DUO system to review emotions is that it enables users to repeat the same intensity of past emotion actions by integrating sensor technologies and digital data.

Responsive image
Responsive image

Responsive image
Responsive image
Responsive image
Responsive image

Responsive image
Responsive image
Responsive image
Responsive image

The demonstration of the DUO system for the four emotions at current stage

. . .

Process Overview of the Thesis

Research Landscape

A ViP (Vision in Product Design) framework was adoptted to structure the research phase of the thesis project. Research questions are:
• What are the elements contributed to a good dailycommunication? What are the barriers of a good communication?
• What’s specific about emotion communication and its current roles in digital communication?
• What are the main channels of expressing and interpreting messages especially emotions nowadays? And what are their advantages/disadvantages compared with each other?
• What are the new exploration and trend in the domain of digital communication and emotion sharing?
• What will be a meaningful target group?
• What are the emotions that mostly hoping to be shared or to establish empathy?
• What will be a meaningful relationship for emotion communication?
• What will be a meaningful moment/context for emotion communication with tangible interactions?
Based on existing knowledge, research intention and questions, a visual landscape of the research scope was mapped out on the right hand.

Responsive image

Trend analysis

DUO-trendanalysis.jpg

The trend analysis was not restricted to only emotion communication, but also looked into insightful projects related to general nonverbal connectedness or emotions. The vertical axis is about the using context, which shows projects from personal level (namely user is the only one getting access to the product and its data) to intimate relations such as family members or lovers, to close friends, and to the level of exposing to social network or public places. The five divisions of using contexts will be further applied for defining the target context of my project. The horizontal axis displays user’s control level (namely whether the user is able to take in charge of the interactions) of the technologies or design from low to high.

Most of them monitor user’s data and communicate them through the designed interactions. Users don’t have much control of the data. They are more like tools for users to know or to exhibit their emotional data or status. These types of projects mainly fall into the categories such as tools for emotion selfawareness, wearable aesthetics that monitor vital signs, and real-time tracking systems.

More to the right side on the diagram, other projects are controlled or initiated by the user, and users have the freedom to choose whether to be connected, whether to continue or stop the pre-programmed communication. In the trend analysis, these types of interaction mainly are in the categories of “user triggers the interaction” and “extra service added to an existing product”. Sensor technologies are largely in use in the above-mentioned project categories.

It is reveled in the matrix that a large part of design with high-level of user’s control (in a way they wanted) on interactions are the ones take use of the advantage and interference of smart phones. They leave the users more freedom to have a variety of autonomous usages and formats. Particularly, the application of IoT (Internet of Things) and the rich formats they are able to create brings more diversity to the projects.

Preliminary User Research

Possible clusters are revealed after foundational research. These three main clusters lead to the conclusion of the foundational research: There exists a great design opportunity to empower people by using body actions in digital communication to enhance emotion awareness in the communication process. It should also be a prompt way for people to connect through long distance. In the meanwhile, the technology development nowadays is able to provide people with more autonomous choices of communication repertoire.

Nevertheless, what will be a meaningful target group for the project brief to focus on? What kind of relationships requires emotion awareness and the emotions users would like to share or get empathy with the most? In order to come up with new design opportunities, it is important to understand in the real life what are the pros and cons when people are using current communication tools. What are the situations the current communication tools are not fulfilled?
To answer the above questions, the user research was designed to generate insights from user’s perspectives.

DUO-userresearch1.jpg
DUO-userresearch2.jpg
DUO-userresearch3.jpg
DUO-userresearch4.jpg

Statement Definition

In a statement definition, all the context factors gathered from foundational and user research were distributed along two dimensions: the first dimension refers to the distance of the communication (face to face vs. long-distance), while the second dimension refers to the engagement of people in either communication of other life events. These two dimensions are visualized in the diagram as x- and y-axis respectively. As shown in the diagram, the design solution focuses on communication type 2 (long-distance communication with the main person involved in life events). The context factors of this type of communication were displayed in turquoise. The design interest is to bring factors of communication type 4 (face to face communication with engaged communication) into communication type 2 in order to empower people in communication type 2 by simulating communication type 4.

To conclude the statement definition,
I want to empower college students or young workers to be able to establish emotion awareness through instinctive1 body actions in communication with their long-distance relatives and intimates while engaging in daily routine.

Responsive image

Co-creation Workshop

Responsive image

The co-creation workshop was organized before ideation phase because of the following intention:

To observe the action tendency or thought-action tendency of target emotions;
To compare the pros and cons of physical interpretation and graphic interpretation;
To generate insights for ideation and to prevent personal interests and bias.

The insights and results from co-creation workshop were used to support ideation and concept development. Emotion action tendencies towards positive, negative and mutual emotions were categorized. Combining insights from trend analysis, one important decision after co-creation workshop is to involve smart phone assistant into the product system, out of the reasons: Improve user’s control level of interaction and data;
Quick launch and implementation;
Variety of usages and formats;
Extra meaningful functions;
Variety of feedbacks than physical actions;
Prevent social awkwardness

Responsive image

Ideation Mindmap & Concept Sketches

DUO-mindmap.jpg
DUO-sketch.jpg

User Test 1

From the sender’s perspective, the first user test was aimed mainly for testing different product formats and the intended interactions for each emotion. From the receiver’s perspective, the test was aimed for testing different graphical elements and movements. In user test 1, the prototypes were created for testing separate design ideas about the formats and interactions for different emotions. And the corresponding perception on graphical elements and movements. The process was utilized to verify the chosen concept features of the system, and to mainly get insights of the selection of product format, material, shape, size, graphical elements for further concept development.

DUO-us1-1.jpg
DUO-us1-2.jpg

User Test 2

The main goal of this user test was to evaluate if the chosen design concept adequately addresses the objectives of the project brief, which is about developing meaningful tangible interactions with digital system to help people express and interpret daily emotions in a humane way. Moreover, the testing would help evaluate if the concept fulfills the five design specifications and product qualities previously identified.
The two main purposes were: 1.The test helps refine the user experience of both tangible and digital interactions involved in the system. 2. The concept was compared to Emoji, the current popular communication system. As they are both digital communication systems specifically targeting on emotions. This helped in identifying the unique values and selling points of the proposed concept system.

Responsive image
Responsive image

Final Evaluation

In order to fulfill the design specifications and product qualities, the main objective of the user evaluation was to test if the final concept proposal had improved user experience and interaction qualities that address the feedback and suggestions from user test 2. The evaluation process utilized a similar testing method as in User Test 2, consisting of user interview and observation by interacting with functional prototypes and answering interview questions.

Responsive image

Responsive image

Overal Conclusion of Achievement

The criteria list on the right is an encapsulation of the design specifications and product qualities that was aimed to be achieved through the final design. Through user test 2 and user evaluation, all the criteria were developed into interview questions and user observation, it was possible to weight how far the final concept embodiment has achieved each criterion according to test feedback. A fully achieved criterion was equivalently rated 100 Percent. Other percentages given for each criterion were not used for quantify the achievement, but were served more like qualified experience from user tests.

Responsive image

Concept Value of DUO

For the test, more than half of the participants brought up that Emoji is more like a casual and fun way of communicating emotion. The diverse choices of emojis make their communication fun, but also cause perceptual gaps among different people in various relationships. They hoped there would be a more straightforward and serious way of communicating emotions. The large amount of choices cost more time on encoding a single message about their emotional status. What’s more, Emoji was considered as a universal communication tool that can be used in multiple relationships, while the concept system was considered more personal and reliable. Even though Emoji provides diversity, almost all participants could not find a proper way to translate the intensity of their emotions into emojis. Some of them mentioned to duplicate the same emoji several times, others mentioned they had to further explain how strong the emotion was in text. However, except for the test result of Tiredness, almost all participants considered the concept system as a more competent tool to express their emotion intensity because of the dynamic actions. Furthermore, an added value of the concept system mentioned by the participant was that it helped release negative emotions to some degree.

In conclusion, as expected in the test purpose, the feedbacks were analyzed into a list of concept values that is able to distinguish the system from current communication experience

Responsive image