Interview with researchers 16 Elucidating the functions of emotion
via research on expressions Profiling facial expressions and
communication to understand the true
nature of our interface with the outside world

Searching for connections between expression and emotion via the dual approaches of psychology and physiology

Communication among people is not simply an exchange of information using words. Visual information is also taken in, including gestures and facial movements. Especially, information from reading faces is thought to be extremely important in communicating.

Specializing in emotional psychology and psychophysiology, Associate Professor Fujimura has made steady progress on elucidating human expressions. “ Psychology of emotion aims to clarify the triggering mechanisms and functions of emotions, including happiness, anger, and affect. Meanwhile, the field of psychophysiology studies relationships between the mind and the body as a means of discovering the structure and behavior of the mind. For example, the pulse rate of most people increases when they have to make a speech. We try to understand this mental state called ‘nervous tension’ using such physiological changes. We are especially moving forward on research that seeks to uncover relationships between expressions and feelings in communication, and to find out how people ‘read’ the expressions of others.”

Using “congruent facial expression” to determine the role of expressions in communication

“When you smile, the whole world smiles with you” describes an experience familiar to everyone: smiles are naturally “contagious.” This kind of “facial mimicry” is known by various terms in the scholarly world, including “congruence of facial expression.” This “congruence” （similarity） in the response to people’s expressions has been a long-term theme in Dr. Fujimura’s research. It is well-known from previous research that cheek muscles move in response to a smile to form a similar smile, while the muscles that control the eyebrow areas move to a frowning position when one encounters an angry person. Dr. Fujimura thought that changes in facial expressions might be dependent on the level of trust one has in the other person. “My idea was that mimicry would differ depending on whether one wanted to forge a better relationship with the other person. So, I started by looking at the degree of trustworthiness, to see how this social factor of interpersonal relationships impacts facial mimicry.”

To verify her hypothesis, Associate Professor Fujimura started by using the “trust game,” a method often used in game theory and economic research. Here, a test participant pairs with a virtual “person” （here, “the Pair”） projected onto a computer screen. The participant plays the role of an “investor,” while the Pair is the “trustee.” Points are assigned to the investor, and any points “invested” （assigned） to the Pair （trustee） are multiplied by three （x3）. The Pair may then allot （“entrust”） the investor with some or none of the points assigned to the Pair as trustee （see Fig. 1）. Prior to playing the game, the participant is informed that points invested will be multiplied by three, and that the points finally held by the participant will be reflected in the remuneration that participant will receive for her/his participation. Here, two pattern types are prepared beforehand for the points returned to the investor by the Pair （as trustee）. When the Pair tends to assign （return） to the investor half of the points （multiplied by three） given to it, the Pair is a “trustworthy person.” Meanwhile, if the Pair tends to keep most of the points, then the Pair is an “untrustworthy person.” After the trust game is completed, a facial expression showing the “emotion” of the Pair is displayed to the participant, while any changes in the expression of the participant are measured using facial electromyography.

Fig. 1 Outline of Trust Game

[first box at left with face]
How many points will you pass over to me?

[middle box with face]
the Pair [6]
You [0]
The number of points has tripled.

[right box with face]
the Pair
You
Points have been reassigned to you.

[text on bottom timeline]
gaze fixed point
the Pair’s face
selection
presentation of selection made by test participant
points are tripled
presentation of selection made by the Pair
execution period interval

The results showed that while the participant mimicked the smile shown by the “trustworthy person” (the Pair), no such mimic response occurred when the “untrustworthy person” was displayed as smiling. It was also found that emotional (facial expression) mimicry in the case of anger or sadness (by the Pair) was not affected by the level of trustworthiness. “Although it was known that emotional mimicry was impacted by differences in the attributes and general impression of the other party, we were able to see changes in mimicry when we added the social factor of trustworthiness (Fig. 2). Mimicry of a smile conveys a message of affiliation to the other smiling person, while refusal to mimic the smile of an untrustworthy person might be a way of ‘adjusting’ the relationship to fit the circumstances.”

Fig. 2 Test results
Changes in amount of cheek muscle movement (extent of facial mimicry) in response to trustworthy person and untrustworthy person

[open circle] Trustworthy person
[closed circle] Untrustworthy person

[y-axis text] EMG change values (z scores)
[text under x-axis] time

This prior research by Dr. Fujimura on the nature of communication as seen in emotional mimicry will be expanded in new studies. “Actually, there are two kinds of ‘trust.’ One is ‘trustworthiness,’ where one undergoes changes in trust level according to the actual behavior of others. The other is ‘general trust,’ which is a measure of how easy it is in general for an individual to come to trust others. As facial and other expressions serve as signals for building trust with another, then I hypothesize that people whom others find easy to trust are likely to be more sensitive and ‘attuned’ to the expressions of others; thus, facial mimicry might more readily occur in so-called ‘easy to trust’ people. Little data exists on the relationships between general trust and mimicry, so I definitely plan to investigate this.”

Dr. Fujimura is also engaged in the development of an expressions database that incorporates both categories and dimensions of emotions. Research on expressions in psychology and other fields requires a set of “standard” expressions for use as stimuli in tests. Already there exist for-pay facial expression databases, their costs have been prohibitive for ready use by researchers and students. Thus, Dr. Fujimura is constructing a database that will be open and free of charge to the general public.

In order to collect data for standardized expressions, Dr. Fujimura selected in tryouts four men and four women, ages 20s through 40s, who are skilled aspiring actors who can make easy-to-identify expressions. She then used a high-definition video camera to record a variety of their expressions. There are twelve expression types included in the database, including surprise, fear, sadness, anger, etc. Then, she collected data on the emotional states of each expression by using emotion classification in which participants were asked to choose the emotion label that best described the shown expression, and a single-item scale of pleasure and arousal. This database is widely used by researchers, as reflected in the growing number of references to this database cited by scholars in their published papers, etc. “We made this database as easy to use as possible in tests, by including all statistical information on ratings made for each specific emotion image. I have been contacted by persons who have used this database for presentation at scholarly conferences, etc., and some of these contacts have actually led to joint research. Certainly, this kind of research involves painstaking efforts, but it has certainly been worth it!”

A yearning to clarify the roles that human emotions play in building society

The inspiration for Associate Professor Fujimura to research emotions and expressions stems from her childhood. “Ever since I can remember, I have loved to watch people. I would ask myself, ‘Why is this person saying this?’ or ‘What is the meaning of this action or behavior?’ Today, we have physiological indices that help us to measure human emotions, including heartbeat, brain waves, etc. However, all of these are measurements made from outside the mind and body. The reason why I insist on studying expressions is that the expressions themselves have an impact even on the person who makes the expression, and that expressions serve as signals to others. In this way, emotions act as a hub or interface between oneself and the world. My desire is to clarify how expressions are used in the construction of human society. I do this by examining the relationships between emotions and expressions, and by determining how people ‘read’ the meanings communicated by expressions within these relationships.” Dr. Fujimura says this with a smile, and with her eyes sharply fixed on her ongoing research.