Facial Expressions Based on the Types of Conversation Contents

This study analyzed changes in facial expressions over time to examine how rapport is formed when discussing any topics. Changes in facial expression are captured by action units (AUs) and 55 locations in the 2D eye region. The topics included (a) introduction and greeting, which are conventional conversations; (b) favorite food; (c) journeys and watching movies; and (d) the future where humans and AI interact. The data used in this study are 15–20-min recorded videos (.MP4 format) between 29 human participants and a virtual agent named Hazumi1902, operated by the Wizard-of-Oz method. Multimodal information generated by the participants during the dialogue was recorded using video and a Microsoft Kinect sensor. The AUs and locations of the 2D eye regions were detected by OpenFace 2.0. The intensity of AUs and the location of 2D eye region landmarks were analyzed using the Kruskal–Wallis test by combining the conversations and rapport criterion-type measurements. This study found that the intensity of AUs was insufficient, and the 2D eye region landmarks was necessary for analyzing how facial expressions were affected by perceptions of rapport and conversation type. One of the criteria measurements,” cold,” was not observed in the intensity change of the AUs. It was concluded that the AUs were not universal, and the location of 2D eye region landmarks played a crucial role in complementing the analysis of their intensity. The intensity of the AUs and the location of 2D eye region landmarks were observed in harmonious conversations. It was discovered that factors hindering the rapport were in the eyes, whereas those promoting rapport were in the AUs. These insights could be invaluable in various fields, from human–computer interaction to non-verbal communication.