Immersive android avatar: enabling operators to inhabit

In a landmark study, a research group led by Associate Professor Yoshihiro Nakata of the University of Electro-Communications, Japan, and Assistant Professor Mizuki Nakajima of Tokyo Denki University, Japan, conducted groundbreaking research on android avatars for tele-communication. Published in IEEE Access, this innovative study improves the experience in tele-communications through android avatars.

This research, which aims to realize natural communication through avatars, focuses on the experience of both the operator and the interlocutor in tele-communication through avatars. An android avatar named “Yui,” with a human-like appearance and designed for operators to inhabit, was developed. This avatar is paired with an interface that presents information perceived by “Yui” to the operator, ensuring a high level of presence. Yui can deform and move in 28 points on its head, allowing it to reproduce human facial expressions through its advanced head deformation and movement capabilities. In addition, a stereo camera and stereo microphone are built into Yui’s eyes and ears, respectively, as sensory organs to acquire images and sounds that can be seen and heard at the location where Yui is, and this perceptual information is presented to the operator via a VR headset. The VR headset is equipped with a microphone for acquiring speech and sensors that can acquire changes in the operator’s head motion and facial expressions. The system synchronizes Yui’s facial expressions and speech with those of the operator by using this information. By integrating these, the proposed system provides a highly immersive experience for the operator and the interlocutor.

The developed system will provide operators and interlocutors with a high satisfaction by offering an experience closer to face-to-face communication than conventional remote communication. In addition, the system has the potential to both support operators and outperform face-to-face communication by emphasizing or suppressing the operator’s facial expressions.

Key contributors include:

1. Kaoruko Shinkawa (graduate student, The University of Electro-Communications)

For a more immersive understanding of our study, please visit [https://youtu.be/UXuQ1RgxPdQ] to watch our research video.