An effective method for face orientation estimation using depth–gyro sensor

In recent years, artificial intelligence has demonstrated tremendous potential for the development and advancement of a wide variety of technologies. A good example is facial direction estimation, which finds applications in driver assistance systems that prevent distracted driving, methods to prevent cheating in examinations, and software for creating three-dimensional (3D) virtual avatars.

Traditional facial orientation estimation techniques recognize the characteristic parts of the face, including the nose, eyes, and mouth, and detect their movements. However, such two-dimensional (2D) image-based methods raise privacy concerns and fail when features of the face are hidden due to a mask, or if the face is turned sideways. The solution may lie in optimizing facial detection using point cloud data (data obtained from a discrete set of data points) and a depth sensor. In fact, some previous studies have employed an estimation model based on the deep learning of 3D point cloud data in five face directions: frontal, diagonal frontal, right, left, and horizontal. However, considering the level of accuracy required for driver assistance systems that crucially verify the driver’s status, this five-class (k = 5) classification is insufficient for satisfactorily detecting the face direction.

To address this limitation, scientists from Shibaura Institute of Technology, led by Professor Chinthaka Premachandra of the Graduate School of Engineering and Science, have developed a more precise, horizontal wide-range angle detection approach (with k > 5). They accurately measured the horizontal angle of the face during the acquisition of the training data using gyroscopic sensors. Their paper was made available online in the IEEE Sensors Journal on 21 July, 2023.

In this study, the scientists gathered point cloud data from various orientations using a depth sensor, which was integrated with a gyro sensor during data collection. This data was employed to train a deep learning-based classification model, which was utilized for face orientation estimation. The scientists changed the horizontal angle of the face relative to the camera from +90 degrees to -90 degrees, using step sizes of 30, 22.5, 18, and 15 degrees between them. As a result, the classification of face direction was represented by more than seven classes (k = 7, 9, 11, 13).

“Precise training data for each orientation was obtained from the integration of the depth and gyro sensors, which reduce the number of point cloud samples required for constructing the classification model. Furthermore, applying a weight reduction process to reduce the weight of point cloud data enhanced training efficiency and resulted in fast face orientation estimation,” explains Prof. Premachandra.

The proposed classification method, designed for more than seven classes, achieves remarkable performance in face direction detection through deep learning. For example, it has demonstrated classification accuracy rates of over 98%, 95%, and 91% for 7, 9, and 11 classes, respectively, representing a significant improvement over the conventional face orientation estimation techniques.

Overall, this study opens new doors to a wide range of practical applications where accurate face orientation detection is required. As Prof. Premachandra explains, “Our findings offer a potential solution for addressing issues such as distracted driving, which may result from looking away or falling asleep while driving. Using this face orientation estimation method, it may be possible to alert drivers in situations of drowsiness and ultimately help in reducing the incidence of traffic accidents at a global scale.”

Let us hope that this novel face direction detection technology enables accurate face orientation detection soon! 

***

Reference

DOI: https://doi.org/10.1109/JSEN.2023.3296531

About Shibaura Institute of Technology (SIT), Japan
Shibaura Institute of Technology (SIT) is a private university with campuses in Tokyo and Saitama. Since the establishment of its predecessor, Tokyo Higher School of Industry and Commerce, in 1927, it has maintained “learning through practice” as its philosophy in the education of engineers. SIT was the only private science and engineering university selected for the Top Global University Project sponsored by the Ministry of Education, Culture, Sports, Science and Technology and will receive support from the ministry for 10 years starting from the 2014 academic year. Its motto, “Nurturing engineers who learn from society and contribute to society,” reflects its mission of fostering scientists and engineers who can contribute to the sustainable growth of the world by exposing their over 8,000 students to culturally diverse environments, where they learn to cope, collaborate, and relate with fellow students from around the world.

Website: https://www.shibaura-it.ac.jp/en/

About Professor Chinthaka Premachandra from SIT, Japan
Chinthaka Premachandra is a Professor at the Department of Electronic Engineering of the Graduate School of Engineering and the Manager of the Image Processing and Robotic Laboratory at SIT. He received B.Sc. and M.Sc. degrees from Mie University in 2006 and 2008, respectively, and obtained his Ph.D. degree from Nagoya University in 2011. His research interests include image processing, audio processing, intelligent transport systems, and mobile robotics. He is a Senior Member of IEEE and has received the IEEE Sensors Letters Best Paper Award and the IEEE Japan Medal in 2022.

Funding Information
This study was partially supported by the Branding Research Fund of the Shibaura Institute of Technology.