Noncontact 3D gesture recognition enabled VR human-machine interface via electret-nanofiber-based triboelectric sensor
image: A noncontact virtual reality (VR) human-machine interface (HMI) was developed by electret-based triboelectric sensor (ETS), which could accurately recognize threedimensional (3D) gestures and largely improve degree of interaction freedom in VR. An intelligent noncontact VR interactive system based on the ETS is further demonstrated to manipulate game characters, indicating great potentials to develop a new generation of noncontact VR HMIs and metaverse.
Credit: Nano Research, Tsinghua University Press
In a significant advancement for noncontact VR HMI, researchers from Zhengzhou University have introduced an intelligent noncontact VR interactive system that could drives VR HMI evolution toward greater interaction freedom. The system, developed by a team led by Yanchao Mao, integrates deep learning with ETS to recognize noncontact 3D gestures for noncontact VR game operation.
The traditional modes of VR HMIs mainly rely on handheld devices and keyboards. Handheld devices typically rely on simple and large-range limb movements rather than diverse and delicate gestures, resulting in the low degree of interaction freedom. The keyboard-operated interfaces can only recognize 2D gestures such as tapping and swiping, which is unable to distinguish 3D gestures and also leads to reduced interaction adaptability. The newly developed system offers a noncontact VR HMI. People can interact with VR intuitively through freehand gestures without the burden of wearing cumbersome devices.
The key to this system’s innovation lies in the ETS, which is fabricated through an electrospun PLA/TPU electret nanofiber film. The high surface roughness of the PLA/TPU electret membrane ensures long-term charge retention stability. By integrating a deep learning based MLP neural network, the ETS achieves an average accuracy of 97.3% when recognizing 18 different types of 3D gestures. This recognition is linked to a VR platform where people can interact with virtual environments, enhancing the gaming interaction experience through immersive training.
Professor Yanchao Mao, the lead researcher, emphasized the significance of this innovation: “Our goal is to eliminate the need for inconvenient hand-worn devices. By integrating deep learning and ETS technology, we can provide people with a more freedom, and accessible game control process. People can now control immersive VR game characters through high freedom 3D gestures.”
This work introduces the triboelectric 3D gesture recognition method to the VR HMIs, and could make the interaction between human and virtual environments become more efficient and fascinating. Noncontact HMI technologies, particularly those based on gesture recognition and ETS, offer promising prospects for VR and immersive systems. By eliminating the need for handheld or wearable controllers, these approaches enable intuitive, unrestricted interactions, enhancing user comfort and engagement. In VR applications, such systems can facilitate natural gesture-based manipulation, immersive gaming experiences, and realistic virtual training scenarios.
Future directions and applications: looking forward, the team plans to further optimize sensor performance and integration, develop more robust gesture recognition algorithms, and explore novel applications, aiming to establish practical, user-friendly, and widely deployable noncontact HMI systems. Beyond immersive VR entertainment, this system also holds potential for medical rehabilitation, and smart environments, and remote collaboration, where seamless, non-invasive interfaces are essential. With ongoing advancements in sensor sensitivity, material flexibility, and signal processing algorithms, noncontact HMI and VR technologies are expected to become increasingly practical and widely adopted, driving the next generation of immersive, user-centric digital experiences.
About Nano Research
Nano Research is a peer-reviewed, open access, international and interdisciplinary research journal, sponsored by Tsinghua University and the Chinese Chemical Society, published by Tsinghua University Press on the platform SciOpen. It publishes original high-quality research and significant review articles on all aspects of nanoscience and nanotechnology, ranging from basic aspects of the science of nanoscale materials to practical applications of such materials. After 18 years of development, it has become one of the most influential academic journals in the nano field. Nano Research has published more than 1,000 papers every year from 2022, with its cumulative count surpassing 7,000 articles. In 2024 InCites Journal Citation Reports, its 2024 IF is 9.0 (8.7, 5 years), and it continues to be the Q1 area among the four subject classifications. Nano Research Award, established by Nano Research together with TUP and Springer Nature in 2013, and Nano Research Young Innovators (NR45) Awards, established by Nano Research in 2018, have become international academic awards with global influence.