New augmented reality tech can turn any surface into keyboard

Virtual keyboards are a frequent source of frustration for augmented reality (AR) users. The virtual surfaces are slow and error prone, and raising an arm to type on them can cause muscle strain known as “gorilla arm.”

To improve virtual-typing experiences, University of Texas at Dallas researchers have designed a unique interface that allows users to transform everyday objects into typing surfaces within an AR environment.

The patent-pending technology, called PropType, overlays an augmented keyboard surface onto a handheld object and can adapt to curved surfaces.

Student researchers have created a video demonstrating PropType being used on surfaces such as water bottles, coffee cups, books and soda cans.

“By integrating objects already present in the user’s surroundings, PropType reimagines text input in AR, creating a seamless connection between the physical and virtual worlds,” said Dr. Jin Ryong Kim, assistant professor of computer science in the Erik Jonsson School of Engineering and Computer Science. “PropType capitalizes on the tactile feedback provided by the objects themselves, offering better key confirmation and reducing reliance on visual cues.”

The technology provides an alternative to common AR typing solutions such as external physical keyboards, which can disrupt a device-free immersive experience.

“PropType leverages the familiarity of handheld objects to offer a more intuitive and accessible alternative to traditional keyboards, particularly in mobile or hands-free scenarios where conventional input methods are impractical,” Kim said.

Researchers in Kim’s Multimodal Interaction Lab presented PropType and received a Best Paper Honorable Mention Award in April at the Association for Computing Machinery (ACM) CHI Conference on Human Factors in Computing Systems, the world’s premier conference in human-computer interaction, held in Yokohama, Japan. They later showcased the work in September at the 38th Annual ACM Symposium on User Interface Software and Technology in Busan, South Korea.

Developing PropType involved challenges such as adapting the interface to different shapes and sizes of objects. The researchers studied how 16 participants interacted with props to understand grab postures and typing gestures. Then they developed custom keyboard layouts for different objects. The researchers included an editing tool that allows users to customize keyboard layouts and visual effects.

Kim’s research focuses on haptics, or touch and physical feedback in digital systems, which can include vibrations and heat.

“Touch carries a lot of information; it’s another form of communication that is underexplored in virtual and augmented reality,” he said.

Kim and student researchers have received honors for other projects, including the Best Demo Award at the IEEE World Haptics Conference 2025 and a Best Demo Honorable Mention at the IEEE Haptics Symposium 2024 for their work on thermal masking and thermal-tactile integration.

Thermal masking is a phenomenon that tricks the brain into feeling heat or cold in a specific spot on the body, even though the source of the heat or cold is generated from a different location. For example, researchers place a heat actuator — a device that produces heat — on a user’s arm and a tactile actuator, which produces vibrations, on the same arm several inches from the heat actuator. When researchers activate both actuators, the brain senses the heat at both locations.

The researchers presented a paper on the subject at the 2024 ACM CHI Conference on Human Factors in Computing Systems and created a video to demonstrate the technology.

“By delving into how tactile sensations can override thermal perceptions, we aim to create more immersive and responsive user interfaces,” Kim said. “Our work holds promise for revolutionizing applications in virtual reality, medical simulations and beyond to create experiences that feel almost tangible.”

Kim’s co-authors on the PropType article include Dr. Hyunjae Gil, a former postdoctoral researcher in computer science who now is an assistant professor at Daegu Gyeongbuk Institute of Science and Technology in South Korea; Iniyan Joseph BS’25; and Ashish Pratap, a computer science doctoral student. The work was supported by the South Korean government through a grant from the Institute of Information & Communications Technology Planning & Evaluation of South Korea.