image: Recently developed projection mapping techniques can achieve a very realistic appearance, even in dynamic scenes. However, they suffer from critical limitations, such as the need for markers on moving projection targets. In the above photographs, the markers can be clearly seen on the bunny. view more
Credit: Yoshihiro Watanabe from Tokyo Tech
An international business–academia collaboration has yielded a new type of projector that can project RGB and invisible infrared images simultaneously and independently at a high speed of almost 1,000 fps. With its high image alignment accuracy enabled by a custom optics engine, this projector will help closely integrate invisible sensing and visible display for a wide range of novel applications, such as dynamic projection mapping.
Recent progress in technologies involved in display and projection are unlocking new ways for us to perceive and interact with digital media. Modern projectors and related techniques, such as projection mapping, allow us to visually augment surfaces, opening up vast possibilities for amusement attractions, concerts, stages, digital signage, task assistance, and more. In many of these applications, the projection target is not just limited to a flat, white, and static screen; the projected image has to be controlled in real time to match the surface of complex moving targets and make their augmented appearance more convincing.
Techniques used for such purposes are referred to as dynamic projection mapping (DPM) and have gathered much attention over the past few years. They rely on visual sensing methods to detect the target surface and require high speed, both in sensing and projection frame rate, to properly merge the projected image and its target without noticeable misalignment. Unfortunately, conventional DPM has limited uses because it either needs markers attached to the target or has to be restricted to a single plane (Figure 1) to meet the high speed requirements.
On the other hand, projectors have also played important roles in visual sensing applications; projector–camera systems, which project a reference image onto the target surface and then capture it, can efficiently acquire much spatial information without using markers. Nevertheless, this type of sensing has not been used for DPM because no existing device has ever met the demands. That is, no device can handle display and sensing simultaneously with high performance and without them interfering with each other.
Fortunately, things might soon change on this front thanks to a new type of projector recently developed by an international business–academia collaboration team led by Associate Professor Yoshihiro Watanabe at Tokyo Institute of Technology (Tokyo Tech), Takeshi Yuasa at Tokyo Electron Device Limited, Uwe Lippmann at Fraunhofer Institute for Applied Optics and Precision Engineering, and Petra Aswendt at ViALUX GmbH. This projector can simultaneously project visible RGB and invisible infrared (IR) images at a high frame rate of 925 fps. “Using our projector, we can construct a system that can sense its target surface using invisible wavelengths based on a projector–camera configuration which is a long-established computer-vision technology,” explains Dr. Watanabe, “At the same time, the system can adaptively manipulate the images to display in visible wavelengths based on its sensing results.”
At the core of this projector (Figure 2) lie two DLP® (Digital Light Processing)[1] digital micromirror devices, which can handle 24-bit RBG and 8-bit IR images simultaneously and independently (Figure 3). The team also developed a novel optical engine to align the images coaxially with excellent precision while allowing for a compact configuration capable of high-power projection.
As a first application case, the team plans to introduce their novel projector for a new type of DPM system, which will be capable of manipulating the appearance of the entire scene based on the sensed shape in a markerless way. “Conventional DPM approaches are limited in that they can only augment the appearance of targets with markers,” states Dr. Watanabe, “The new type of mapping enabled by our projector will hopefully extend the fields of application of DPM.” Moreover, while only IR projection will be used for shape sensing in their first project, a projector–camera configuration using the entire system could be designed to acquire deeper spatial and physical information, thereby greatly enhancing projection mapping. The future of projection technologies sure does looks bright!
###
Acknowledgement
This work was supported by JST SICORP Grant Number JPMJSC1808, Japan and BMBF Grant Number 01DR18009A and 9B, Germany.
Term
[1] DLP® Digital micromirror devices: Device having an array of microscopic mirrors (1024x768 mirrors in this study). Each mirror switching can be controlled independently and used for the projection image control. Projection method using this device is called Digital Light Processing (DLP).
###
Related Links
Watanabe Laboratory, Tokyo Institute of Technology
http://www.vision.ict.e.titech.ac.jp/
Depth-aware dynamic projection mapping using High-speed RGB and IR Projectors
http://www.vision.ict.e.titech.ac.jp/projects/DepthAwareDPM/index.html
The future of the virtual and the real | Tokyo Tech Research Story
https://www.titech.ac.jp/english/public-relations/prospective-students/first-step/watanabe-lab
Tokyo Electron Device Limited
https://www.teldevice.co.jp/eng/
Fraunhofer Institute for Applied Optics and Precision Engineering
https://www.iof.fraunhofer.de/en.html
ViALUX GmbH
About Tokyo Institute of Technology
Tokyo Tech stands at the forefront of research and higher education as the leading university for science and technology in Japan. Tokyo Tech researchers excel in fields ranging from materials science to biology, computer science, and physics. Founded in 1881, Tokyo Tech hosts over 10,000 undergraduate and graduate students per year, who develop into scientific leaders and some of the most sought-after engineers in industry. Embodying the Japanese philosophy of “monotsukuri,” meaning “technical ingenuity and innovation,” the Tokyo Tech community strives to contribute to society through high-impact research.
https://www.titech.ac.jp/english/
About Japan Science and Technology Agency
JST is an organization that leads Japan’s science and technology (S&T) development as an innovation navigator. We aim to contribute to the lives of people and the achievement of a sustainable society by promoting S&T for the purpose of opening up opportunities in innovation. Since its foundation, JST’s many outstanding achievements accomplished in collaboration with the government, universities, the industrial sector and public have been earned global recognition.
Method of Research
Experimental study
Subject of Research
Not applicable
Article Title
High-Speed RGB+IR Projector based on Coaxial Optical Design with Two Digital Mirror Devices
Article Publication Date
1-Dec-2021