image: Figure | Schematic diagram of the SSSR-FPP system. The SSSR-FPP system includes two high-speed CMOS scientific cameras and a customized digital-light-processing projection system. With the increase in a camera’s maximum frame rate, the exposure time is too limited to capture a sufficiently bright image, resulting in images with poor SNR and evident read noise superimposed.
Credit: by Bowen Wang, Wenwu Chen et al.
As one of the most widely adopted 3D sensing techniques, fringe projection profilometry (FPP) reconstructs the depth information of a scene from stereo images taken with sequential structured illuminations. However, for measuring dynamic or even transient events, the imaging speed of FPP is capped by two fundamental factors: (1) hardware: the speed of the projector and camera; and (2) software (algorithm): the number of patterns required per 3D reconstruction.
With the advancements in artificial intelligence, particularly deep learning (DL), optical metrology — a field dedicated to the precise measurement and characterization based on optical signals, has experienced a paradigm shift. In particular, it has been demonstrated that properly trained deep neural networks (DNNs) can retrieve phase and unambiguous 3D coordinates of complex objects from only a single fringe pattern, effectively pushing the 3D imaging speed to align with the camera’s native frame rate for 2D image acquisition. Nevertheless, as mentioned earlier, the high-speed cameras currently available can only achieve an imaging frame rate of a few kHz with a decent resolution. Consequently, the highest 3D imaging frame rate of reported FPP techniques only reaches ∼ 20 kHz, which still falls short of the requirements for capturing ultra-fast phenomena.
In a new paper published in Light: Science & Applications, a research group, led by Professors Qian Chen and Chao Zuo from Smart Computational Imaging Laboratory (SCILab), Nanjing University of Science and Technology, China, has developed a novel learning-based ultrafast 3D imaging technique, termed single-shot super-resolved fringe projection profilometry (SSSR-FPP), which enables ultrafast 3D imaging at 100,000 Hz. By leveraging the substantial speed gained from reducing the imaging window of conventional highspeed cameras, SSSR-FPP attempts to retrieve a high SNR and high-quality 3D image from a pair of single low-SNR, low-resolution fringe patterns. This breakthrough offers a new tool for studying ultra-fast dynamic processes and could revolutionize fields such as mechanics, physics, and biology by enabling the capture of events that were previously too fast to observe.
The proposed method reveals the potential of combining deep learning with FPP for ultrafast, super-resolved, ambiguity-free 3D imaging, pushing the 3D imaging frame rate into the 100 kHz regime. These scientists summarize the operational principle of their technique:
“SSSR-FPP employs two structurally similar but functionally different convolutional neural networks (CNNs), CNN1 and CNN2, which work in concert to achieve both super-resolved phase retrieval and phase unwrapping. It's worth noting that SSSR-FPP uses only a pair of low-resolution, pixelated fringe images and a trained deep learning network to recover high-resolution 3D coordinates in a single shot. This method significantly enhances the frame rate by reducing the imaging window of traditional high-speed cameras and utilizing deep learning to "regenerate" lost spatial resolution. With this approach, the 3D imaging frame rate has been increased by an order of magnitude, pushing it to 100,000 fps.”
“Moreover, owing to its single-shot nature, the SSSR-FPP method fundamentally overcomes the phase-shifting errors and associated artifacts induced by object motion.
To demonstrate the high spatio-temporal resolution of SSSR-FPP, we present 3D videography of several transient scenes, including rotating turbofan blades, exploding building blocks, and the reciprocating motion of a steam engine, etc., which were previously challenging or even impossible to capture with conventional methods. Experimental results establish SSSR-FPP as a significant step forward in the field of 3D optical sensing, offering new insights into a broad spectrum of dynamic processes across various scientific disciplines.” they added.
“By simply utilizing imaging sensors with a higher frame rate in conjunction with a high-power light source and large-aperture imaging optics could, in principle, further push 3D imaging frame rates up to the million-frame-per-second regime. Experimental results suggest that SSSR-FPP is expected to offer new insights for studying a multitude of ultra-fast dynamic processes, advancing our knowledge across various scientific disciplines.” the scientists forecast.
Journal
Light Science & Applications
Article Title
Single-shot super-resolved fringe projection profilometry (SSSR-FPP): 100,000 frames-per-second 3D imaging with deep learning