Robots learn to 'see' and control themselves with simple 2D cameras, paving the way for greater autonomy

Imagine a robot that, much like a person learning to use their body, can look at itself and understand its own shape and movement. Researchers from Sun Yat-sen University have turned this vision into reality by developing a groundbreaking framework that enables robots to build a dynamic, 3D model of themselves using only a stream of standard 2D RGB images—the kind produced by any ordinary smartphone camera.

This advance in "robot self-modeling" is a significant leap toward creating more adaptive, resilient, and cost-effective autonomous machines. The work, published in a leading robotics journal, could accelerate the deployment of robots in unpredictable environments, from busy households to disaster zones, where the ability to self-correct after damage is critical.

"For a robot to operate effectively in the real world, it needs an accurate internal understanding of its own body—a 'digital twin'," explained Professor Ning Tan, the lead researcher. "Traditional models are rigid and break down the moment the robot gets damaged or wears out. Our goal was to create a system where the robot can autonomously learn and update this model, ensuring it always has an accurate sense of self."

Overcoming the Cost and Complexity Hurdle

While the concept of robot self-modeling isn't new, previous methods often relied on expensive, high-precision equipment like depth cameras and LiDAR. Other vision-based approaches struggled with poor quality and couldn't grasp the articulated structure of a robot—they saw it as a single, blurry object rather than a collection of rigid links and joints.

The team's innovative solution was to teach the robot to see itself in parts. Using an advanced AI technique called a part-based Neural Radiance Field (NeRF), their framework automatically identifies and segments a robot's body into individual rigid components, like an arm segment or a gripper. Each part is modeled by its own small neural network, and the system learns how these parts move in relation to the robot's joint commands.

How It Works: Learning by "Looking"

The process is elegantly self-supervised. The robot moves its joints randomly, observing itself with a simple camera. Its internal AI then generates its own "mental image" of what it should look like in that pose. By continuously comparing this self-generated image to the actual camera feed, the AI refines its internal model and part segmentation—all without any human labels or expensive sensors.

The results are striking. In tests with a complex 7-degree-of-freedom robotic arm, the system generated high-fidelity 3D models that rivaled the quality of those built with costly depth-sensing hardware and significantly outperformed other image-only methods.

From Model to Action: Proving Practical Utility

The true test of a self-model is whether a robot can use it to perform tasks. The researchers demonstrated that their system could do just that. Using only its self-learned model, a robot was able to successfully calculate the joint movements needed to position its end-effector at a target point in space—a fundamental challenge in robotics known as inverse kinematics.

The authors are hopeful and excited about the future of their findings. "Our approach can be seen as a preliminary step for robot self-awareness," says the researcher. “We hope our research paves the way for future research of development of general-purpose robot.”

About Sun Yat-sen University

Located in the Guangdong-Hong Kong-Macao Greater Bay Area, SYSU is committed to serving the social development of the region and the nation, and to playing an active role in the vision of building a community with a shared future for mankind. Today, SYSU has become a cradle for talents, a center for knowledge and technology innovation, a pllar to serve society and a base for cultural inheritance and innovation. SYSU is being built into a university that is first considered when the students at home and abroad are choosing a higher education institution, first considered when the country is planning to promote major strategies, and first considered when global academics are discussing cutting-edge issues. SYSU is now standing on a new starting point and strving to become a world-class university with Chinese characteristics.

Website:  https://www.sysu.edu.cn/sysuen/

About Professor Ning Tan from Sun Yat-sen University,China

Ning Tan is an Associate Professor and Ph.D. Supervisor at the School of Computer Science and Engineering, Sun Yat-sen University. He also serves as a Youth Editorial Board Member for the journal SmartBot. His research focuses on the design, optimization, planning, and control of various robotics and embodied intelligence systems. Professor Tan is a recipient of the Guangdong Provincial Outstanding Youth Fund and has been listed in the World's Top 2% Scientists. He has published over 100 academic works, including more than 30 papers in prestigious journals such as The International Journal of Robotics Research (IJRR), IEEE Transactions on Robotics (T-RO), Automatica and IEEE Transactions.