Clinical co-pilots: how artificial intelligence can impact and assist clinicians

Advances in foundation models (FM) have opened the gates to an advanced use of artificial intelligence for healthcare. FMs are an AI system with a great ability to perform accurately on new sets of data or tasks, instead of just memorizing the material it was trained with. Vision-language foundation models (VLFM) are particularly of interest to clinicians as a means to aid in report generation and question answering, with vision-language agents (VLAs) being representative of the next frontier for AI acting in a healthcare space.

 

Researchers published their findings in Computational Visual Media in May 2026.

 

VLAs take “artificial intelligence” to the next level with their generizable capabilities and integration with both visual and language models, allowing for the generation of accurate information based on encountering entirely new data. This ability is what would allow for the alteration of the workflow in healthcare places, as these agents would be able to understand and apply tool use, planning and interactions with patients’ data. Another key aspect of VLAs is the ability to understand and utilize data in multiple forms, such as pictures, text, and audio.

 

“VLA systems are poised to evolve from passive analytical tools into proactive, interactive clinical co-pilots,” said Younhyun Jung, author and researcher.

 

Vision-language agents can thank the architecture of vision-language foundation models for its adaptability across multiple modes of information and the generation of intelligent reports. The specific capabilities of VLAs are not only report generation and Q&As to get preliminary information, but also the ability to generate images and explainable representations for patients and clinicians. Intraoperative assistance and abnormality detection and segmentation are also tasks VLAs are capable of, which respectively consist of in-operation guidance (such as analysis of real-time video streams), plus having the capacity to differentiate common and rare pathologies without being explicitly trained on either.

 

Humans are still a large part of this process, responsible for confirming diagnoses, treatment planning and monitoring, and engaging with patients. The goal with VLAs is not to make human-patient interactions obsolete, but rather to enhance the experience for both sides by streamlining the process from initial diagnosis through treatment and beyond.

 

The success is dependent on its ability to learn dynamically, the scalability of an intelligence system like this, and the continued ability to work harmoniously with confidential patient data and the human healthcare workers. The step from simple data analysis to an active, autonomous agent capable of complex multi-step processes is a huge one, and isn’t without its issues and the need for necessary regulations.

 

There are quite a few barriers that would need to be broken for VLFMs to take off and reach their full potential. These barriers include the accessibility to high-quality multimodal datasets, the need for trustworthy outputs, and the need to create infrastructure that can support the load of VLFMs. Additionally, the operational scalability would need to be evaluated along with the economic feasibility.

 

Ethics are also a large potential issue, which would need to be carefully and diligently addressed to ensure responsibility and consideration when making use of AI agents. With ethics come policies and other regulatory measures to ensure the safety of patients and their data, together with the autonomous nature of the vision-language agents.  

 

Aside from the logistical barriers, there will also be the barrier of explaining the use of VLAs to patients. No doubt there will be skeptics and those opposed to the use of the degree of AI integration in healthcare. These barriers are nothing to take lightly, though they are surmountable with time, patience and continuous efforts to make VLAs a safe and helpful clinical copilot.  

 

Abdulrhman Hussein Al-Jebrni and Bin Sheng of the School of Computer Science at Shanghai Jiao Tong University, with Bin Sheng also of the MoE Key Lab of Artificial Intelligence at Shanghai Jiao Tong University, Zehua Jiang, Yiming Qin, Tien Yin Wong and Dian Zeng of the Beijing Visual Science and Translational Eye Research Institute at Tsinghua University, with Tien Yin Wong also of the Beijing Key Laboratory of Intelligence Diagnostic Techonology and Devices for Major Blinding Eye Diseases at Tsinghua University and Singapore Eye Research Institute at Singapore National Eye Center, Yue Qiu of the Institute for Hospital Management at Tsinghua University, Nan Jiang of the School of Healthcare Management at Tsinghua University and the Bloomberg School of Public Health at John Hopkins University, Guanqiao Li of the Vanke School of Public Health at Tsinghua University, Yingfeng Zheng of the State Key Laboratory of Ophthamology at Sun Yat-sen University and Suhyeon Kim and Younhyun Jung of the School of Computing at Gachon University contributed to this research.

 

The National Natural Science Foundation of China and the National Research Foundation of Korea.

 

DOI Link:

https://doi.org/10.26599/CVM.2026.9450530

---

---