Can reasoning AI grade reflective writing for teachers? A medical education case study offers a cautious answer

Generative AI is moving into many corners of medical education. One of the most tempting uses is also one of the most time-consuming parts of teaching: assessing student work. For short-answer questions with clear reference answers, large language models have already shown some ability to score responses. But reflective writing is different. It has no single correct answer. It asks students to draw on coursework, clinical exposure, personal experience, and professional values. Can AI still judge such work reliably?

A case study published in the Chinese Journal of Medical Education Research offers a cautious answer. Researchers from West China Second University Hospital, Sichuan University; the National Research Center for Educational Materials; and West China Medical Center examined how reasoning-capable large language models performed when assessing reflective writing assignments by medical students. They found that the models could generate relatively complete comments, but still showed clear limitations in scoring consistency, scoring stability, alignment with the grading rubric, and judgments related to academic integrity. The authors conclude that current reasoning-capable large language models are not yet effective tools for independently assessing medical students’ reflective writing. A more realistic path, they argue, is not substitution, but human–AI collaboration.

Reflective writing is difficult to standardize by nature. It does not simply test whether students have memorized course content. It asks whether they can think through medical professionalism, doctor–patient relationships, medical ethics, public health, and their own professional development. Compared with short-answer items that have reference answers, this kind of open-ended long-form assignment requires a reader to weigh originality, depth of reflection, connection to real experience, written expression, and academic norms. That makes it a revealing test case for AI-assisted assessment.

The study used reflective writing assignments from the 2017 fall semester of the Introduction to Clinical Medicine course at Sichuan University. This timing was important: the assignments and the original instructor assessment were completed years before publicly accessible large language models became available, so neither student writing nor teacher grading was influenced by AI tools.  A total of 293 reflective writing assignments were submitted in the course. The researchers sent these assignments to two representative domestic deep-reasoning large language models at the time, DeepSeek-R1 and Kimi K1.5 Long Thinking, for assessment. Five assignments were rejected by DeepSeek on the grounds of “potential violation of usage policies,” leaving 288 assignments for final analysis of scores, comments, and consistency.

In terms of overall scores, the median scores given by the models and the teachers were all 34. However, the score distributions were not the same. Teacher scores were the most concentrated, with an interquartile range of 1.00; Kimi had an interquartile range of 2.00; and DeepSeek had the widest variation, with an interquartile range of 3.00. Further analysis showed that similar central scores did not mean consistent assessment. The intraclass correlation coefficient among the three sets of scores was 0.126, and Kendall’s W was 0.012 (P=0.033), indicating poor agreement in both absolute scores and relative rankings. In pairwise comparisons, the Spearman correlation coefficients for DeepSeek versus teachers, Kimi versus teachers, and DeepSeek versus Kimi were 0.204, -0.030, and 0.136, respectively, all indicating weak correlations.

Scoring stability was also unsatisfactory. The researchers used the “regenerate” function of both models and asked them to score the same 288 assignments a second time. The results showed that the Spearman correlation coefficient between DeepSeek’s first and second scores was 0.358 (P<0.001), while the corresponding coefficient for Kimi was 0.354 (P<0.001). This indicates that, although the two rounds of scoring by the same model showed statistically significant correlations, the strength of those correlations remained limited. For course assessment, scores must not only appear reasonable, but also be stable, fair, and interpretable. In this respect, the current models still fall short of the requirements for direct application.

The study does not dismiss AI’s potential role. Compared with the instructor’s brief comments, the model-generated feedback was longer and more complete in structure. The median comment length was 84 Chinese characters for DeepSeek and 87 for Kimi, compared with 14 for the instructor. The models typically summarized the topic or writing approach, identified strengths and weaknesses, and offered suggestions for revision. Some comments were useful and specific, pointing out, for example, an incomplete understanding of medical professionalism, insufficient supporting evidence, or weak connection between course content and clinical observation.

Text analysis showed that the comments generated by the two models did not fully align with the focus areas of the scoring rubric. The models tended to focus more on content quality, topic selection, and depth of writing, while giving insufficient attention to academic conventions such as writing structure, assignment format, and citation standards. Kimi’s comments showed a relatively larger deviation from the scoring rubric, whereas DeepSeek was closer to the rubric but still showed gaps. In addition, the models’ judgments of plagiarism and assignment similarity were unreliable, suggesting that they cannot yet independently handle assessment tasks involving academic integrity.

The quality of suggestions also varied. Some were concrete, such as advising a student to connect the essay more closely to course concepts, relevant policies, or clinical observations. Others were vague, urging students to “offer more forward-looking views” or “further improve depth and organization.” Some comments appeared to set expectations beyond the level of first-year medical students, asking for more advanced theoretical analysis or more rigorous survey design than the assignment required. Others contained internal tension, praising an essay for having depth while also saying its arguments were insufficiently developed.

The models were especially unreliable when judging plagiarism or similarity. In some cases, they supplied links or article titles that could not be verified. In others, they appeared to treat poor citation practice as serious plagiarism. There were also cases in which the instructor had identified problems that the models missed. These findings matter because academic integrity judgments have consequences; they cannot be delegated to a model without careful human review.

The study’s central message is not that AI has no place in educational assessment. It is that generating feedback and making an evaluative judgment are different tasks. In medical education, reflective writing is not only a writing exercise. It is also a way for students to think through professional responsibility, ethical judgment, and the human side of medicine. Assessing such work requires more than fluent comments. It requires stable scoring, accurate feedback, and judgment that fits the course’s educational aims.

For that reason, the authors emphasize human–AI collaboration rather than replacement. Teachers may use AI to summarize assignment themes, draft preliminary feedback, or reduce some of the writing burden involved in commenting. But scores, revised feedback, plagiarism judgments, and value-laden educational decisions still require teacher review. When different models give sharply different scores for the same paper, or when their comments conflict, those cases should receive closer human attention.

The study also raises a deeper question for educators. If teachers use AI to design assignments, students use AI to complete them, and teachers then use AI to grade them, what happens to the educational purpose of the assignment itself? When AI is used to handle tasks seen as burdensome, teachers may also need to ask whether those tasks are truly meaningful for learning. For reflective writing, the point is not simply to produce a score. It is to help students develop a deeper understanding of medicine, patients, and their own professional growth.

The findings should be read within the limits of a case study. The sample came from one course in one semester. The instructor scores were the original scores given by one teacher in 2017, which limits their value as a definitive benchmark. The study also did not examine whether students would revise their work in response to AI-generated comments, or whether such feedback would actually support learning. As model capabilities and educational uses continue to evolve, these findings will need to be tested in more settings.

Still, the study offers a clear warning for medical education: reasoning ability does not automatically translate into reliable assessment ability. For open-ended, reflective, and value-rich learning tasks, AI may help in the assessment process, but teachers should remain at the center of evaluation. The more important question may not be whether AI can grade assignments for teachers, but what kinds of assignments deserve students’ serious effort, what kinds of feedback help them grow, and how human–AI collaboration can make assessment fairer and more meaningful.

Article information：

English Title: From substitution to human-AI collaboration: a case study on assessment of reflective writing assignments of medical students using reasoning-capable large language models

Journal: Chinese Journal of Medical Education Research

DOI: 10.3760/cma.j.cn116021-20250805-02199

Fulltext: http://doi.org/10.3760/cma.j.cn116021-20250805-02199

Institutions: West China Second University Hospital, Sichuan University; National Research Center for Educational Materials; West China Medical Center, Sichuan University

Chinese Journal of Medical Education Research is a monthly peer-reviewed journal sponsored by the Chinese Medical Association and hosted by Chongqing Medical University, under the supervision of the China Association for Science and Technology.
Launched in 2002, the journal publishes research and practice-oriented studies on medical education, with a focus on teaching reform, clinical teaching, curriculum development, residency training, graduate education, nursing education, educational technology, and international medical education. It also features themed columns and special issues on emerging topics and institutional innovations in medical education.
The journal is recognized as a Source Journal for Chinese Scientific and Technical Papers and Citations and is included in the Chinese Science and Technology Core Journals. It is indexed in Wanfang Data, Index Copernicus, the WHO Western Pacific Region Index Medicus, and Ulrich’s Periodicals Directory.

Journal page: http://yxjyts.alljournals.ac.cn/homeNav?lang=zh