Audits help change a chatbot’s bad behavior
University of Texas at AustinPeer-Reviewed Publication
In new research, Yan Leng, assistant professor of information, risk, and operations management at the McCombs School of Business at The University of Texas at Austin, has devised a sort of personality test — more precisely, a behavioral audit — for large language models (LLMs), the technology that drives products such as ChatGPT.
By understanding an LLM’s existing tendencies, an organization can decide whether an available model already fits its values and usage scenarios. If not, it might need to fine-tune a model before putting it to use.
Leng compares her framework to trying to understand a person through their actions and thought processes. “For a human, we would have our values, and our values would dictate how we make decisions, so we try to have that for LLMs as well,” she says.
- Journal
- Information Systems Research