Study finds single AI evaluations of strategic decisions are biased, but “AI crowds” can match expert human judgment

Executives increasingly turn to generative AI to help evaluate strategic decisions, from ranking business models to weighing competing options — but new research published in Strategic Management Journal cautions that no single AI-generated evaluation should be trusted at face value. The study finds that individual generative AI evaluations of strategic decisions are inconsistent and often biased, simply by the order in which two options are presented — but aggregating many evaluations across different prompts, roles, and models produces judgments that closely resemble those of human experts.

The research team of Anil R. Doshi of University College London, J. Jason Bell of the University of Oxford, and Emil Mirzayev and Bart Vanneste, both of University College London, tested large language models by having them rank 60 business models through pairwise comparisons, repeated across multiple models, ten assumed roles — including journalist, strategy professor, and industry expert — and two prompting strategies. No human expert panel evaluates the same option set that many ways, giving the researchers a rare way to measure where AI’s judgment holds up, and where it doesn’t.

Reversing the order in which two options were presented was often enough to change which one a model preferred, with individual-model consistency ranging from just below 30% to about 81% across the study. “We find that individual generative AI evaluations are inconsistent and biased,” the authors write. Once evaluations were pooled across different models, roles, and prompts, however, agreement with expert human judgment increased substantially. “The wisdom of the crowd depends on two mechanisms: the crowd’s diversity and scale,” the authors note — a diversity effect from combining different models, roles, and prompts, and a scaling effect from aggregating more evaluations of each kind.

The findings carry a practical lesson for managers: good judgment comes from a deliberate process, not a single AI-generated answer. Before acting on any one AI-generated recommendation, the researchers suggest aggregating across multiple prompts, roles, and models — then applying two tools AI cannot supply on its own: design thinking, to frame what the organization or customer actually needs, and critical thinking, to test why the answer holds up under scrutiny.

To read the full context of the study and its methods, access the full paper available in Strategic Management Journal.

About the Strategic Management Society

The Strategic Management Society (SMS) is the leading global member organization fostering and supporting rigorous and practice-engaged strategic management research. SMS enjoys the support of 3,000 members, representing more than 1,100 institutions and companies in more than 70 countries. SMS publishes three leading academic journals in partnership with Wiley: Strategic Management Journal, Strategic Entrepreneurship Journal, and Global Strategy Journal. These journals publish top-quality work applicable to researchers and practitioners with complementary access for all SMS Members. The SMS Explorer offers the latest insights and takeaways from the SMS Journals for business practitioners, consultants, and academics.

Click here to subscribe to the monthly SMS Explorer newsletter.

Click here to learn more about the programs and opportunities SMS has to offer.