Cambridge-led study finds AI essay grading matches human marks only half the time
TL;DR:
- Researchers at Cambridge’s Institute for Technology and Humanity, working with Nottingham and Manchester Metropolitan, tested the latest Claude, ChatGPT and Gemini models on 761 undergraduate psychology essays and found AI grading matched human classifications only 35–65% of the time, depending on institution.
- All three models showed a “central tendency bias” — clustering marks in middle bands, undervaluing essays awarded top marks by human examiners and over-rewarding weaker submissions; marks across the three models were closer to each other than to human assessor marks.
- The models were “oversensitive to linguistic features”, rewarding denser sentence structures and complex vocabulary regardless of argumentation quality — a failure mode with direct implications for any UK institution considering AI-assisted assessment at scale.
The study is the most rigorous UK academic test of generative-AI essay grading to date and lands as financial pressures push universities to consider AI assessment as a workload-reduction tool, a temptation Dr Deborah Talmi explicitly warns against.
Context and Background
The institutional pattern in the results is telling. Cambridge essays — written under invigilated exam conditions with the narrowest grade range — produced the highest AI accuracy at 63%. Manchester Metropolitan coursework, with the widest grade range, produced the lowest at 35%. That is not a flattering result for any AI vendor: the systems performed better on the narrowest grade distribution and worst where they would be most useful in practice.
Co-author Dr Alexandru Marcoci’s framing — “Human assessors judge each essay on its own argumentative and conceptual merits, while AI marks are based on statistical predictions” — captures the structural mismatch. Statistical prediction trained on aggregate human marks will gravitate to the centre of the distribution and miss both the outliers that mark genuinely exceptional work and the subtle failures that distinguish a 2:i from a third. This is a familiar pattern in earlier AI-assessment research (the Education Endowment Foundation flagged comparable bias in 2024 marking pilots), but the Cambridge study is the first UK academic publication to put concrete cross-university numbers on it.
The student-attitudes finding — that some participants said they would feel “cheated” if their work was assessed primarily by AI — is the part most likely to land politically. UK universities considering AI marking deployments will need to address that consent question as well as the accuracy question, particularly in fee-paying programmes where the value proposition includes human academic engagement. The Lords Library briefing on AI’s societal impact (covered separately by Resultsense today) cites NEU data showing two-thirds of secondary teachers think pupils’ critical thinking has declined under AI usage.
Looking Forward
The Cambridge team’s “a human should always determine the final mark” conclusion will likely become the default position for UK university AI assessment policies through 2026, with AI restricted to first-pass screening or feedback drafting rather than autonomous grading. The Office for Students has indicated it will revisit AI assessment guidance later this year; expect a tightening rather than relaxation of human-in-the-loop expectations. Vendors selling AI-assisted marking platforms to UK higher education will need to address the central-tendency-bias finding specifically.