AI Now Outperforms Lawyers on Legal Research Tasks

TL;DR:

  • AI tools now beat human lawyers on legal research across accuracy, authoritativeness and clarity metrics
  • Legal-specific AI scored 76-78% whilst ChatGPT achieved 74%, both above the 69% lawyer baseline
  • Study tested 200 legal research questions designed with input from major US law firms

A new benchmarking study by Vals AI has found that artificial intelligence tools now outperform human lawyers on legal research tasks, marking a significant milestone in legal technology development. The research tested both legal-specific and general-purpose AI tools against experienced lawyers across 200 complex legal research questions.

Context and Background

Vals AI evaluated three legal-specific tools—Alexi, Counsel Stack and Midpage—alongside ChatGPT, using questions developed with input from major US firms including Paul Weiss, McDermott, Reed Smith and Paul Hastings. Responses were scored across three criteria: accuracy, authoritativeness and clarity.

All AI tools exceeded the human lawyer baseline of 69% across all three metrics. The legal-specific tools achieved overall scores between 76-78%, whilst ChatGPT scored 74%. On accuracy alone, both ChatGPT and legal-specific AIs reached approximately 80%, compared with 71% for human lawyers.

Technical Reality: Legal-specific AI tools demonstrated notably stronger performance on authoritativeness, scoring six points higher than ChatGPT on average, primarily due to their access to proprietary legal databases.

Looking Forward

Vals AI calculated its human baseline by having “lawyers experienced in conducting legal research for client matters” from an unnamed US firm answer questions without generative AI assistance. However, the study’s methodology relied on single-prompt “zero-shot” questions without follow-up context, which Vals acknowledged “is not necessarily true to life” compared with typical legal research workflows.

Human lawyers continue to demonstrate advantages in deeper contextual understanding, nuanced judgement and multi-layered reasoning. Several prominent legal AI companies notably declined to participate in the study, potentially due to the simplified prompt methodology that may have favoured general-purpose tools over workflow-optimised legal platforms.

Share this article