TL;DR

Despite fears that AI would devastate legal jobs, new research shows LLMs are far from replacing lawyers. Professional benchmarks reveal the best-performing models score only 37% on complex legal problems. Meanwhile, law graduate employment reached record highs in 2024 at 93.4%.

The Reality Behind the Hype

When GPT-4 passed the bar exam in 2023, predictions of mass legal job losses followed swiftly—one industry report suggested 44% of legal work could be automated. But practising lawyers paint a different picture.

Junior associate Lucas Hale uses AI for routine tasks like document review but finds ChatGPT “spewing hallucinations” on complex legal questions. “In the case where we have a very narrow question or a question of first impression for the court, that’s the kind of thinking that the tool can’t do,” he explains.

Senior associate Allison Douglis puts it bluntly: “Right now, I would much rather work with a junior associate than an AI tool.”

What Benchmarks Actually Show

Passing a standardised test differs markedly from practising law. The Professional Reasoning Benchmark from ScaleAI found the best-performing model met just over a third of possible points on the most difficult legal problems. Models frequently made inaccurate judgments, and even correct conclusions came through “incomplete or opaque reasoning processes.”

The AI Productivity Index found similar limitations, with the best model scoring 77.9% on legal tasks—potentially useful in some industries, but problematic in fields where errors carry significant consequences.

Why Law Resists Automation

Legal reasoning may be fundamentally challenging for current AI architectures. Unlike coding or mathematics, law deals with ambiguous real-world problems that often have no right answer. Much legal work isn’t recorded in training-friendly formats, and documents span hundreds of pages across complex hierarchies of statutes, regulations, and case law.

Stanford Law professor Julian Nyarko suggests LLMs may lack the mental models humans use for complex reasoning: “The reasoning models still don’t fully reason about problems like we humans do.”

Looking Forward

Law graduate employment actually reached 93.4% in 2024—the highest rate on record. Law firms aren’t reducing headcount, though they acknowledge AI is reshaping grunt work.

For professional services firms, the lesson is nuanced: AI offers genuine productivity gains for routine tasks, but complex judgment work remains firmly human territory. The question now is how firms will train junior staff when repetitive work diminishes.