Why lawyers must read everything, and why AI checking AI is a dangerous illusion.
Automation bias is the cognitive tendency to favor results generated by automated systems over those from non-automated sources, even when the automated system is clearly wrong.
In studies, radiologists' accuracy dropped from nearly 80% to under 20% when AI suggested an incorrect finding. Not because the radiologists were bad at their jobs. Because automation bias made them trust the machine over their own eyes.
What Happened at Sullivan & Cromwell
By now, you've heard the story. A junior associate at Sullivan & Cromwell used AI to prepare a filing. The AI hallucinated cases. The associate didn't catch them. The firm had to withdraw the filing.
What makes this different from earlier "ChatGPT filed a fake brief" incidents?
The firm had AI checks in place.
They ran the AI output through AI detection tools. Those tools said the citations looked real. Because AI detection tools look for patterns, and hallucinations still look like patterns.
This is automation bias in action:
- The associate trusted the AI's output because it sounded confident
- The firm trusted the detection tool because it was "automated verification"
- Nobody actually opened Westlaw or CourtListener to check
Automation bias made everyone delegate judgment to machines that have none.
Why AI Checking AI Is Like Asking a Liar for a Reference
Here's what AI detection tools actually do:
- Scan for statistical patterns
- Flag text that looks "too predictable"
- Compare against training data
Here's what they don't do:
- Open the actual court record
- Know that "F.4th" stops at volume 200
- Verify that a citation exists in reality
An AI detector sees 987 F.4th 456 and thinks: "Looks like a citation. Statistical pattern matches. Probably fine."
Automation bias makes the lawyer think: "The detector said it's fine. I can trust that."
But the detector never checked the source. It just guessed based on patterns. And it guessed wrong.
AI checking AI is automation bias squared. You're trusting one machine to catch another machine's errors, when neither machine has ever opened a real case.
The Construction Site Analogy
Think of your legal drafting process like a construction site.
You have architects (partners), foremen (senior associates), and workers (juniors and AI tools). Everyone is trying to build a solid brief.
AI detection is like someone standing at the site entrance, looking at bricks, and saying: "These bricks look like bricks. The shape is right. The color is right. Probably fine."
They never test the bricks. They never check if the brick actually came from a factory. They just look at patterns.
Automation bias makes everyone accept this: "The inspector looked at the bricks. They said they're probably fine. Let's build the wall."
We have built a free tool called GhostCite, available at CourtRecordChecker.com. This tool is different.
GhostCite is the inspector who actually calls the brick factory and asks: "Did you make this brick? What's the serial number? Show me the record."
CourtListener is the factory record. GhostCite asks the source directly, every single time. No pattern matching. No guessing. Just a yes/no answer from the actual court record.
The Role of Automation Bias in Legal Briefing
Automation bias shows up in three dangerous ways when lawyers use AI:
| Bias Manifestation | What It Looks Like | The Risk |
|---|---|---|
| Omission bias | The AI didn't flag an error, so the lawyer assumes no error exists | Missing hallucinations because the detector stayed silent |
| Commission bias | The AI suggested something wrong, but the lawyer assumes it's right because the AI sounds confident | Filing fake cases because the AI "cited" them |
| Verification delegation | The lawyer assumes someone else (or some other tool) will catch errors | Nobody actually opens the cases to read them |
The Sullivan & Cromwell incident had all three.
The associate delegated verification to the AI. The firm delegated verification to detection tools. When the detection tools said nothing, automation bias made everyone assume nothing was wrong.
What GhostCite Actually Does
When you paste a brief into GhostCite, here's what happens:
- Extraction, It finds every citation, quote, and link in your document
- Direct Query, It sends each citation to CourtListener's API and asks: "Does this exist in the actual court record?"
- Verification, It returns a simple answer: Yes (verified) or No (not found)
No AI generation. No pattern matching. No hallucinations.
Just a direct check against primary law.
GhostCite doesn't "detect" patterns. It doesn't "guess" if a citation looks real. It asks the source directly and reports the answer.
This breaks automation bias because there's no opaque "trust me" layer. The tool either found the case in the court record or it didn't. You can verify GhostCite's work by opening the same case yourself.
The Bottom Line
Automation bias is not a character flaw. It's a cognitive feature of how humans process trust. We evolved to trust machines more than people because machines don't have hidden motives. For a broader look at the professional liability that accumulates when AI replaces human judgment, see The AI Liability Trap.
But AI isn't a machine in that sense. AI makes confident mistakes. And automation bias makes us trust those mistakes.
The only fix is verification from the source.
AI can draft. Use it.
AI can summarize. Use it.
AI can speed up research. Use it.
But AI cannot verify itself. That's still your job. GhostCite just makes it faster. For a complete 5-minute pre-filing verification workflow, see our brief sanity check guide.
GhostCite is free. No AI. Just primary law verification through CourtListener.
Try it at rule26ai.com or courtrecordchecker.com.