AI Agents Hallucinate Too — Design For That Risk

Here is a statistic that should concern every business deploying AI agents: Stanford researchers tested legal AI tools marketed specifically as hallucination-free, grounded in real case law with citations. The tools still produced incorrect answers in roughly one out of every six to one in three queries. That gap between marketing claim and measured reality matters for any industry, but it matters most once AI stops just answering questions and starts taking action on its own — which is exactly what is happening as businesses move from chatbots to autonomous agents. ZTS Infotech caught this risk firsthand during a client build, and it changed how every agent the company ships gets designed. 

The Stanford Study Business Leaders Should Know About 

Stanford's RegLab ran the first preregistered empirical evaluation of AI-driven legal research tools, testing products from major legal vendors that market themselves as hallucination-free. The results: Lexis+ AI produced incorrect or unsupported answers roughly 17% of the time, and Westlaw's AI-Assisted Research tool did so in about 33% of queries — despite both being built specifically to ground responses in verified case law with citations. 

Context matters here. Those error rates are a real improvement over general-purpose tools like ChatGPT, which researchers found could be wrong as often as 82% of the time on legal questions. The point isn't that specialized, grounded AI tools are worthless — it's that "hallucination-free" was a marketing claim the data didn't support, even for tools built with exactly that goal in mind. 

Why AI Models Hallucinate: Plausibility Over Truth

The technical explanation is straightforward. AI language models are trained to prioritize fluency and pattern completion, not factual verification. A model is optimized to sound confident and coherent, not to be correct. Researchers studying this behavior have a useful phrase for it: a plausibility engine, not a truth engine. That distinction is easy to nod along to in the abstract and easy to forget in practice, which is exactly the problem. 
 

The Real Danger: Chatbot vs. Agent 

A chatbot that hallucinates embarrasses you. An agent that hallucinates acts on it. It sends the email. It updates the record. It approves the transaction. That is the entire risk calculus shift businesses need to internalize as AI tools move from answering questions to executing tasks with real operational and financial consequences. 

A Real Example: The Invoice That Almost Went to the Wrong Vendor 

Early in a client onboarding automation build, ZTS Infotech tested an agent handling vendor invoice matching. Given an ambiguous purchase order reference, the agent did not flag the ambiguity. It confidently matched the invoice to the wrong vendor account and would have processed payment. The error was caught in testing only because the system required a source citation for every match before approval — a design decision that is now standard on every agent the company builds. 

Three Design Principles Now Standard on Every Agent 

The fix isn't avoiding agents. It's designing them around the reality that confident and correct are not the same thing. ZTS Infotech now applies three principles to every agent it ships: 

• Ground every agent in verified data sources, not open-ended reasoning. An agent should retrieve from a defined, checkable source rather than generate an answer from general pattern-matching. 
• Give the agent specific tools for verification rather than asking it to simply "know" the answer. Verification should be a step the agent performs, not an assumption baked into its confidence. 
• Strictly scope what the agent can act on independently versus what requires human sign-off, based on how reversible and how costly a mistake would be.

Expert Perspective 

The Stanford study's real significance isn't that AI legal tools are unreliable — it's that vendor marketing around reliability is currently running ahead of what the data supports, in a domain where vendors had every incentive to get it right first. That gap should inform how any business evaluates vendor claims about agentic AI more broadly, not just in legal research.

As agentic AI adoption accelerates, expect procurement and governance standards to catch up to this risk faster than most vendors are currently prepared for. A citation trail a human can check in seconds, and a scoping policy tied to reversibility and cost, are not exotic requirements — they are the baseline any business should expect to demonstrate, whether to a client, an auditor, or a regulator asking how an autonomous system's decisions can be verified after the fact. 

Businesses that build these guardrails in from day one will have both a risk-management advantage and, increasingly, a compliance advantage over those that discover the gap only after an agent ships something it shouldn't have. 
 

Key Takeaways

• A Stanford RegLab study found legal AI tools marketed as hallucination-free still produced incorrect answers 17% (Lexis+ AI) to 33% (Westlaw AI-Assisted Research) of the time. 
• AI models are trained to optimize fluency and pattern completion, not factual accuracy — a "plausibility engine, not a truth engine." 
• A hallucinating chatbot produces a wrong answer; a hallucinating agent takes a wrong action — sending emails, updating records, approving transactions. 
• ZTS Infotech caught an agent confidently matching an invoice to the wrong vendor account during testing, prevented only by a required source-citation check before approval. 
• Three design principles now standard at ZTS Infotech: ground agents in verified sources, give them explicit verification tools, and scope autonomy by reversibility and cost of error. 
• Vendor claims of "hallucination-free" AI should be treated skeptically and verified against independent data, not accepted as marketing fact. 
• No agent ZTS Infotech deploys acts on a high-stakes decision without a citation trail a human can verify in seconds. 

Conclusion 

Confident and correct are not the same thing, and the gap between them costs nothing when a chatbot gets it wrong in a conversation — and potentially a great deal when an autonomous agent acts on it. The businesses that design for that difference now, with grounding, verification tools, and reversibility-based scoping, will be the ones that catch the mistake in testing rather than finding out about it after something ships.