Your new AI-powered sales assistant just confidently emailed a prospect a detailed competitive comparison — complete with performance metrics and pricing. The problem: it invented the competitor’s product name and hallucinated every data point. This isn’t a rare glitch. It’s the default failure state of a poorly-architected AI system. Fixing it isn’t about better prompts. It’s about architecture.
Your LLM Is a Synthesizer, Not a Database
Hallucinations aren’t a bug you can patch with better phrasing. They’re an emergent property of how LLMs work: the model is a statistical synthesizer trained to produce plausible text, not a retrieval system that fetches verified facts. When you ask it to recall specific figures — revenue numbers, product specs, competitor pricing — it generates what sounds right. Sometimes it is. Often it isn’t.
The highest-impact architectural fix is Retrieval-Augmented Generation (RAG): stop asking the model to remember facts and start feeding it the facts at inference time.
Here’s what that looks like in practice. The naive prompt:
"Write a summary of our Q3 2024 earnings report."The model will generate something plausible. It will also invent numbers.
The RAG version:
CONTEXT:
[Insert full text of Q3 2024 earnings report here]
TASK:
Write a 3-paragraph executive summary of the earnings report above.
Base your summary only on the provided context. If the context does not
contain enough information to answer a question, say so explicitly.
Do not introduce any figures, names, or claims not present in the context.The structural difference matters: you’ve shifted the model’s job from recall to synthesis. Vectara’s HHEM leaderboard benchmarks show that grounding the model in provided context reduces critical factual hallucinations — incorrect numbers, invented entities — by over 75% compared to a vanilla LLM baseline on the same question without context. That’s a meaningful reduction in the specific error class that destroys trust in production systems.
RAG is not a drop-in solution. The engineering work is real. Chunking strategy alone involves genuine tradeoffs: fixed-size chunks are fast and predictable but split sentences mid-thought; semantic chunking preserves meaning but requires more compute and can produce variable-length chunks that complicate downstream processing. Your embedding model needs to be domain-fit — a general-purpose embedder will underperform on legal or medical text. RAG also has a data freshness problem: if your retrieval index isn’t updated when source documents change, you’re grounding the model in stale facts, which is arguably worse than no grounding because it looks authoritative.
RAG handles the factual grounding problem. It doesn’t control how the model behaves once it has the context.
System Prompts Are Code — Treat Them That Way
A system prompt is a behavioral contract with the model. It defines what the model is, what it can say, and what it must refuse. Most teams write one, ship it, and forget it. That’s how you end up with a customer service bot that starts hallucinating product details six weeks after a model update.
Here’s a concrete example for a customer service bot:
You are a support agent for Acme Inc. Your job is to help customers
resolve issues with their orders and accounts.
Rules you must follow:
1. Only answer questions related to Acme Inc. products and services.
2. Never invent product features, pricing, or policies. If you don't
know the answer, say: "I don't have that information — let me
connect you with a specialist."
3. Do not discuss competitor products.
4. If a customer asks about a refund, always confirm their order number
before proceeding.
If you cannot answer a question within these rules, respond with:
"That's outside what I can help with directly. I'll escalate this to
our team and someone will follow up within 24 hours."The fallback phrase matters. Giving the model a specific, pre-approved string to use when it hits a boundary is more reliable than asking it to improvise a polite refusal — improvised refusals drift.
System prompts require version control, A/B testing, and regression testing. A prompt that produces the right behavior on gpt-4o today may behave differently after a model update — OpenAI doesn’t guarantee behavioral consistency across versions. Keep your prompts in git. Run evals against a fixed test set before and after any model or prompt change.
System prompts constrain behavior. They don’t enforce output structure — and unstructured output breaks parsers.
Enforce Structure at the API Level
Asking an LLM to “respond in JSON” is a soft constraint. The model will usually comply, but “usually” is not a reliability guarantee you can build a production pipeline on. The model might prepend an explanation, add a trailing comment, or wrap the JSON in a markdown code block — any of these break a naive parser.
The prompt-only approach:
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "user", "content": "Extract the order number and status from this email. Respond in JSON."}
]
)
# Might work. Might not. Depends on the day.The better approach uses API-level enforcement:
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "user", "content": "Extract the order number and status from this email."}
],
response_format={"type": "json_object"}
)OpenAI’s response_format={"type": "json_object"} constrains the decoding process itself — it guides token generation to guarantee syntactically valid JSON, eliminating the conversational filler that breaks parsers. For more complex schemas, function-calling mode lets you define the exact shape of the output and the model is constrained to match it.
Syntactically valid JSON is not semantically correct JSON. The model might return {"order_number": null} when you needed a real value. That’s where schema validation comes in. In Python, use Pydantic:
from pydantic import BaseModel, validator
class OrderExtraction(BaseModel):
order_number: str
status: str
@validator('order_number')
def must_be_nonempty(cls, v):
if not v.strip():
raise ValueError('order_number cannot be empty')
return v
# Parse and validate
result = OrderExtraction.model_validate_json(response.choices[0].message.content)In TypeScript, Zod does the same job. API enforcement guarantees structure; schema validation catches semantic failures. Prompt instruction alone is a soft constraint — API enforcement is a hard one.
Verify the Output — With a Second LLM and a Deterministic Check
No single LLM output should be trusted blindly in a production system. Two verification layers catch different error classes.
LLM self-critique: After generating a summary or analysis, pass it to a second LLM call with a critique prompt:
ORIGINAL ARTICLE:
[article text]
GENERATED SUMMARY:
[model output]
Does the summary make any claims not explicitly supported by the article text?
List each unsupported claim. If the summary is fully supported, respond with "VERIFIED".This catches hallucinations detectable by reading comprehension — claims that contradict or extend beyond the source. Using a more capable model for the critique step (e.g., gpt-4o critiquing output from a cheaper model) improves reliability. But the critique LLM can also hallucinate, particularly for errors requiring external knowledge the model doesn’t have. A second LLM call also adds latency (typically 1–3 seconds) and doubles your token spend for that pipeline — the right tradeoff for a customer-facing financial summary, wrong for an internal autocomplete.
Deterministic external checks handle a different class of errors — specific, verifiable facts. If your model extracts a figure like “$5.2 million” from a document, you can check it programmatically:
import re
def extract_dollar_amount(text):
match = re.search(r'\$[\d,]+(?:\.\d+)?(?:\s?(?:million|billion))?', text)
return match.group(0) if match else None
extracted = extract_dollar_amount(model_output)
expected = project_database.get_budget(project_id)
if extracted != expected:
flag_for_human_review(model_output, extracted, expected)For structured facts against a known database, this check is near-100% reliable. For fuzzier claims — strategic direction, qualitative assessments — you’re in probabilistic territory. A search API against an internal knowledge base can return supporting documents, but entity extraction and semantic matching produce confidence scores, not binary results. High-stakes outputs in this category need human-in-the-loop review built explicitly into the workflow.
This Is Risk Mitigation, Not a Cure
These four layers work as a system. RAG grounds the model in verified facts. System prompts constrain its behavior. API-level enforcement makes output parseable. Verification catches what slips through. None of them is sufficient alone — each layer handles a different failure mode, and each has its own failure modes.
The operational question most teams skip: how do you know when your guardrails are degrading? Log hallucination incidents when users or downstream systems flag bad output. Monitor your critique-step rejection rate — if it spikes, something changed upstream (model update, data quality, prompt drift). Track structured output parse failures as a leading indicator of prompt or model issues.
The engineers who ship reliable AI systems treat the LLM as a managed component — something with known failure modes, observable behavior, and costs that have to be justified against business stakes. The ones who ship unreliable systems trusted the demo.