An agent demo is the easiest impressive thing in software right now. Wire a model to a few tools, give it a system prompt, run it against a happy-path task, and it looks like magic. We've built plenty of these demos. We've also been the team that inherits them six months later, when the magic has been in front of real users and the ticket queue tells a different story.
The gap between those two states isn't model quality. It's engineering. Here's what actually closes it.
The demo lies to you by construction
A demo is evaluated on the runs you watched. Production is evaluated on the runs you didn't. Those are different distributions, and the second one includes the user who pastes in a 40-page PDF, the API that starts returning 503s at 2 a.m., the ambiguous request that has two defensible interpretations, and the customer actively trying to jailbreak your support bot into quoting a refund policy you don't have.
Nothing about a successful demo predicts behavior on that distribution. This is the single hardest thing to communicate to stakeholders: the demo is evidence the system can work, not evidence that it does. Getting from can to does is most of the project, and it should be scoped and billed that way.
Evals are the test suite. Write them first.
We don't start agent work by prompting anymore. We start by collecting or writing an eval set: fifty to a few hundred realistic tasks with a programmatic definition of success. For a scheduling agent, that means "the correct calendar event exists afterward," checked against the calendar — not "the transcript looks reasonable," checked by vibes.
Three things we've learned the hard way:
- Grade outcomes, not transcripts. An agent can take a bizarre path to a correct result, and a beautiful-looking path to a wrong one. Assert on the state of the world after the run.
- Use an LLM judge where you must, rubrics where you can. Judge models are fine for tone and format. For anything factual or stateful, write a deterministic check. Judges drift, and a judge grading its own model family has known biases.
- Track the distribution, not the average. A change that moves pass rates from 82% to 86% while introducing a new catastrophic failure mode is not an improvement. Look at the failures individually, every time.
Once evals exist, prompt changes stop being arguments and start being measurements. That alone changes the tenor of a project — "I feel like it got worse" becomes a diff against a number.
Tool calls fail in ways chat never did
A chat model that's wrong produces a wrong sentence. An agent that's wrong produces a wrong action, and the failure modes are specific enough to enumerate. The ones we design against on every engagement:
- Wrong tool, plausible arguments. The model calls
delete_recordwhen it meantarchive_record, with a perfectly valid ID. Schema validation passes. Nothing catches this except authorization design. - Hallucinated arguments. IDs, dates, and email addresses that look right and don't exist. Validate arguments against reality, not just against the schema.
- Retry loops. A tool errors, the model retries with the same input, forever. Cap iterations, make error messages instructive ("this ID does not exist; search first"), and fail closed.
- Partial completion. Step three of five fails and the agent reports overall success anyway. Multi-step work needs transactional thinking — what's the rollback story when the sequence dies in the middle?
The structural answer is the same one we apply to human operators: least privilege and blast-radius control. Read tools are cheap to grant. Write tools get scoped narrowly. Anything irreversible — sending money, deleting data, emailing a customer — goes through a confirmation gate or a human approval queue. We treat "the model decided" exactly like "an intern decided": fine for drafting, insufficient for destructive commits.
const TOOL_POLICY = {
search_orders: { requiresApproval: false },
update_shipping_address: { requiresApproval: false, maxPerRun: 1 },
issue_refund: { requiresApproval: true, maxAmountUsd: 200 },
} as const;
Policy as data, enforced outside the model. The prompt can ask nicely; the runtime says no.
Observability, or it didn't happen
When a customer reports "the assistant did something weird yesterday," you need the full trace: every model call, every tool invocation with arguments and results, latencies, token counts, and which prompt and model version were live at the time. Without that, every incident report is unreproducible folklore.
We log traces on every production agent, sample them into a weekly human review, and feed the interesting failures back into the eval set. That loop — production failure becomes eval case becomes regression test — is the whole quality flywheel. It's also the honest answer to "how do you keep it from breaking again": the same way we do with normal software, by turning incidents into tests.
Version prompts like code, because they are code. A prompt edit ships through review, runs against the eval set, and can be rolled back. The alternative — someone tweaking the system prompt in a dashboard on a Friday — is how you get a support agent that's been quietly overpromising for a week.
RAG, fine-tuning, and the boring answer
Clients ask about fine-tuning in the first meeting, almost every time. Our answer is almost always the same: retrieval first.
Retrieval wins when the problem is knowledge — your docs, your policies, your catalog. It updates instantly when a document changes, it cites sources so you can audit answers, and it doesn't require retraining when the return policy changes. Fine-tuning wins when the problem is behavior — a specific output format, a house style, a narrow classification task where you have thousands of labeled examples. It does not reliably teach a model new facts, and using it as a knowledge store means every content update becomes a training run.
Most production systems we ship use retrieval plus careful prompting, on a frontier model, with no fine-tuning at all. It's the boring answer. It's also the one that's still maintainable a year later by whoever inherits it.
What we tell clients before we start
Three honest limitations, stated up front. First: the system will sometimes be wrong, so we design workflows where wrong is cheap — drafts a human approves, answers with citations, actions with undo. Second: reliability comes from the scaffolding, not the model, so most of the budget goes to evals, guardrails, and observability rather than prompt wizardry. Third: this is not a ship-and-forget project. Models get deprecated, usage drifts, and the eval suite needs feeding.
An agent that survives production isn't the one with the cleverest prompt. It's the one wrapped in the most honest engineering.