This is a sequel to the viral prequel AI Agents 101 - which went so viral they made a coin for it.
by
Introduction
So, you've built an agent that kinda barely works in demos and now you want to deploy it for real users with real use cases. Demos and production have different requirements; that transition is where most agent projects die.
This is a sequel to the viral prequel AI Agents 101 - which went so viral they made a coin for it. This article will cover what comes after you've built something that works in controlled conditions. Let's get started.
Why Demos Don't Translate to Production
Demo agents operate under conditions that simply don't exist once real users start interacting with your system.
When you demo an agent, you choose inputs that showcase its capabilities, meaning you know which examples work well and which don’t. Production doesn't give you that luxury because users can send whatever they want, ranging from ambiguous phrasing to incomplete information. A lot of these requests fall outside what the agent was designed to handle, which ultimately breaks your parsing logic.
When you demo an agent, you're actively watching it work, meaning if something goes wrong, you notice immediately and can intervene. This also gives you the luxury of being able to explain what happened, or restart with different inputs. Production agents run autonomously without anyone watching each interaction. This leads to failures that accumulate in logs until your customers start complaining about getting the wrong support tickets or architectural damage. The agent in itself might have hundreds of bad decisions, but now needs manual cleanup.
It’s natural to avoid edge cases because they complicate the narrative and make the demo longer, but that’s exactly where production agents prove their value or create their biggest problems. The common, straightforward cases are usually easy to handle and often could be solved with simpler automation. But the weird cases are where agents either demonstrate genuine intelligence or expose their brittleness.
Production agents often run extended sessions where context accumulates across many interactions, which means that state management becomes critical. Context window limits become real constraints because large sessions fill available tokens and force you to decide what to keep, which is often one of the most important decisions you can make when building agents for production.
Closing this gap requires deliberate work across three areas: hardening the agent to handle real-world conditions, building eval systems, and creating knowledge infrastructure that improves over time.
The 6 Pillars of Agents in Production (notice how a lot of this is just software engineering)
Input validation and normalization - should happen before your agent's core logic ever sees the data. Define exactly what input formats you accept and build a preprocessing layer that validates incoming data against those formats. When the formatting doesn’t match, just reject them with error messages. Example: if your agent expects product names, build a normalization layer that handles common typos, abbreviations, case variations, etc.
Graceful degradation - explicit behaviors for situations where the agent cannot complete its intended task. You need to decide whether the agent should return a partial result with a clear indication of what's missing, or should it ask the user for clarification before proceeding? The worst possible outcome is an agent that silently produces garbage when something goes wrong.
Timeout and retry logic - acknowledging that external dependencies are unreliable. APIs can go down for maintenance, and databases sometimes slow under unexpected load. When these calls fail, your retry logic should distinguish between transient failures (temporary rate limit) and permanent failures (invalid credentials). Transient failures warrant retries with exponential backoff: wait a short time, try again, wait longer if it fails again, up to n number of attempts.
State checkpointing - define checkpoints at natural boundaries in your workflow where the agent has completed a meaningful chunk of work. Save the current state to persistent storage at each checkpoint, including everything the agent would need to resume: what inputs it received, what decisions it has made so far, and what remains to be done. If the agent crashes or times out mid-task, it can load the most recent checkpoint and resume rather than starting over from scratch.
Rate limiting and resource management - the things that protect you from runaway costs. An agent that makes API calls in a loop without rate limits can generate thousands of dollars in charges, which is why you need to build explicit limits on every resource your agent consumes: maximum API calls per minute, maximum database writes per task, maximum tokens generated per session, maximum execution time per workflow. Monitor consumption against these limits and alert when usage approaches thresholds.
Audit logging - what helps you capture the decisions that your agent makes with enough context to understand why it made that decision. For each action, log: the input state the agent was responding to, any external data the agent retrieved, the reasoning process, the action the agent chose, the parameters of that action, the outcome after execution, and any error conditions encountered. Structure these logs so they can be queried and analyzed programmatically. If something goes wrong, check your logs.
Simplifying Agent Evals
Most teams either skip systematic evaluation entirely (relying on manual testing and vibes) or build evaluation infrastructure way too complex. Both approaches fail because the first gives you no real confidence in your agent's behavior while the second creates evaluation systems that fall out of sync with reality.
Evals needs to answer three core questions about your agent: 1) Is it doing what it should do in the cases where it should act? 2) Is it avoiding action in cases where it shouldn't act or doesn't have enough information? 3) Is it performing acceptably on the cases that matter most to your users and your business?
Golden datasets form the foundation of systematic evals, which you can do by collecting real examples from your domain, ideally from actual user interactions if you have them. For each example in your golden dataset, document: the exact input the agent receives, the expected output or action, the reasoning for why that output is correct, and any relevant metadata like case type or difficulty level. ~100 examples are enough for narrow scoped tasks (which is all your first agent should be), and you can focus on situations where the consequences actually matter.
Metrics need to reflect what actually matters rather than collapsing everything into a single accuracy number that hides important variation, so if 90% of your cases are straightforward and 10% are complex, an agent that handles easy cases perfectly but fails on hard ones will show 90% accuracy while delivering poor value on the cases that actually need agent intelligence.
Evaluation should be integrated into your development workflow. You can configure your CI/CD pipeline to execute your evaluation suite on every pull request that touches agent logic, and if eval scores drop below acceptable thresholds, block the change from merging until someone investigates whether the regression is real and how to address it.
Note: this section (and others, tbh) could be entire books on their own. I’m being intentionally brief here to give you a starting point to dive deeper. I highly encourage you to do so.
Knowledge Infra that Improves Itself
Agents depend on knowledge to do their jobs, which can mean product information, company policies, historical context, domain expertise, etc. How you manage this knowledge determines whether your agent gets smarter and more accurate over time or gradually degrades.
Common patterns for knowledge storage include vector databases for semantic search over unstructured documents, relational or document databases for structured factual data with clear schemas, and configuration files for explicit policies. The right choice depends on how knowledge will be queried and how frequently it changes. Most production systems use a combination, with different types of knowledge in different stores suited to their access patterns.
Versioning all knowledge changes gives you the ability to understand what changed when, diagnose behavioral shifts, and also roll back if an update causes problems, which is especially useful when your agent starts behaving differently. If knowledge changes aren't tracked, you have no way to answer this question and debugging becomes much harder, so use version control.
Multi-Agent Orchestration in Production
Some workflows exceed what any single agent can handle effectively, and that’s because of context windows. The limit usually prevents a single agent from maintaining all relevant information across a complex multi-step process, which is why you need multiple agents all running in parallel, with their isolated context windows.
But since there are more agents, there are more evals, which means more failure nodes to take care of. That’s why pipeline architecture is important. Pipeline architectures arrange agents in sequence where each agent completes its work before passing results to the next. Example: agent A receives raw input and performs classification, determining what type of request this is and which downstream path it should follow, while agent B receives classified input and performs core processing, applies business logic, and generates initial outputs. Agent C receives these outputs and handles final formatting responses.
Parallel architectures are just as important. They run multiple agents simultaneously on different aspects of the same input, merging results when all agents complete. A deal approval workflow might run a legal review agent, a financial analysis agent, and a technical assessment agent in parallel, each examining the deal from their specialized perspective.
Production Readiness Checklist
Before deploying an agent to handle real work, you need the following to work:
Input handling: Does the agent validate inputs against expected formats? Does normalization handle common variations users actually send?
Failure modes: Are fallback behaviors defined for each external dependency failure? Does the agent recognize when it lacks sufficient information to proceed?
Resource management: Are timeouts configured for all external calls? Does retry logic distinguish transient from permanent failures?
Observability: Does audit logging capture every decision with full context? Can you reconstruct agent reasoning from logs after the fact?
Evaluation: Does your golden dataset cover common cases, edge cases, and high-stakes cases? Do evaluations run automatically when code changes?
Knowledge: Is knowledge externalized from agent code? Are knowledge changes versioned and tracked?
Human oversight: Are escalation criteria defined? Do escalations include enough context for humans to resolve quickly?
This checklist will get you the right infrastructure to start thinking about how to deploy agents into production. Hope you found something useful.
If you're a business that needs AI agents built for your internal processes, book a free AI audit at varickagents.com
For weekly insights on building agents that survive real-world conditions, subscribe at varickagents.com/newsletter
