Visual Guide to Agentic AI System Design
Let's get the most common misconception out of the way first: a modern agent is not a smarter chatbot. It's a compound system, a foundation model augmented by scaffolding that enables memory, planning, and tool use. The model is one component, and often not the most consequential one.
Five layers constitute that scaffolding, arranged from the inside out.
The policy core is the LLM itself, the function that maps everything the agent currently knows to a next action. Around it sits the memory layer, which determines what the agent retains across steps and sessions. Outside memory is the planning layer, where the agent sequences actions toward a goal rather than just generating a response. Further out is tool use, the interface through which the agent acts on the world: querying databases, calling APIs, writing files, controlling software. At the outermost ring sits coordination, the architecture for how multiple agents divide work when a single agent isn't sufficient.
The visual logic here matters. This is not a pipeline where input enters one end and output exits the other. It's a loop. Perception and tool output feed back into the context at every level. An agent reads a document, forms a plan, calls a tool, reads the result, revises the plan. That loop structure is what makes agents qualitatively different from a single model call, and it's also what makes failure modes propagate in ways that aren't obvious until you've watched one cascade in production.
Here's what that means practically. A weak memory layer doesn't just limit what the agent remembers; it undermines planning, because the planner is working with incomplete context. An ungoverned tool layer doesn't just expose the tool — it exposes every layer above it, because the model controlling that tool has no principled way to distinguish a legitimate instruction from an injected one. Each layer is load-bearing. You find that out either by designing around it or by debugging around it later.
The LLM policy core: what sits at the center of every agent
The model at the center of an agent is doing something specific: not generating prose, but making decisions. Given a context window populated with instructions, retrieved documents, tool outputs, and conversation history, it produces a next action. A plan step. A tool invocation. A response. That decision function is what "policy" means here.
In 2025, the leading models were explicitly built for this kind of work. Anthropic's Claude 4 variants were designed for agentic deployment: tool invocation, file access, extended memory, long-horizon reasoning, and what Anthropic calls "computer use," the ability to control a virtual screen as a tool surface. Google's Gemini 2.5 brings large context windows, native multimodal input, and integrated tool use for browser navigation and document manipulation. Meta's Llama 4 Scout and Maverick offer multimodal capability and extremely large context lengths, with the accessibility advantages of an open model.
The context window is the architectural constraint that every other design decision negotiates with. Everything the agent sees at decision time must fit inside it: instructions, retrieved memory, tool outputs, the running conversation. This is precisely why external memory and retrieval systems exist. The window is finite; the task usually isn't.
Model choice shapes the scaffolding you build around it in ways that aren't always obvious upfront. A model with a million-token context window substantially reduces retrieval pressure, because more can live in-context rather than being fetched from a vector store at runtime. A model with high reliability on structured tool calls reduces error-handling complexity in the planning layer, because malformed invocations become a smaller problem to architect around. Teams that treat model selection as purely a capability question tend to re-encounter it as a systems design question, usually at the worst possible time.
How the memory layer gives agents continuity across steps and sessions
Without external memory, an agent starts from zero every time. Its parametric knowledge, what the base model absorbed during training, is static. It doesn't update. The context window, however large, is finite and ephemeral; when a session ends, it's gone.
Four memory types address different parts of this problem, and in practice, architects combine them.
In-context working memory is the active context window: fastest, most immediately accessible, most expensive to maintain, and completely transient. Retrieval-augmented memory, implemented through vector stores or knowledge graphs queried at runtime, extends the agent's reach without expanding the per-call context cost. Fetch what you need when you need it. External structured storage, databases, documents, third-party APIs, handles persistent state that lives entirely outside the model. And parametric memory is simply what the base model already knows: always present, never modified by anything that happens at runtime.
The 2025 research frontier is instructive about where retrieval still breaks down. Mem0's benchmark at ECAI 2025 compared ten memory approaches on the LoCoMo benchmark; the two largest performance gains in the leading approach were on temporal queries, up 29.6 points, and multi-hop reasoning, up 23.1 points. Those two categories keep appearing in failure analyses, and for good reason. Temporal queries require an agent to reason about when something occurred, not just that it did. Multi-hop reasoning requires chaining evidence across multiple retrieved fragments. Systems that treat memory as a static lookup table consistently struggle with both. It's less a retrieval problem than a representation problem, and that distinction changes what you build.
Several architectures addressing these failure modes emerged in 2025 and early 2026: MAGMA, working with multi-graph memory structures; HyMem, combining hybrid memory types; Zep, using a temporal knowledge graph. Each is a different answer to the same underlying question: how do you make retrieved memory reason-ready rather than merely recall-complete?
I keep coming back to this failure mode because it's so easy to misattribute. A planning module working with a shallow memory layer will re-derive conclusions it already reached, ask questions it already answered. That inefficiency compounds across every subsequent step, and it presents as a planning problem until you trace it back to the memory layer.
Four single-agent reasoning patterns and when each one fits
Planning is the layer where the agent decides what to do next. Not what to say; what to do. It decomposes a goal into a sequence of executable steps, typically using chain-of-thought reasoning as the generative mechanism. Four patterns dominate production implementations.
ReAct, short for Reasoning plus Acting, cycles through reason, act, observe, and loops until the task resolves. It's the right default when each next step depends on what the previous step returned, when the task is open-ended enough that you can't anticipate the path upfront. Flexible, widely applicable, and it accumulates token costs in proportion to task length. On long tasks, that accumulation is not trivial.
Plan-and-Execute commits to a full plan upfront, then uses a cheaper executor to carry out individual steps. On well-defined long tasks, this is meaningfully more efficient than ReAct because you eliminate the repeated prompting overhead of cycling through the loop. A common hybrid in practice: Plan-and-Execute as the outer architecture, with ReAct sub-agents handling individual steps internally. Planning efficiency at the macro level, adaptive flexibility at the micro level.
ReWOO, Reasoning Without Observation, pushes the separation of concerns further. The planner generates all tool calls upfront. A worker executes them. A solver integrates the results. No interleaving of reasoning and execution. The token reduction relative to ReAct on equivalent multi-step workflows runs roughly 30 to 50 percent. The trade-off is brittleness: if an intermediate result would have changed the plan, ReWOO has no mechanism to incorporate that signal. You're committing to a plan made without seeing the evidence that might have revised it. That's fine when the task structure genuinely supports it, and it's a real problem when it doesn't.
Reflexion retries with self-critique stored in memory, improving output quality across iterations. It addresses the problem of a single attempt being insufficient by building a feedback loop into the execution architecture. The documented failure mode is telling: when the same model generates both the output and the critique, it reinforces earlier misconceptions rather than surfacing them. The critic and the actor share the same blindspots. The practical fix is pairing ReAct at the execution layer with Reflexion at the workflow level, keeping the critic structurally separate from the executor.
One figure worth keeping in mind: Tree of Thoughts, a less common fifth pattern, can cost ten to one hundred times more than standard reasoning approaches. Pattern selection is a cost architecture decision as much as a capability architecture decision, and that calculation needs to happen at design time.
How multiple agents divide work: six coordination patterns in production
Single agents have ceilings. A finite context window limits how much information one agent can reason over simultaneously. A single capability set limits what kinds of subtasks it can execute well. When tasks grow complex enough, coordination architecture becomes the central design question.
Orchestrator-Worker is the right starting point for most production systems. A central orchestrator decomposes the task and delegates to specialized workers. Control flow is debuggable; accountability is clear; costs are predictable. The overhead to keep in mind: the orchestrator makes multiple LLM calls for decomposition and aggregation on top of every worker call. A workflow that costs fifty cents in testing can reach tens of thousands of dollars monthly at scale. That arithmetic needs to happen before production.
Sequential Pipeline runs agents in a fixed linear chain, each consuming the previous agent's output. Deterministic order, set at design time. The natural fit is structured document workflows: contract generation, content moderation, multi-stage analysis where the sequence is known and invariant.
Fan-Out / Fan-In dispatches to multiple agents simultaneously and aggregates when all return. Wall-clock latency is bounded by the slowest branch, not the sum of all branches. When the workload is dominated by independent parallel tasks, this is the pattern that makes the latency economics work.
Hierarchical architectures introduce layers of orchestrators managing sub-orchestrators managing workers. This becomes appropriate when the worker count exceeds the range where a flat orchestrator-worker model remains manageable, roughly five to eight workers. Below that threshold, the coordination overhead typically isn't justified.
Swarm and Mesh patterns distribute coordination across peers without a central orchestrator. The use cases are genuinely narrow: exploration workflows, tight peer-collaboration scenarios where no single agent has enough information to direct the others. Most teams that reach for swarm architectures early would be better served by a well-structured orchestrator-worker system. Swarm feels architecturally sophisticated, and that is usually why it gets chosen prematurely.
Supervisor / Reflection inserts a dedicated critic agent that reviews results and forces revision before they surface. This directly addresses the single-agent Reflexion problem. The critic is a different agent from the actor, which means their failure modes don't perfectly overlap. A structural solution to a structural problem.
The principle across all six patterns: match the pattern to the task structure. Escalating to hierarchical or swarm architectures prematurely almost always reflects an aesthetic preference for sophistication rather than an honest analysis of what the task actually requires.
The two protocols that wire agents to tools and to each other
Coordination patterns are logical. Protocols are what make them executable across different tools, frameworks, and vendors. Two protocols have solidified as the dominant standards, and their division of labor is precise.
MCP, the Model Context Protocol, is Anthropic's open standard for agent-to-tool connectivity, introduced in November 2024 and donated to the Linux Foundation's Agentic AI Foundation in December 2025. MCP is how an agent reaches a capability in the outside world: a database, a browser, a code execution environment, a file system. Every MCP server an agent can reach is a capability surface. Think of MCP as the universal adapter between agent and environment.
A2A, the Agent2Agent Protocol, is Google Cloud's open standard for structured communication and delegation between autonomous agents, announced April 2025 and donated to the Linux Foundation in June 2025 with more than fifty partners including AWS, Microsoft, Salesforce, and SAP. A2A is how agents hand tasks to each other across vendor and framework boundaries. Where MCP wires an agent to the world, A2A wires agents to each other.
IBM's Agent Communication Protocol, ACP, merged into A2A under the Linux Foundation in August 2025. That consolidation matters: the inter-agent communication space is converging on A2A rather than fragmenting across proprietary implementations.
MCP and A2A coexisting in an agentic stack is analogous to HTTP, WebSocket, and gRPC coexisting in web infrastructure. They handle different connection types, and a mature stack uses both intentionally. They are not competitors.
What follows from that framing has real operational weight. Every MCP server in the stack is a capability surface requiring intentional access control. Every A2A connection is a trust boundary requiring intentional verification. Both are points of design, not points of convenience, and treating them as the latter is how ungoverned surfaces accumulate quietly until they become a serious problem.
Where safety and governance fit in the architecture, not after it
Conventional AI risk is largely contained: a model produces text, a human reads it. Agentic risk is categorically different. An agent that can read a document and then act on its contents can also be instructed by that document. Prompt injection requires no malware, no credential compromise, no sophisticated attack vector. It requires only text the model is inclined to trust: a webpage, a PDF, an email body, a tool output. The attack vector is the input stream itself.
Guardrails operate at two levels, and both are necessary.
Design-time controls are constraints built into the architecture before deployment: data validation, scope limits, restricted capability grants, the deliberate decision about which MCP servers the agent can reach. These are structural, not operational.
Runtime controls are enforced as the agent executes: human-in-the-loop checkpoints, anomaly detection, access controls that verify before the agent acts rather than log after it does.
Human-in-the-loop is not a single setting; it's a spectrum calibrated to risk. It is appropriate when an action is high-stakes, a financial disbursement, a legal agreement, an irreversible write to a production system. As a blanket policy, it is actively counterproductive, because it eliminates the latency advantage that makes agentic systems worth building in the first place. The design question is where in the workflow the action's risk profile justifies the pause, not whether to have human oversight at all.
Least privilege applies to agents as directly as it applies to human users. Every tool connection, every MCP server the agent can reach, is a capability that should be granted explicitly and reviewed deliberately. An ungoverned tool surface becomes an attack surface at registration, not at exploitation.
Visibility is the difference between governance and the appearance of governance. An audit log read after an incident is a postmortem. Actual governance means knowing what an agent touched, as it touches it, with enough resolution to intervene. The teams I've seen move most durably into production agentic systems built the observability layer before they needed it. Not because they were uniquely cautious, but because without it, their legal and compliance stakeholders had no defensible basis for approval. The governance layer wasn't something bolted on after the architecture was settled. It was what made the architecture approvable.
