Why isn't a single retrieval-plus-generate call enough for an AI agent?

Plain RAG handles one retrieval and one generation pass well, but breaks down for queries that need multiple steps — calling an internal API for one fact and a search tool for another, then reasoning over both. An agent loop lets the model decide, step by step, whether to retrieve more, call a tool, or answer, rather than forcing every query through exactly one retrieval-then-generate shape.

What stops an agent loop from running forever?

An explicit step budget — a maximum number of loop iterations set before the interaction starts. Without one, a model that keeps deciding 'I need one more piece of information' has no natural stopping point, and both latency and cost become unbounded. The budget is a deliberate trade-off: more steps can mean better answers on complex queries, at the cost of slower and more expensive responses.

Why is prompt injection a bigger concern in agent systems than in plain chatbots?

Because an agent doesn't just read retrieved text and summarize it — it can act on tool outputs and retrieved content by feeding them back into its own reasoning loop. If a retrieved document or a tool's response contains text designed to look like an instruction ('ignore previous instructions and...'), an agent that treats all input as equally trustworthy can be hijacked into taking unintended actions. Retrieved and tool content has to be treated as untrusted data, never as instructions, throughout the loop.

What's the detail that separates a senior answer from a mid-level one here?

Recognizing that failure containment has to happen at the level of an individual step, not the whole session — a single tool timeout or a bad retrieval shouldn't crash an entire multi-turn conversation. The failure needs to be fed back into the model's own context as information it can reason about (retry, try a different approach, answer with a caveat), rather than the orchestrator simply giving up.

RAG and AI Agent Platform Design

A chatbot that answers from one retrieval pass is a good demo. The genuinely hard version is a system that can decide it needs to check an internal API, notice the API failed, try a different approach, and still return a grounded, safe answer. What makes it hard specifically isn't the retrieval and it isn't the generation — both of those are solved problems on their own. It's that the model now gets to make decisions mid-flight, and every one of those decision points is a new place for the system to be wrong.

What the Platform Is On the Hook For

Answer a query by retrieving relevant context and generating a grounded response; support multi-step reasoning where the model calls tools (search, internal APIs, calculators) and uses the results to continue; maintain conversation memory across turns; validate outputs for groundedness and safety before they reach the user. Out of scope: the LLM inference platform itself (this system is a client of it), the vector database's internals (indexing, ANN search — a dependency this system calls, not implements), and document ingestion/chunking (a knowledge base of already-chunked, already-embedded documents is assumed to exist).

At ~10,000 agent sessions/second, each involving an average of ~3 underlying LLM calls (an initial reasoning step, a tool-call decision, a final synthesis), that's roughly 30,000 LLM calls/second directed at the inference platform — the number that actually matters for capacity-planning the dependency, not the session count alone. Most sessions trigger at least one retrieval too, on the order of 10,000 vector-search queries/second. An end-to-end interaction should complete in a few seconds even though each step can itself take hundreds of milliseconds; a single tool-call failure mid-chain must never crash the whole interaction.

Five decisions, each a real fork, get from "answer a question" to that.

Fork One: Stuff the Prompt, or Retrieve First?

Stuffing the user's query and as much of the knowledge base as fits into one giant prompt, then making a single LLM call, doesn't scale — context windows are finite and expensive, most of a large knowledge base is irrelevant to any given query, and the model has no way to take an intermediate action if the answer needs something the stuffed-in text can't provide. Retrieval-augmented generation — embed the query, pull the top-k relevant chunks from a vector store, insert them into the prompt — bounds context to what's actually relevant and grounds the answer in real material.

Fork Two: One Retrieval Pass, or a Loop?

Standard RAG breaks down for queries that need more than one retrieval or an action beyond retrieval — "compare our Q3 revenue to the industry average" needs an internal API call for one fact and a search for the other, then reasoning over both, which a single retrieve-then-generate pass structurally can't do. An explicit agent loop fixes this: on each iteration, the model gets the current context (query, retrieved chunks so far, tool results so far) and decides whether to retrieve more, call a tool, or produce a final answer, continuing until it answers or a step budget — a hard cap on iterations, bounding worst-case latency and cost — runs out.

Fork Three: Trust the Tool Call, or Validate First?

A model deciding to call a tool is a request, not a command that should execute unchecked. Every tool call goes through a registry with a defined argument schema, and the model's requested call is validated against that schema before anything actually executes — a malformed or unsafe request never reaches the real tool, regardless of how the model arrived at it.

Fork Four: Crash the Session, or Reason About the Failure?

A tool timing out or erroring mid-chain can either take down the whole interaction or get caught and fed back into the model's own context as information — "this tool call failed, here's why." The second path lets the model itself decide whether to retry, fall back to a different tool, or answer with an explicit caveat about the missing information, rather than the orchestrator simply giving up on behalf of a model that might have had a perfectly good fallback.

Fork Five: Trust the Generation, or Check It?

Retrieval happening somewhere upstream in the chain doesn't guarantee the final answer is actually grounded in it. A guardrail step checks the generated answer against the chunks that were actually retrieved — often a lighter-weight consistency/entailment check, sometimes a smaller dedicated model call — and either passes it through, flags it, triggers a regeneration, or annotates it with citations back to specific source chunks.

Where the Step Budget and Memory Compaction Fit In

Two details that don't fit neatly into any single fork above but shape the architecture as much as any of them. Within one step, if the model requests multiple independent pieces of information (two unrelated tool calls), running them concurrently rather than serially meaningfully cuts that step's latency, since neither depends on the other's result. And session memory can't just grow forever: recent turns are kept verbatim, but older turns get compacted — summarized into a condensed representation — once the conversation grows past a length where full verbatim history would consume an unreasonable share of the context budget on every subsequent call. This isn't a UX nicety; context consumption directly multiplies the compute cost of every call on the inference platform, so compaction is cost control wearing a memory-management hat.

http

POST /v1/agent/sessions
{ "user_id", "initial_query" }
→ { "session_id", "response" (streamed), "steps": [...] }

POST /v1/agent/sessions/{id}/messages
{ "message" }
→ { "response" (streamed), "steps": [...] }

What Makes a Bad Answer Debuggable

AgentSession — one conversation, spanning multiple turns, holding accumulated memory/state
Step — one iteration within a session: a tool call or an LLM generation, with its inputs and outputs
Tool — a registered capability the agent can invoke, with a defined argument schema
RetrievedChunk — a piece of context pulled from the vector store for a given step
GuardrailResult — the pass/fail outcome (and reason) for a step's output

The modeling choice worth explaining if asked: Step is logged as its own durable record, not just an ephemeral in-memory value inside a loop — every tool call, every retrieval, and every intermediate generation is individually reconstructable after the fact. This is what makes debugging a bad agent answer tractable (which step went wrong) and what makes resuming a crashed session possible, instead of a session being an opaque black box that either fully succeeds or restarts from scratch.

What a Dependency Outage Actually Means Here

Failure	Impact	Response
Orchestrator crash mid-session	Session state could be lost	Session state (steps completed so far) is persisted durably after each step, not just held in memory — a new orchestrator instance resumes from the last completed step instead of restarting the whole session
LLM serving platform outage	This system cannot function at all — it's a hard dependency	Worth stating explicitly rather than pretending otherwise; the honest mitigation is the inference platform's own redundancy, not anything this layer can compensate for
Vector store outage	Retrieval-dependent steps degrade	The agent can fall back to a "no retrieval" mode for queries that don't strictly require grounding, with reduced answer quality flagged to the user, rather than failing the whole session
Tool provider outage	One tool becomes unavailable mid-chain	Handled the same way as any tool failure — surfaced to the model as context it can reason around, not a hard stop

The Threat Model Unique to Agents

Tool execution is effectively the model requesting an action be taken on its behalf, which means every tool call should run with least privilege and validated arguments — never assume a model-generated request is automatically safe to execute as-is. The more distinctive risk here is prompt injection through retrieved or tool content: if a document in the knowledge base, or a tool's response, contains text crafted to look like an instruction, an agent that treats all input uniformly can be hijacked into taking unintended actions. The mitigation is architectural, not just a filter — retrieved and tool-returned content must be treated as untrusted data the model reasons about, never as instructions it reasons from.

Signal	Why it matters	Alert on
Per-step latency breakdown	Multi-step latency issues are invisible in an end-to-end number alone	Any single step type (retrieval, tool call, generation) regressing vs. its own baseline
Tool call success/failure rate, per tool	Flags a specific failing dependency fast	Sharp rise vs. baseline for one tool
Guardrail flag rate	A rising rate can mean the underlying model or retrieval quality has regressed	Sustained rise vs. baseline
Average steps per session	A cost proxy — more steps means more LLM calls means more spend	Sustained upward drift, which may indicate the model is looping unnecessarily

The Bar by Level

Mid-level: Designs a standard RAG pipeline (retrieve once, generate once), and can be prompted toward a multi-step agent loop once a query requiring multiple actions is raised.

Senior: Independently proposes the agent loop with an explicit step budget; addresses tool-call failure containment (surfacing failures into the model's context) without prompting.

Staff+: Raises prompt injection from retrieved/tool content as a first-class threat model unprompted, not a generic security afterthought; designs grounding/hallucination checks as an explicit guardrail step rather than trusting retrieval-implies-grounded; reasons about session memory compaction as a cost-control mechanism tied directly to the underlying inference platform's economics, not just a context-window technicality.

RAG / AI Agent Orchestration Platform Design