The PM's Guide to AI Agents

An AI agent is a system where a language model does not just generate text but takes actions in a loop. The model receives an input, decides what to do, calls a tool or function, observes the result, and decides what to do next. This loop continues until the task is complete or the agent determines it cannot proceed. That is it. The core concept is simple even though the implementations can be complex.

Contrast this with the copilot pattern, which is what most AI products ship today. In a copilot, the human makes the decisions and the AI assists. GitHub Copilot suggests code, but the developer decides whether to accept it. ChatGPT answers questions, but the user decides what to do with the answer. In an agent pattern, the AI makes the decisions and the human supervises. A coding agent does not just suggest a fix; it writes the code, runs the tests, and submits a pull request. A customer service agent does not just draft a response; it looks up the order, processes the refund, and sends the confirmation email.

Real-world agent examples help clarify the concept. Coding agents (like Claude Code, Cursor, or Devin) can plan multi-step code changes, execute them, debug failures, and iterate. Customer service agents can resolve tickets end-to-end by querying databases, applying policies, and taking actions in backend systems. Research agents can search multiple sources, synthesize findings, and produce structured reports. Workflow automation agents can monitor conditions, trigger actions across systems, and handle exceptions. Each of these follows the same pattern: observe, decide, act, repeat.

The hype around agents has outpaced the reality. As of early 2026, most production agent deployments are narrow in scope, operating within a single domain with a limited set of tools. Fully autonomous general-purpose agents remain unreliable for anything beyond demos. This does not mean agents are not valuable. It means you need to scope them carefully, which is the PM's job.

The decision between an agent and a copilot is fundamentally about where you want the locus of control. Use a decision framework with four criteria: task definition clarity, error cost, user intent, and guardrail feasibility.

Agents make sense when the task is well-defined with clear success criteria (process this return, schedule this meeting, deploy this configuration). They work when the cost of errors is manageable, meaning mistakes can be detected and reversed without catastrophic consequences. They fit when users explicitly want automation rather than assistance, because they are doing the task repeatedly and find it tedious. And they are viable when you can build reliable guardrails: input validation, output checks, budget limits, and escalation triggers.

Copilots make sense when human judgment is critical to the outcome (strategic decisions, creative work, nuanced communications). They are the right choice when errors are expensive or irreversible (legal advice, medical decisions, financial transactions above a threshold). They work when users want to stay in control because the task requires their expertise or the stakes are high. And they are appropriate when the action space is too broad or too sensitive to constrain with automated guardrails.

Most products should start as copilots and graduate to agents. This is not a compromise. It is a sound product strategy. A copilot lets you observe how users interact with the AI, understand the common workflows, identify the error patterns, and build the evaluation infrastructure you need. Once you have high confidence in the model's judgment within a specific scope, you can automate that scope. Slack's AI features started as search and summarization copilots before adding automated workflow suggestions. This incremental approach builds user trust and gives you the data to make agents reliable.

As a PM, you do not need to build the architecture, but you need to understand the components well enough to make informed product decisions and have productive conversations with your engineering team. The agent stack has five key layers.

The orchestration layer is the loop that drives agent behavior. It receives an input, calls the model, parses the model's response into actions, executes those actions, feeds the results back to the model, and repeats. Product decisions here include: how many iterations should the agent be allowed (iteration limits prevent infinite loops and control costs), and what should happen when the agent exceeds its iteration budget (fail gracefully, escalate to a human, or return a partial result).

Tool and function calling is how the agent interacts with the world. Tools can be APIs, databases, file systems, web browsers, or any external system. The model decides which tool to call and with what parameters. The PM's job is to define which tools the agent has access to, which is a critical scoping decision. Every tool you add expands the action space and increases both capability and risk. MCP (Model Context Protocol) is emerging as a standard for connecting agents to tools, making it easier to give agents access to external systems without custom integration code for each one.

Memory determines what the agent knows and remembers. Short-term memory is the conversation context within a single session. Long-term memory stores information across sessions (user preferences, past interactions, learned facts). The PM decides what gets stored, how long it persists, and what privacy constraints apply. Planning is how the agent breaks complex tasks into steps. Some agents plan upfront (create a full plan before executing), others plan incrementally (take one step and reassess). The PM influences this by defining the complexity of tasks the agent should handle.

Guardrails are the safety infrastructure that keeps the agent within bounds. They include input validation (reject malicious or out-of-scope requests), output validation (check that actions are within policy), budget limits (maximum API calls, maximum cost per task), and human-in-the-loop triggers (escalate to a human when confidence is low or stakes are high). The PM defines the guardrail policies. Engineering builds the enforcement mechanisms.

The single most important scoping decision for an agent feature is how narrow or broad its capabilities are. Start narrow. A customer service agent that can handle returns and refund status checks across two backend systems is a viable first version. A customer service agent that can handle any customer issue across fifteen backend systems is not a first version; it is a multi-year platform investment.

Define the happy path explicitly: what is the most common task the agent will handle, and what does successful completion look like end to end? Then define the confusion path: what happens when the agent encounters an input it cannot handle, a tool that returns an error, or a situation that requires information it does not have? The confusion path is where most agent products fail. Users do not abandon agents when they succeed; they abandon agents when failure is confusing or opaque.

Set clear boundaries on what the agent can and cannot do, and communicate those boundaries to users. An agent that tries to do everything and fails at half of it is worse than an agent that does three things well and clearly tells the user when something is out of scope. Boundaries should be enforced technically (the agent literally cannot call certain tools or take certain actions) and communicated in the UI (the agent tells the user what it can help with).

The biggest scoping mistake is giving the agent too many tools too early. Each tool the agent can call increases the combinatorial complexity of its decision-making. An agent with 3 tools has a manageable action space. An agent with 30 tools has an action space so large that you cannot reliably test all possible tool-calling sequences. Add tools incrementally based on usage data that shows which tasks users actually need the agent to perform.

Agents fail in ways that copilots do not. A copilot that generates a bad suggestion is easy to ignore. An agent that takes a bad action in a loop can compound the error across multiple steps before anyone notices. Consider a workflow agent that misinterprets a user request, calls the wrong API, gets an unexpected response, misinterprets that response, and then takes a follow-up action based on its misinterpretation. By the time the user sees the result, the agent has made a chain of errors that is difficult to understand and potentially difficult to reverse.

Build defensive infrastructure from the start. Retry logic handles transient failures (API timeouts, rate limits). Timeout limits prevent the agent from running indefinitely. Cost caps set a maximum spend per task to prevent runaway token usage. Human escalation triggers route the agent to a human when it detects low confidence, encounters an error it cannot resolve, or reaches a decision point that exceeds its authorized scope. These are not optional safety features. They are core product requirements.

Monitoring for agents requires more instrumentation than traditional software. You need action traces that log every step the agent takes (what it decided, which tools it called, what results it got, what it decided next). You need tool call success rates to identify integration failures. You need task completion rates to measure whether the agent actually finishes what it starts. And you need cost-per-task tracking to ensure the agent is economically viable.

The PM's job is to define acceptable failure modes. When the agent cannot complete a task, should it return a partial result? Ask the user for clarification? Escalate to a human? Do nothing and explain why? These decisions depend on the use case and should be specified before engineering begins building the agent.

Task completion rate is the primary metric for any agent product. Of the tasks the agent attempts, what percentage does it complete successfully? This sounds simple, but defining 'successfully' requires thought. A customer service agent that processes a refund but sends a confirmation email to the wrong address has not completed the task successfully, even though it finished the workflow. Define success criteria for each task type the agent handles.

Time-to-completion compared to the human baseline tells you whether the agent is actually saving time. If your agent takes 4 minutes to complete a task that a human does in 2 minutes, the agent is not providing value even if it is technically autonomous. Track this metric carefully, and include the time users spend reviewing and correcting agent work, not just the agent's execution time.

Error rate and escalation rate are inverse indicators of agent reliability. The error rate measures how often the agent produces incorrect results. The escalation rate measures how often the agent gives up and routes to a human. A high escalation rate is not necessarily bad in early versions. It means the agent is correctly recognizing its limitations rather than plowing ahead and making mistakes. Over time, you want to see the escalation rate decrease as you expand the agent's capabilities.

User trust is the metric that determines long-term adoption. Measure it by observing behavior: do users let the agent run autonomously, or do they override it at every step? Do users increase their usage over time, or do they revert to manual workflows after trying the agent? Trust is built through consistent reliability and destroyed by unpredictable failures. One catastrophic error can set user trust back by months. A warning about cost: 'tokens used' or 'API calls made' are operational metrics, not success metrics. An agent that uses twice as many tokens but completes tasks correctly is better than an efficient agent that fails.

When an agent acts on behalf of a user, the question of responsibility becomes complicated. If an agent sends an email that offends a client, is that the user's fault (they authorized the agent), the company's fault (they built the product), or the model's fault (it chose poor wording)? Legally and practically, the answer is the company's fault. You built the product, you defined its capabilities, and you shipped it to users. This is why scoping and guardrails are not just technical concerns but ethical and business obligations.

Transparency means users should be able to see what the agent did and why. Every action the agent takes should be logged and visible to the user. The agent should explain its reasoning when asked. If the agent accesses data or systems, the user should know which data and which systems. This is not just good ethics; it is good product design. Users who cannot understand what the agent did will not trust it, and users who do not trust the agent will not use it.

Reversibility should be a core design principle. Can the user undo agent actions? If the agent sent an email, can it be recalled? If the agent modified a database record, is there an audit trail? If the agent made a purchase, can it be cancelled? Design agent actions to be reversible wherever possible, and clearly warn users before the agent takes irreversible actions.

Consent is about scope. The user authorized the agent to perform a specific task, but did they authorize every individual action the agent takes to complete that task? A user who asks the agent to 'clean up my calendar' probably expects it to reschedule low-priority meetings. They probably do not expect it to cancel a meeting with their CEO. Define consent boundaries clearly and enforce them technically.

Start with a workflow agent, not a general-purpose assistant. A workflow agent handles a specific, repeatable process: processing expense reports, triaging support tickets, generating weekly status updates from project data. These tasks have clear inputs, defined steps, and measurable success criteria. A general-purpose assistant ('just talk to the AI and it will figure out what you need') is dramatically harder to build reliably.

Pick a task with clear success criteria that you can evaluate automatically. 'Help users write better emails' is hard to evaluate. 'Process refund requests according to our policy, resulting in the correct refund amount applied to the correct payment method' is easy to evaluate. Your first agent should have a binary success metric: did it complete the task correctly or not?

Build comprehensive logging from day one. Log every model call, every tool invocation, every decision the agent makes, and every result it produces. This logging is not optional overhead. It is how you debug failures, improve performance, and build your evaluation dataset. You cannot improve what you cannot observe. At a minimum, capture the full action trace for every agent run, searchable by user, time range, and outcome.

Launch with a human in the loop and remove the human gradually based on performance data. In version one, the agent proposes actions and a human approves them. Once the approval rate exceeds 95% for a specific action type, consider auto-approving that action type. This graduated autonomy approach lets you expand the agent's independence based on evidence rather than hope. It also gives you a natural dataset of human-approved vs human-rejected actions that you can use to improve the agent's judgment over time.