← All writing

2 min readBuilding AI systems

Designing AI agents that know when to stop

The hard part of agentic workflows isn't getting the model to act. It's making it stop, escalate, or retry predictably. A practical checklist for tools, permissions, and loop control.

  • Agents
  • Tool design
  • Reliability

Give a capable model a few tools and a goal, and it will happily take actions. The question that matters in a business setting is different: will it stop at the right moment, ask before doing something expensive, and leave a trail someone can follow afterwards?

Start with the workflow, not the agent

Write the process down as a person would do it, including the points where they check with someone. Many steps turn out to be deterministic and are better as plain code. Use the model where judgment is genuinely needed, such as reading an email, choosing a category, or drafting a reply.

Design tools like public APIs

  • Small, single-purpose tools with clear names and strict input schemas.
  • Errors that explain what went wrong and what to try, not stack traces.
  • Read and write separated, so a lookup can never accidentally change data.
  • Idempotent writes where possible, so a retry doesn't send the invoice twice.

Put limits outside the model

Budgets for steps, time, and spend belong in the orchestration code, not in the prompt. When a limit is hit, the agent should stop with a useful summary of what it did and what it was trying to do next.

Decide which actions need a human

Sort every tool by its blast radius. Reading is usually safe. Sending messages to customers, moving money, or deleting records should require approval, at least until the evaluation data says otherwise. Approval should be one click, with the agent's reasoning and the exact action shown.

Autonomy is something an agent earns with evidence, one action type at a time.

Make every run explainable

Log each step: the input, the tool called, the arguments, the result, and the model's stated reason. When something goes wrong (and it will), this trace is what turns a mystery into a fix.

A short pre-launch checklist

  • Can the agent finish, stop, or escalate on every path you've tested?
  • Are step, time, and cost limits enforced in code?
  • Does every high-impact action require approval?
  • Can you replay a failed run from its logs?
  • Do you have a baseline to compare it against: the time and error rate of the manual process?