Human-in-the-Loop AI Agents: The Pattern That Works

GitHub's engineering team recently published details on an internal Copilot workflow they call "PR Sous Chef." It doesn't run continuously. It doesn't act on every pull request the moment something changes. Instead, it wakes up every 15 minutes, looks at the state of every open PR, decides which ones actually need attention, and only then triggers a targeted action — a nudge, a review, a summary — for that narrow set. Everything else, it leaves alone. In an industry currently selling "fully autonomous agents" as the end goal, a team as sophisticated as GitHub's shipped the opposite: an agent that spends most of its time deciding whether to act at all. That restraint isn't a limitation. It's the design choice that's actually making it to production, while the always-on version mostly stays in the demo.

The Two Philosophies Businesses Are Actually Choosing Between

Most vendor pitches frame AI agents as a spectrum with one direction: less human involvement is always better, and the goal is an agent that needs no supervision at all. In practice, businesses deploying agents today are really choosing between two different architectures, and they trade off very differently:

• Always-on autonomous — the agent watches a stream of events continuously and acts the instant a trigger condition is met. It feels the most "agentic," but every action it takes is a live decision made with no human anywhere in the loop, and the cost meter runs the entire time it's watching, not just when it's doing useful work
• Interval-based, supervised — the agent wakes up on a schedule, evaluates the current state against explicit thresholds, and only escalates or acts on the subset that crosses them. Between check-ins, nothing is happening, nothing is costing compute, and nothing is capable of going wrong unsupervised

The always-on version sounds more impressive in a sales deck. The interval-based version is what GitHub, and most teams that have actually run agents in production for a while, land on — because the failure modes of "always watching, always empowered to act" are worse than the inefficiency of checking in every fifteen minutes instead of instantly.

Why Always-On Agents Quietly Become a Liability

An agent that can act the instant a condition is met also has to be trusted to act correctly the instant a condition is met, with no window for a second look. That trust is expensive to build and easy to lose, and three problems show up almost every time a business runs this pattern past the demo stage:

1. 1Unbounded blast radius — if the agent misreads a signal, it has already acted before anyone notices, and the fix becomes "undo what the agent did" instead of "catch it before it happened"
2. 2No natural audit trail — a stream-reactive agent that's always watching produces a wall of micro-decisions with no obvious checkpoint to review, so teams either drown in logs or stop reviewing at all
3. 3Compute cost that scales with attention, not output — the agent is billed for every moment it's watching the stream, whether or not anything worth acting on ever happens, which is a different cost curve than "billed for the fifteen-minute checks that actually produced a decision"

None of these are exotic edge cases. They're the default behavior of the architecture. And they're exactly the failure modes that show up in the AI agent liability and spending-control conversations enterprises are already having — not because agents are unreliable in general, but because the always-on pattern removes the one thing that used to catch a bad call before it became a bad outcome: a checkpoint.

What the Check-In Pattern Actually Buys You

The interval-based design isn't a downgrade from autonomy — it's a different, more deliberate allocation of it. Each check-in is a bounded, loggable decision: here's the state, here's the threshold, here's what did or didn't cross it, here's what action followed. That structure produces three concrete advantages over the always-on version:

• A natural audit trail — every check-in is a discrete, reviewable event, so when something goes wrong you're looking at one decision with clear inputs, not reverse-engineering a continuous stream
• Cost that matches the interval you chose, not the volume of raw events — a 15-minute check-in costs the same whether the underlying stream generated 3 events or 3,000, which is the opposite of how always-on compute scales
• A built-in place to insert a human — because the agent is already pausing to evaluate, adding "and if the threshold is close, ask a person" costs almost nothing architecturally, whereas retrofitting a pause into a continuously-reactive agent means redesigning it

A Realistic Scenario

A Wizeb client, a regional home-services company, had an always-on agent watching its incoming lead queue, built by an earlier vendor to auto-assign and auto-message every new lead the moment it landed. It looked impressive in the sales demo. In production, it paged the on-call dispatcher for every borderline lead at 2 a.m., auto-sent a follow-up message to a duplicate lead before a human ever saw it was a duplicate, and racked up a steady background compute bill for a queue that, in reality, only needed real attention a handful of times per day. We replaced it with a check-in pattern: the agent evaluates the lead queue every 10 minutes, matched to how fast a genuinely hot lead actually goes cold, applies explicit scoring thresholds, and only escalates the leads that cross them — everything else waits quietly for the next check-in or gets handled automatically within pre-approved bounds. False overnight escalations dropped to zero, the duplicate-messaging problem disappeared because the check-in step catches duplicates before anything sends, and the compute bill dropped by more than half because the agent isn't billed for watching a queue that isn't moving.

How to Check Where Your Own Agents Stand

1. 1For each agent currently running, ask whether it acts the instant a trigger fires or on a defined interval — if it's the former, that was likely a default, not a deliberate choice
2. 2Match the check-in interval to how fast the underlying process actually changes, not to what feels most impressive; a faster interval than the process warrants only adds cost and false escalations
3. 3Write down the explicit threshold each agent uses to decide "act automatically" vs. "escalate to a human" vs. "do nothing" — if that threshold only exists implicitly in a prompt, it will drift the first time someone edits the prompt
4. 4Check whether every check-in decision is logged, including the "no action needed" ones — that log is what turns a black-box agent into one you can actually audit after something goes wrong
5. 5Before your next agent build, design the check-in point first and the action second — it's cheap to add now and expensive to retrofit later

How Wizeb Approaches This

When we design AI agents at Wizeb, we don't default to "as autonomous as possible." We start by mapping how fast the process the agent is watching actually changes, set explicit thresholds for automatic action versus human escalation, and build the check-in cadence and audit log in from day one — so the agent's blast radius stays bounded and its cost stays predictable, without losing the automation that made it worth building in the first place. That's usually the difference between an agent that gets quietly turned off after a bad night and one that keeps earning trust month after month. If you're not sure whether your current agents are reacting continuously when they should be checking in on a schedule, that's worth reviewing before the next one gets built the same way. Start at wizeb.com/services/ai-agents.