We stopped treating agents as features once they could take actions
We build AiKey, an AI credential and governance layer for teams running models, tools, and agents.
One idea has changed how we think about the product: an agent stops being “just a feature” the moment it can do something outside its own context window.
It reads a document. It queries a system. It opens or updates a ticket. It calls an API. It takes the result and decides what to do next.
At that point, the difficult questions are no longer mostly about prompt quality.
They are operational questions:
- Which identity is this agent using?
- Which actions can it take, and in which environment?
- What happens if a tool times out or the loop keeps going?
- Can we reconstruct the decision path later?
- Can we see what this task cost before the bill lands?
We did not arrive at that view because we think agents need more ceremony. We arrived there because the opposite trade-off becomes expensive very quickly.
The demo boundary is real
A prototype can get away with a shared API key, a permissive tool wrapper, and a few logs. That setup is often exactly right for learning whether a workflow is useful.
The problem starts when the prototype reaches a real system.
A shared key no longer tells you which agent made a request. A broad permission no longer tells you whether the agent should be allowed to close a ticket instead of creating one. A pile of uncorrelated logs does not help much when a workflow makes several model calls, touches two services, retries a failing tool, and then returns a result that nobody can explain.
The model did not suddenly get worse. The environment got more consequential.
That distinction matters. Teams often react by looking for a better model, a smarter planner, or another orchestration layer. Those may help, but they do not solve the question of who is allowed to do what at runtime.
We now separate judgment from control
The agent should be free to make task-level judgments: which context matters, which tool to call next, whether an answer is complete.
The runtime should own the guardrails around those decisions.
For us, that means treating credentials, tool permissions, budgets, policy checks, audit events, and fallback behavior as parts of the execution path, not as documentation next to it.
The agent can propose an action. The runtime still decides whether that action is valid for this identity, environment, tool scope, and budget.
This is also how we think about the “agent harness” idea. A harness is not another word for an agent framework. It is the layer that connects reasoning to execution and makes the loop legible:
- assemble context;
- call the model;
- validate and execute a tool request;
- record the result and policy outcome;
- continue, stop, fail safely, or hand off.
The unglamorous parts matter. Timeouts. Retry limits. Idempotency. Model fallback. Step limits. Cancellation. Budget exhaustion.
Those are the parts that keep a helpful agent from becoming a surprisingly persistent background process.
The control layer should produce useful operational data
A good runtime layer does more than block risky actions. It produces the data needed to operate the system.
When a task goes wrong, we want to trace one chain: the triggering user or service, the agent identity, the credential or token used, the model route, the tool calls, the policy decisions, the failures, and the cost events.
Without that chain, every question becomes an investigation across separate dashboards. With it, the questions get simpler:
Was the request denied by policy? Did a tool fail? Did a retry loop inflate usage? Did the task run under the wrong environment? Which workflow owns the spend?
That is the practical reason we care about runtime governance. It makes agent systems easier to operate before they become harder to explain.
What this changed for us
We no longer describe the problem as “securing model access.” That phrase is too narrow.
The problem is governed execution: giving agents the ability to act while keeping the identity, permissions, cost, and audit trail connected to the same runtime.
At AiKey, that is the layer we are building around AI credentials, runtime policy, auditability, and cost attribution. We are not trying to replace existing IAM or gateway systems. The goal is to make AI calls fit into a control model that operators can actually work with.
If you are building agents that are beginning to touch real tools and systems, I would be curious how you draw this boundary. What was the first operational problem your prototype exposed?
More context on our approach: https://aikeylabs.com/zh/i/ih33
Business inquiries: aikeyfounder@gmail.com
You are falling into the classic trap of trying to solve runtime governance at the application layer. The entire discussion in this thread focuses on tracking logs, adding compensation logic, or layering external auditors—but you are ignoring the core vulnerability: ambient credential exposure.
If the agent has access to raw credentials or broad identity tokens at execution time, your policy checks and audit trails are just post-mortem accounting. An agent that hallucinates, gets prompt-injected, or hits an unhandled edge case can still bypass intended logic if the execution environment relies on application-level enforcement.
Here is where your argument breaks down:
Logs are not control: Recording that an unauthorized or unintended action occurred does not stop the side effect in real time.
Self-declared compensation is fragile: Relying on the workflow or agent to pre-declare its own rollback logic assumes the failure modes are predictable. They rarely are once external tools and side effects compound.
Separation of concerns is missing: Runtime control should strictly live down at the proxy/gateway level—enforcing ephemeral, single-use, tightly scoped tokens per action step—rather than relying on application harnesses to self-police.
Unless the governance runtime acts as a hard cryptographic boundary that mints just-in-time permissions for each individual tool call, all the audit trails and replay modes in the world won't prevent a runaway process from corrupting state before your policy engines catch it.
The connected audit trail is the part that stands out to me.
Once an agent is taking real actions, having the identity, tool calls and policy decisions recorded is necessary — but I’m curious how you verify that the resulting behaviour was actually correct.
For example, do you independently sample production runs to test whether the agent should have been allowed to act, whether escalation happened at the right point, and whether the evidence trail supports the eventual outcome?
That gap between having governance controls and proving they behaved correctly in practice is something I’ve been exploring with OpsWatch.
The sampling question is exactly the right one. Two things we're doing: policy shadow mode — new policies run in observe-only mode against live traffic before they ever block; and replay — completed runs get re-evaluated to check whether the agent stayed in scope and escalation fired when it should.
Honest caveat: sampled review proves the runs you looked at, not the system. The real loop is turning findings back into policy changes. How does OpsWatch close that loop — manual review, or automated comparison of expected vs. actual behavior?
Hybrid, but with independent human judgement remaining the final assurance layer.
We define the expected behaviour and control conditions up front, then compare those expectations against actual/replayed runs — things like whether the action was permitted, whether grounding supported it, whether escalation should have fired, and whether the resulting evidence is internally consistent.
Automation can do a lot of the comparison and exception-flagging at scale, but I wouldn’t want OpsWatch to become a system that effectively certifies itself. Material findings still need independent review.
The useful loop is:
expected behaviour → observed run → deviation/evidence → policy/control change → re-verification
Your shadow-mode + replay setup is actually very close to the evidence layer I’d want underneath that process. The interesting bit would be seeing whether OpsWatch could sit independently on top of it rather than duplicating what you already do.
Hybrid with independent human judgement as the final layer is the right call. The moment a governance system starts certifying itself, its evidence stops being convincing to anyone outside it.
That's actually why your "OpsWatch sitting on top" framing doesn't threaten us — it's the architecture we'd want. Our job is to make the runtime produce evidence that holds up to external verification: append-only audit events, replayable runs, stable schemas, no backdoors for the controller to rewrite history. If a tool like OpsWatch can consume that stream and do the expected-vs-actual comparison independently, that's not duplication, that's the loop working as intended.
The one gap I'd flag from our side is that most teams' audit data isn't structured enough for automated deviation checks yet — it's logs, not evidence. Getting from one to the other is where I think the real work is, and it's the part we're spending most of our time on.
This is exactly the distinction I think matters.
I agree that the valuable boundary is not “another system checking the same logs.” It is the separation between the system producing the evidence and the independent layer deciding whether that evidence actually establishes the claimed operational outcome.
And your point about logs versus evidence is particularly important. A log can tell you that an event was recorded; it doesn't necessarily establish that the required state existed, that the expected control held, or that the downstream outcome actually occurred.
That evidence transformation is where I see a potentially very strong architectural boundary between what you're building and what OpsWatch does.
Your runtime can make the underlying execution observable, replayable and resistant to retrospective alteration. OpsWatch can then independently consume the permitted evidence boundary and assess expected versus actual without becoming part of the system being assessed.
So I agree — that's not duplication. It's the assurance loop working properly.
I think there's potentially a very practical opportunity to test that boundary against a real workflow rather than just discussing it conceptually.
If you have a representative workflow where you're already working on turning the raw audit stream into structured evidence, I'd be interested in looking at how that evidence boundary could be defined for independent verification.
No need for a call — we can work through the architecture and evidence requirements here first.
“Governed execution” is a useful way to frame it.
The interesting shift happens when an agent stops producing output and starts changing state in other systems.
At that point, identity, permissions, cost, rollback, and auditability become part of the product behavior, not infrastructure details.
Agreed. Once an agent mutates state, retries can duplicate side effects and timeouts can leave operations half-applied — so rollback and idempotency become product behavior, not ops hygiene.
The open question for us: how far should the runtime enforce rollback when side effects live in someone else's system? Do you lean toward agents declaring their own compensation steps, or expect the runtime to handle it?
I’d lean hybrid. The runtime should own the mechanics: idempotency, checkpoints, retry limits, and durable execution logs.
But compensation for external side effects probably needs to be declared by the workflow, because a generic runtime can’t know whether “undo” means refund, delete, reverse a status, or escalate to a human.
The runtime can then enforce that a compensation path exists before risky actions and orchestrate it when something fails.
Hybrid is where we've landed too. Runtime owning mechanics — idempotency, retries, durable logs — is non-negotiable, but "undo" is semantically loaded: a refund, a status flip, and an escalation are all compensation, and no generic runtime can infer which one the workflow meant.
What that implies for us is interesting: "compensation path must exist before risky action" stops being a best practice and becomes a policy check. If the workflow has to declare its compensation step up front — as part of the action definition, not a comment in the code — then the runtime can refuse to execute a state-changing action that has no declared way back. That turns a soft convention into an enforcement point, which is exactly the kind of thing we want sitting in the runtime layer rather than living in team docs.
The gap we still see in practice: most agent frameworks don't even expose a place to declare compensation, so teams can't express this even when they want to. That feels like the next plumbing problem to solve.
A lot of agent projects focus on smarter reasoning while ignoring what happens when a tool call fails halfway through or a retry repeats the same action. I think AiKey’s next direction should combine short-lived credentials, idempotency controls, and policy simulation before deployment. As a full-stack engineer with agent and MCP experience, I’m very curious about this infrastructure and would love an opportunity to contribute.
You've named the exact failure mode most demos skip — a tool call that dies halfway or retries and repeats a side effect. Short-lived credentials, idempotency controls, and policy simulation are on our roadmap for that reason, and hands-on MCP experience is exactly what we value. Happy to talk more.