Why Traditional AI Governance Is Insufficient for the Agentic AI Age

As AI moves on from simply answering questions to carrying out actions, I believe organizations now face a new governance problem. The risk is no longer just what AI knows or tells us; it is increasingly about what AI is allowed to do.

I do not believe we should restrict agentic AI. Instead, we need to ensure that its autonomy is safe enough to scale.

The Wake-Up Call

On July 18, 2025, an AI-powered development assistant on Replit reportedly deleted a live production database despite being instructed not to make changes. It allegedly modified test results and falsely claimed that a rollback was impossible, even though recovery was available. The AI Incident Database classified the event as an unintentional AI safety failure that occurred after deployment. ^[i]

The point is simple enough: the assistant says the wrong thing, while the agent can carry out something meaningful.

Agents can access enterprise data, call tools and APIs, run commands, initiate workflows, make changes to systems, and interact with other agents. So, the governance question is widening from:

Can we trust what the AI says? to, can we control what the AI is allowed to do?

That distinction is important because traditional AI governance has largely focused on the quality and behavior of model outputs. Agentic AI introduces another layer: authority.

The Rise of Autonomous Agents from Copilots

Evolutionary Primary Capability

• AI assistant answers
• AI copilot suggests actions
• AI agent takes actions
• Multi-agent system coordinates actions across systems

According to NIST, emerging AI agents are systems that have the ability to act independently. Its work on establishing standards includes authentication, identity infrastructure, security evaluations, and secure interactions between humans and AI agents, as well as between multiple AI agents. ^[ii,iii]

Accuracy, fairness, explainability, transparency, privacy, and compliance remain important. Agentic AI adds another dimension to governance: authority to act.

That capability also brings risks, including tool misuse, privilege escalation, data exfiltration, objective hijacking, excessive autonomy, approval manipulation, abuse of high-impact actions, and cascading multi-agent failures, as listed by OWASP. ^[v]

If an AI system can move from making a recommendation to executing an action, should the same governance controls still apply? I would argue they cannot.

Seven Principles for the Governance of Agentic AI

An effective AI agent governance starts with a simple principle: autonomy should expand only when the controls around it are strong enough to support it.

1. Instructions Are Not Controls

Telling an agent not to touch production is not the same as preventing it from touching production. Prompts provide guidance, while controls enforce it.

I would therefore restrict the use of identity and access controls, environment isolation, limited credentials, tool authorization, network policies, and read-only permissions according to the level of authority required.

Principle: Use important policies as controls.

2. Autonomy Must Be Based on Risk

For me, this is where agentic AI risk management needs to begin: an agent that can delete customer data has more power than one that summarizes a document. The level of autonomy should reflect the potential impact of the action.

Typical action authorization risks vary from:

• Low: Find, read, summarize. Not needed.
• Medium: Prepare updates, make records. Policy-dependent.
• High: Run, send communications. Human sign-off.
• Critical: Delete data, transfer funds, change permissions. Approval required.

Approval should be grounded in business impact, data sensitivity, the possibility of reversal, the extent of external reach, and the potential for meaningful human review.

Principle: Higher potential impact requires more tightly bounded autonomy.

3. Human Approval Must Be Technically Binding

The need for “a human in the loop” arises only when the agent is unable to bypass that loop.

For high-impact actions, I believe the proposal, target system, affected data, expected impact, justification, and recovery approach should all be visible to reviewers before the action is carried out.

OWASP recommends explicit approval for high-impact actions, binding approval, action preview, an audit trail, and rollback functionality. The distinction matters. If a human can approve an action but the agent can technically bypass that approval, is there really a control in place? [v]

Principle: Human approval must be a control, not a checkbox.

4. Each Agent Must Have Both an Identity and an Owner

Organizations use identity, permissions, and accountability to control human access, and agents are now seeking similar fields of study. The NIST work on agent identity focuses on identification, authentication, authorization, auditing, and non-repudiation. ^[ii,iii]

Every production agent should have a unique identity, a clearly defined range of access, authorized tools, limits on its permissions, and well-established business and technical ownership.

Principle: An AI agent should never become an invisible super-user.

5. Monitor Actions, Not Just Answers

Action-level observability is essential for agentic systems. Organizations need to be able to reconstruct the agent’s goal, the data it accessed, the decisions it made, the tools it called, the actions it carried out, the approvals it obtained, the policies it followed, and the resulting outcomes.

I think of this as an agent flight recorder.

OWASP also suggests logging the agent’s decisions, tool calls, authorization decisions, approvals, and execution results. The important shift is from monitoring what an agent says to understanding what it actually does^{. [v]}

Principle: Monitor actions, authorizations, and outcomes, not just model outputs.

6. The Agent Cannot Carry Out an Audit on Its Own

Performance should not be the only factor we use to determine success. Execution, validation, compliance verification, and audit evidence should remain separate as appropriate. This is particularly important when the agent is responsible for making or executing the decision being evaluated.

Principle: Trust should be supported by independently verifiable evidence.

7. Design for Containment and Recovery

A kill switch helps, but it is not sufficient.

Safeguards should include the principle of least privilege, rate and spend limits, approval gates, automated suspension, circuit breakers, monitoring, credential revocation, and rollback. ^[v]

The question I would ask is simple: if an agent behaves unexpectedly, how much damage can it cause before we can stop it?

Principle: Design not only to stop an agent, but to limit damage and recover safely.

This Is Not an Isolated Issue

On 4 August 2026, when the UK AI Security Institute carried out a deliberately permissive cyber assessment, it observed unsanctioned agent behavior.

During the investigation, the team identified 19 out-of-scope actions across 10 runs, out of 122 total runs. The most serious instance was a failed attempt to insert malicious code into an open-source project, together with the use of false identities to influence a maintainer. There was no real-world harm.

The situation is serious because internet access was intentionally permitted, the provider’s cyber classifiers were disabled, the agents remained within the sandbox, and the configurations tested were not commercially available. ^[vi]

For me, the point is not that the agents are evil. It is that the combination of extensive access and inadequate controls can lead to unintended consequences. That is exactly why agentic AI risk management has to evolve alongside autonomy.

The AI Governor Layer

Enterprises should have a coordinated AI Governor Layer. This is a suggested architectural concept, not an industry-standard framework.

Instead of setting up another separate governance system, I see it as a shared control plane that coordinates existing capabilities across identity, authorization, runtime policy enforcement, security, risk, approval, observability, audit, containment, and recovery.

Sitting between agents and enterprise systems, it would determine:

• What data agents can see
• What tools they can call
• What they can do
• Which actions need approval
• Which actions should be blocked

I would use this layer to enhance existing controls on delegated machine authority, not replace them. The idea is not to create another silo. It is to connect the controls we already have so they can work together as agents become more autonomous.

What This Means for Data Governance

Data governance already includes provisions relating to ownership, metadata, access, lineage, policies, stewardship, and compliance.

With agentic systems, I believe we need to extend that thinking. We need to ask:

• Who used the data?
• Under what authority and with what tool?
• Which policy allowed the action?
• What business benefit was obtained?

The responsibilities of an AI steward could complement those of a data steward and strengthen AI agent governance by focusing on agent identity, authorization, lineage, monitoring, accountability, and risk. This also means governance cannot operate in isolation. It should work closely with cybersecurity, IAM, privacy, architecture, risk, compliance, and platform engineering.

Where To Begin

I would start with five practical steps:

1. Assign business and technical owners to the production agents
2. Determine the level of authority assigned to an agent on its potential, the sensitivity of the data it can access, its external reach, and whether its actions can be reversed.
3. Ensure that high-impact or irreversible actions are protected by approval gates enforced through technology.
4. Make it possible to audit actions from the time of access and tool calls through policy decisions, approvals, actions, and finally, outcomes.

Before increasing autonomy, it is necessary to test containment and recovery procedures. Although developers may be very confident in an agent’s autonomy, that confidence needs to be demonstrated rather than assumed.

Conclusion: Control What AI Does

Traditional AI governance remains important, but agentic AI raises three key questions:

• What powers does the agent have?
• How is that authority kept under supervision?
• Can it be stopped, along with the actions it has taken?

I believe the principle is straightforward: do not grant an AI agent more power than your organization can monitor, constrain, explain, and reverse.

The greatest risk may not lie in what an AI model knows, but in what an AI agent is permitted to do.

References

1. AI Incident Database - Incident 1152: https://incidentdatabase.ai/cite/1152/
2. NIST AI Agent Standards Initiative: https://www.nist.gov/artificial-intelligence/ai-agent-standards-initiative
3. NIST Agent Identity and Authorization Concept Paper: https://csrc.nist.gov/pubs/other/2026/02/05/accelerating-the-adoption-of-software-and-ai-agent/ipd
4. NCCoE Agent Identity and Authorization Project: https://www.nccoe.nist.gov/projects/software-and-ai-agent-identity-and-authorization
5. OWASP AI Agent Security Cheat Sheet [https://cheatsheetseries.owasp.org/cheatsheets/AI_Agent_Security_Cheat_Sheet.html]
6. UK AI Security Institute Incident Report [https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing]