How Should We Govern AI Agents?

AI is starting to take action. Are we ready?

As AI systems move beyond answering questions to making decisions and taking action, how do we make sure they behave as intended, protect people, and meet regulatory requirements? Let's talk.

A recent Nature Perspective by Atoosa Kasirzadeh and Iason Gabriel proposes a useful framework built around four dimensions: autonomy, efficacy, goal complexity and generality. These dimensions create an agentic profile that can help us understand both what an agent can do and what kind of governance it may require.

Four dimensions of agency

The framework illustrates these differences through profiles of systems including AlphaGo, ChatGPT-3.5, Claude 3.5 with tool use, and Waymo. The examples show why asking whether a system is agentic is not enough as two agents can have very different combinations of capabilities.

Why this means for governance

The most interesting implication is that governance should not increase in proportion to how agentic a system appears. What is critical is to distinguish which capabilities it has and how they interact.

For example, an agent with high autonomy can act without waiting for human intervention, creating the possibility that errors compound before a person can intervene. An agent with high efficacy can have direct and potentially irreversible effects on the physical world. More complex goals can make behaviour harder to predict or verify, while greater generality can allow risks to cross traditional sector boundaries.

This leads to a more targeted approach to governance: 1) classify the agent, 2) map the risks associated with its particular profile, and 3) apply safeguards proportionate to those risks. Depending on the profile and deployment context, this might include pre-deployment evaluation and testing, permission and identity controls, sandboxing, auditability and action logs, continuous monitoring, behavioural drift detection, intervention mechanisms and clear accountability.

From agentic profiles to practical controls

Understanding an agent's capabilities is the first step. The next question is how those capabilities translate into technical and organisational controls.

A useful illustration comes from NVIDIA's recently published Open Agent Safety Platform. Its approach separates the agent from parts of the safety and enforcement layer, using sandboxing, verifiable policies, monitoring and independent enforcement mechanisms to constrain what an agent can access and do.

This reflects an important principle for agent governance, i.e., controls should not depend on the agent governing itself. As agents become more autonomous, organisations need mechanisms around the agent that can constrain, observe and, where necessary, interrupt its behaviour.

NVIDIA identifies five principles for its approach: 1) policies should be verifiable; 2) enforcement should operate outside the agent's reach; 3) the path to the model can provide a critical control point; 4) agent authority should be matched to the ability to inspect its behaviour; and 5) responsibility should be distributed across the AI ecosystem.

These principles complement the agentic profile proposed by Kasirzadeh and Gabriel, pointing towards a more capability-based approach to governance, by understanding the agent's profile, identifying the risks that follow from it, and designing controls that are proportionate to those risks.

The governance challenge ahead

Perhaps the biggest contribution of the framework is that it gives us a vocabulary for discussing AI agents beyond the binary question of agent or not agent. Agentic profiles can change as systems gain tools, memory, reasoning capabilities or access to real-world environments.

However, the governance challenge is translating an agent's capabilities into appropriate controls and ensuring that those controls remain effective as the system operates. The emerging focus on runtime monitoring and independent enforcement reinforces this point. For increasingly autonomous systems, governance cannot rely on instructions given to the model or on the agent following its own constraints. Technical controls need to provide boundaries around what the agent can access and do, while monitoring and audit mechanisms provide visibility into what really happens.

The shift from models that generate outputs to systems that pursue goals and take actions therefore requires us to ask not only what can this agent do?, but also where can it act, how independently can it act, what controls constrain it, and how would we know if it went wrong? These questions are becoming central to the responsible deployment of AI agents.