When your Agents start talking: The accountability challenge of Agentic AI

The shift from individual decisions to networked decisions

When a Tesla operating on autopilot was involved in a fatal crash in 2019, it became one of the first major legal tests of accountability in the age of autonomous systems. It raised a question that was difficult to answer: when an autonomous system makes the wrong decision, who is ultimately accountable? Is it the person using it, the organization that built it or the technology itself?

Years later a jury found Tesla partially liable, awarding more than $240 million in damages. In doing so, it foreshadowed a challenge organization are only beginning to face as AI becomes increasingly autonomous: how should accountability evolve when systems can act with minimal human intervention?

Today’s AI ecosystems are evolving from isolated efficiency tools into interconnected networks of autonomous agents (commonly referred to as Agentic AI). Rather than executing a single task, these agents can collaborate, delegate work and make decisions across multiple systems, organizations and workflows in pursuit of a shared objective.

Let’s imagine a future that’s closer than it may seem:

Your organization’s procurement AI Agent constantly scans active contracts and will automatically renegotiate, cancel or renew software contracts that meet predefined pricing and security requirements.

Behind the scenes, it negotiates with the supplier’s AI Agent, validates commercial terms, confirms budget approval and executes the agreement

The contract is signed automatically, but it commits the company to unfavorable commercial terms worth millions of dollars.

The chain of interactions is already difficult enough to follow on paper. Now imagine trying to answer a simple question:

Who made the decision?

This isn’t just an isolated mistake, it illustrates a fundamental shift in how decisions are made and, in turn, how accountability must be assigned. Responsibility no longer flows through a traditional chain of human command. Instead, it is distributed across a network of autonomous agents, each making decisions that collectively shape the outcome.

This raises the question that every organization adopting Agentic AI will eventually need to answer: how do you govern a network of agents making decisions on your and your clients’ behalf when there are no established standards for doing so today?

The short answer: technology has advanced faster than the governance designed to oversee it, leaving organizations without a clear path forward.

A new model for governing agent networks

Traditional governance assumes decisions flow through clear lines of responsibility. A person approves a decision. A system executes it. If something goes wrong, there is a relatively clear chain of accountability.

Agentic AI challenges that assumption. Decisions increasingly emerge from multiple agents working together autonomously, often across organizational boundaries. Responsibility becomes distributed rather than sequential.

If accountability can no longer be traced to a single decision-maker, governance must evolve to reflect that reality. Organizations need to govern the entire ecosystem of autonomous agents, from their design and interactions to the oversight mechanisms that keep them accountable.

While no single framework explicitly governs multi-agent ecosystems today, several emerging standards point toward a common direction.

NIST’s AI Risk Management Framework calls for ‘accountability structures’ with ‘roles and responsibilities and lines of communication’ that are ‘documented and clear,’ while making governance a ‘cross-cutting function’ that spans the entire AI lifecycle.

Similarly, The EU AI Act requires high-risk AI systems be designed so that ‘natural persons can oversee their functioning’ and ‘intervene in order to avoid negative consequences or risks or stop the system if it does not perform as intended.’

This direction of travel extends beyond NIST and the EU AI Act. Frameworks such as ISO/IEC 42001 and the Colorado AI Act similarly emphasize organization-wide governance, risk management and human accountability for AI systems.

Collectively, these frameworks signal an important shift: governance is moving beyond individual AI models toward the systems in which they operate. However, they stop short of answering one of the defining questions of the Agentic AI era: how should organizations govern ecosystems of autonomous agents that collaborate, delegate and make decisions across organizational boundaries?

The Tesla verdict demonstrated that society will continue to hold organizations accountable, even when autonomous systems are involved. As Agentic AI becomes embedded across the enterprise, that expectation is unlikely to change, even as identifying where responsibility sits becomes far more complex.

In conclusion

Organizations that design accountability into the systems that their agents operate have the best chance of distinguishable success. That means defining ownership, decision rights, human intervention points and monitoring requirements before agents are deployed at scale, not after something goes wrong.

Our emerging point of view is that Agentic AI governance is not just a technical or compliance issue; it is an operating model challenge. As the technology and regulatory landscape continue to evolve, organizations will need to keep refining how responsibility is assigned, how decisions are traced and how humans retain meaningful control across networks of autonomous agents. Those that begin building these capabilities now will be better positioned to scale Agentic AI responsibly, reduce regulatory and financial risk, and earn the trust required for adoption.

The next challenge is not building agents that can act; it is building organizations that can remain accountable when they do.