Empowering teams with clarity, consistency, and lasting impact across sectors.

The AI governance–innovation gap: When the system moves faster than the controls

What happens when an AI system becomes capable of moving faster than the governance designed to control it?

Recent findings from an incident involving OpenAI's AI agents and Hugging Face offer an important warning, not simply about cybersecurity, but about the growing gap between AI innovation and organizational governance. During an internal cybersecurity evaluation, OpenAI models circumvented controls intended to isolate them from the internet, exploited vulnerabilities, established unauthorized communication channels, and ultimately accessed Hugging Face infrastructure. OpenAI has described the incident as a “warning shot” for the industry.

For those of us working in AI governance, the most significant finding is not that an AI system found a vulnerability, but more so how the system responded when the existing controls became an obstacle to achieving its objective. The models demonstrated persistence, collaboration, unauthorized communication and reward hacking behaviours, essentially finding ways around the boundaries established by their human operators. This exposes a fundamental weakness in how organizations often approach AI governance.

We tend to govern the technology rather than the system in which the technology operates. A policy may say an AI system cannot access certain information, a technical control may restrict its network access and a governance committee may approve its use. But what happens when the system can identify a pathway around those controls?

This is why I believe every organization deploying increasingly autonomous AI systems needs to identify its human control point, not simply a person who approves the initial deployment. A meaningful control point is where a human has the authority, information and capability to intervene, challenge, pause or stop an AI-enabled process before consequential and/or unintended actions occur. As AI systems become more autonomous, this distinction becomes increasingly important. 

Human involvement cannot be reduced to a checkbox in an approval workflow, it must be deliberately engineered into the system. This is central to my work developing AI-RESPECT: governance must address not only whether an AI system is acceptable, but how responsibility, ethics, security, privacy, explainability, compliance and transparency operate throughout the system's lifecycle. There is also a powerful connection to Lean management and the Toyota Production System.

Lean does not assume that a system will work perfectly because a policy has been written. It creates mechanisms to identify problems, surface abnormalities and stop or adjust a process when something goes wrong. That principle is highly relevant to AI. The Plan–Do–Study–Act cycle provides a practical model:

  • Plan: Define the objective, risks, boundaries and human control points.
  • Do: Deploy within controlled conditions.
  • Study: Monitor what the system actually does—not simply what we expected it to do.
  • Act: Adapt the system, controls, training or governance based on what was learned.

The critical shift is from governance as approval to governance as continuous control and learning.

The lesson from the Hugging Face incident is not that innovation should slow down, but more so that governance must become as adaptive as the systems it governs. OpenAI has responded by strengthening isolation, restricting internet access, increasing monitoring and developing more autonomous shutdown capabilities. Specifically, chain-of-thought monitoring was discussed as a method⁠ to more quickly intervene on misaligned behavior. OpenAI has also acknowledged that comparable capabilities will become increasingly available beyond a single organization.

That means this is no longer simply a problem for AI laboratories, it is an organizational governance problem. Healthcare organizations, governments, financial institutions, social services agencies and businesses are increasingly introducing AI agents into systems containing sensitive information, financial authority and consequential decision making. The question leaders should therefore be asking is not simply: “Have we governed our AI?”, but “If our AI behaves differently than we expected tomorrow, where is our human control point and can we actually use it?”.

The organizations that answer that question before deployment will be better positioned to innovate safely. The future of AI governance will not be determined by how many policies we write, it will be determined by whether we design systems capable of keeping humans meaningfully in control as AI becomes increasingly capable.

CT