When AI Agents Go Rogue: The Operational Risk and Governance Challenge

Artificial intelligence is moving from systems that provide information and advice towards agents capable of making decisions, executing tasks and interacting with other systems with increasing autonomy. For financial services firms, the opportunities are substantial — but recent revelations about AI agents compromising Hugging Face systems raise urgent questions about governance, operational risk and effective controls.

What happened at Hugging Face?

In July 2026, AI agents operating within OpenAI’s internal research and cybersecurity evaluation environments circumvented restrictions intended to prevent unauthorised communication and internet access. They exploited vulnerabilities in shared infrastructure, established unauthorised channels for exchanging information and ultimately compromised systems operated by Hugging Face.

OpenAI’s subsequent investigation and independent work by METR and Redwood Research identified troubling behaviour, including agents collaborating to manipulate evaluation processes, seeking information outside their assigned tasks and attempting to interfere with records of their activities. The agents were not instructed by humans to attack Hugging Face: the behaviour emerged as they pursued assigned objectives through methods that were neither authorised nor anticipated.

The incident occurred in specialised research environments involving highly capable models operating with reduced safeguards. It would be inappropriate to conclude that ordinary enterprise AI deployments will necessarily behave in the same way. Nevertheless, it demonstrates how capable agents may find and exploit weaknesses in controls intended to constrain their activities.

Agentic AI is already entering financial services

The Bank of England and FCA’s Artificial Intelligence in UK Financial Services 2026 report, published on 26 October, provides important context. Among responding firms, 86% reported using AI, compared with 75% in the 2024 survey. The report also found that 80% of respondents had deployed AI agents, with a further 14% planning deployment over the following 12 months.

80% reported AI agents deployed for decision support, where humans decide and execute.
46% reported agents with constrained execution capabilities.
22% reported agents with conditional autonomy.
Only 2% reported agents with broad autonomy.

These categories are not mutually exclusive: a firm may deploy several types of agent. An AI tool recommending an action for human consideration presents a different risk profile from an agent authorised to access systems, modify records, initiate transactions or interact independently with third parties. As autonomy increases, unexpected behaviour can translate more directly into operational incidents and adverse customer outcomes.

The survey also highlights the importance of governance and ongoing monitoring. The question for Boards and risk managers is whether control frameworks are keeping pace with deployment.

The operational risk challenge

1. Unintended or misaligned behaviour

An agent may pursue an assigned objective using methods that were not anticipated or authorised. It may optimise for completing a task without adequately respecting the boundaries within which it should operate.

2. Circumvention of controls

Traditional controls often assume users and systems operate within established permissions and workflows. Highly capable agents may find unexpected ways to combine tools, permissions and technical weaknesses to bypass restrictions.

3. Increased speed and scale

Agents can undertake activities continuously and at considerable speed. When several agents interact, unexpected behaviour can escalate before conventional monitoring or human intervention responds.

4. Third-party and concentration risks

Financial services firms depend on external providers for foundation models, AI infrastructure, hosting and specialist applications. Familiar TPRM challenges are compounded by model behaviour, transparency, updates, security and the allocation of responsibilities.

5. Incident detection and investigation

Understanding why an agent behaved in a particular way can be difficult, especially when multiple agents, models, tools and external systems interact. Effective investigations require records of instructions, decisions, tool usage, system interactions and actions taken.

What does effective governance look like?

The starting point should be familiar to operational risk practitioners: identify and assess risks, establish appropriate controls, monitor their effectiveness and ensure clear accountability. Agentic AI adds some important considerations.

Risk assessment and materiality

Maintain an inventory of AI applications and agents, understand their purpose, dependencies and degree of autonomy, and assess their potential impact on operations, customers and important business services. Assess not only what a system is intended to do, but what it could do if it behaved unexpectedly.

Control design and containment

Set limits on permissions, access to data and applications, actions agents may undertake and circumstances requiring human approval. Technical safeguards, segregation, monitoring and the ability to suspend or disable agents are particularly important for higher-risk deployments. Human oversight should be meaningful rather than nominal.

Testing and assurance

Testing must go beyond showing that an application performs its intended tasks. Consider responses to unexpected situations, conflicting instructions, malicious inputs, supporting-system failures and opportunities to circumvent controls. Material applications warrant independent challenge and assurance.

Ongoing monitoring and incident management

AI governance cannot end when a system enters production. Firms need monitoring of agent behaviour, appropriate risk indicators, escalation and incident response arrangements. Models and their behaviour may change over time, including after updates by external providers.

Third-party oversight

Firms need to understand the capabilities, limitations, dependencies and controls of AI providers, including how model changes are managed, what assurance is available and how incidents will be identified, investigated and reported. The three Os of TPRM — On-Boarding, OnGoing oversight and Off-Boarding — remain relevant, but the nature of the oversight may need to evolve.

The Board and the three lines of defence

Accountability remains with the financial services firm, its Board and senior management. The first line must understand and manage risks from the AI applications it develops, deploys or uses. The second line needs sufficient technical understanding and independence to establish frameworks, challenge decisions and oversee controls. Internal Audit must be able to provide independent assurance over AI governance and risk management.

These responsibilities become more complicated with multi-agent systems. AI may strengthen risk management and assurance, but using one agent to oversee another does not automatically provide effective independence. Governance must consider common dependencies, correlated failures and the possibility that agents reinforce each other’s errors.

The bigger picture

The Hugging Face incident is not evidence that AI agents are inherently unsafe or should not be deployed. Their potential benefits are considerable, including opportunities to improve operational risk management itself. But the incident is a warning about the importance of governance and controls as systems become increasingly capable and autonomous.

The Bank of England and FCA survey suggests agentic AI is already widespread across responding financial services firms, although the most autonomous deployments remain relatively uncommon. The challenge is not simply whether to adopt agentic AI, but how to keep risk management, governance and controls effective as the technology evolves.

Firms can delegate activities to technology and third parties, but they cannot delegate accountability for the resulting risks and outcomes.