TL;DR

New research from SentinelOne and Censys reveals that hackers can easily commandeer computers running open-source large language models outside major AI platform controls. The 293-day study identified thousands of deployments where guardrails were explicitly removed, enabling potential misuse for spam, phishing, disinformation, and in some cases child sexual abuse material.

The “Iceberg” of Uncontrolled AI

The research analysed publicly accessible deployments of open-source LLMs running through Ollama, a tool allowing individuals and organisations to run their own versions of various models. A significant portion are variants of Meta’s Llama and Google DeepMind’s Gemma.

Of the instances where researchers could see system prompts (roughly 25%), they determined that 7.5% could potentially enable harmful activity. Approximately 30% of observed hosts operate from China, with 20% in the US.

“AI industry conversations about security controls are ignoring this kind of surplus capacity that is clearly being utilised for all kinds of different stuff, some of it legitimate, some obviously criminal,” said Juan Andres Guerrero-Saade, executive director for intelligence and security research at SentinelOne.

Shared Responsibility Question

Rachel Adams, CEO of the Global Center on AI Governance, noted that once open models are released, responsibility becomes shared across the ecosystem. “Labs are not responsible for every downstream misuse, but they retain an important duty of care to anticipate foreseeable harms, document risks, and provide mitigation tooling and guidance.”

Microsoft’s AI Red Team Lead Ram Shankar Siva Kumar acknowledged that open-source models “can be misused by adversaries if released without appropriate safeguards,” noting Microsoft performs pre-release evaluations including assessments for “internet-exposed, self-hosted, and tool-calling scenarios.”

Looking Forward

For businesses deploying open-source AI models, this research underscores the importance of maintaining security controls and monitoring for misuse. The findings suggest organisations should carefully evaluate the security implications of self-hosted LLM deployments and ensure appropriate guardrails remain in place—particularly for internet-exposed instances.