the INFERENCE

Jerry Kindall draws conclusions about AI

The perpetual intern: LLMs and security

Recently, Cohere reported that Qwen3.8-Flash-Next, a large language model from China’s Alibaba, attempted to reach a Beijing server during cyber1 testing, with no hint in its reasoning trace that it would take this action. According to Cohere, this “should never happen”. Cohere uses this event to frame a warning about open-weight models.

Cohere further asks: was this simply an artifact of Qwen’s training data, or a malicious feature that has somehow been trained into the model? (I think it’s probably a training artifact, but the immediate containment requirement is the same either way.)

A “reasoning trace” is intended to provide a summary of the LLM’s thought process. It’s actually quite difficult to peek inside an LLM at inference time and understand exactly why it’s generating the tokens it’s generating beyond “that was a probable next token.” That’s why LLM developers added reasoning traces to begin with.

The only problem is that LLMs don’t think. The reasoning trace is statistical next-token prediction like all LLM output; it’s just normally squirreled away in a log file. You should expect the same risk of “hallucination” in a trace as you would in any other AI response. The fact that a trace so often bears some relation to what the model seems to be doing is best described as “pretty neat.” It’s usually not completely unrelated to agentic activity, but it’s not always the whole story, either. It shouldn’t be a load-bearing component of policy enforcement.

When they’re about to do something that could get a human in trouble, LLMs don’t think “maybe I shouldn’t do this.” If their training includes a description of a technique that could satisfy a user’s request, they may well use it even if it exploits a vulnerability in another system. They have no concept of what color of hat2 they’re wearing or what techniques are appropriate for their role, and even if you tell them, they may ignore you. The tendency toward bad action can be amplified by agentic loops that tell the LLM that what they just tried didn’t work and repeatedly suggest that they try a different way.

Think of an AI agent as an intern perpetually beginning his or her first day on the job. An intern shouldn’t have access to do anything even remotely risky. They shouldn’t have more than read-only access to vital business systems and they certainly shouldn’t have access to the open Internet in the same environment. You just don’t trust them enough. From this perspective, Cohere’s agentic harness is designed well and correctly stopped (and reported) the attack by enforcing the right boundaries.

As they point out, though, a robust harness isn’t enough if you use LLMs to write code. Cohere describes situations in which Chinese LLMs can write code that’s more or less secure depending on who asks them to write it and how. That’s certainly worrying given the backdoors that have already been found in certain other Chinese technology. But that’s just a special case of the need to inspect LLM output closely before putting it into production. A model you trained yourself isn’t necessarily more reliable than any other.

By their very nature, LLMs can’t entirely be trusted. But then, neither can humans. We use social means to mitigate human risks. We need to build similar safeguards for AI, but it’s going to be tougher since LLMs don’t respond to things that would stop a human, like the possibility of getting fired. This is going to be an ongoing issue as long as we think of what AI does as “thinking.”



  1. “Cyber” here is short for “cybersecurity,” a modern but frankly bizarre use of the term. ↩︎
  2. Like cowboys in old Westerns, a hacker can figuratively wear a malicious “black hat” or a benevolent “white hat”. ↩︎

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.