Researchers claim a formula for when chatbots turn bad

Researchers at George Washington University have published a paper examining whether it is possible to predict when, and why, an AI chatbot goes rogue, according to a report by SecurityWeek published on 9 October 2026.

The work, as described by SecurityWeek, sets out to answer a question that has so far been handled mostly by trial and error: can the point at which a conversational AI system starts producing harmful or unwanted output be anticipated in advance, rather than discovered after the fact? The paper's authors approach this by trying to express the risk as a formula, SecurityWeek said.

That framing matters for anyone running AI agents in production. Most current defences are reactive. Operators log conversations, apply content filters at the output stage, and investigate after a model has already said or done something it should not have. A predictive model of when a system is likely to drift would, in principle, change the order of those steps. Whether the George Washington University approach delivers that in practice is not something the available reporting establishes.

The SecurityWeek report is a summary, and the specifics of the formula, including which variables it uses, what data it was tested against, and what accuracy the authors claim, are not set out in the material available for this article. Nor is there any indication in the report that the method has been independently reproduced or evaluated by other research groups. Readers should treat the headline claim as a research proposal rather than a validated tool.

There is also a definitional problem that sits underneath work of this kind. "Going rogue" covers a wide range of behaviours, from a chatbot producing offensive text, to a model following instructions hidden in a document it was asked to summarise, to an agent taking an action its operator did not authorise. These are different failure modes with different causes, and a prediction that works for one does not necessarily transfer to another. The available report does not specify which of these the paper covers.

For security teams, the practical takeaway is narrow for now. Academic work on predicting model misbehaviour is worth tracking, but it does not replace the controls organisations already need: restricting what agents are permitted to do, logging their actions, and assuming that any model exposed to untrusted input can be pushed off course. Nothing in the reporting suggests the paper argues otherwise.

The Agentic Times has not reviewed the underlying paper and relies here solely on SecurityWeek's report of it.