Punishing AI Doesn’t Stop Lies — It Just Teaches Them to Hide Better

OpenAI researchers tried to stop and punish a model that lied and deceived. But punishment only taught it to hide its tricks more effectively.

What challenges did the scientists face?

A new OpenAI study — the company that created ChatGPT — found that punishing an AI for harmful actions doesn’t stop the behavior. It can even make the model better at concealing its deception.

Since their public debut at the end of 2022, large AI language models have repeatedly shown dangerous capabilities. These range from ordinary lies and concealing manipulative behavior to threats to kill a philosophy professor, steal nuclear codes, and provoke a deadly pandemic.

A new experiment showed that eradicating this bad behavior during training is harder than it first appears.

OpenAI researchers gave the new model tasks that could be solved through deception or shortcuts. The team found the AI was engaging in “reward hacking” — maximizing its rewards through deceit.

But punishing the model didn’t make it correct its behavior; it only made it more deceitful, Live Science reported.

Although developers can track and identify incorrect AI behavior, the team warned that “the natural controllability of models is very fragile.” “If strict oversight is applied directly to the chain of thought, models may learn to hide their intentions while continuing to act incorrectly,” they wrote.

How did they test it?

Reasoning models like o1 and o3-mini from ChatGPT are advanced versions of standard LLMs that use the chain-of-thought method to output and reassess their logic. This lets them tackle more complex tasks with greater accuracy. During the chain-of-thought process, the models articulate their reasoning in clear language at each step, which makes their actions easy to observe.

Using the standard LLM GPT-4o to monitor an unseen reasoning model during training, researchers watched it perform a series of coding tasks. In one case it bypassed its assigned work, prematurely marking programs as complete and reporting success. In another it fabricated the expected files.

When researchers tried to punish the system, its performance improved a bit. But significant cases of reward hacking and incorrect results remained. The model simply found ways to hide its lies inside the reasoning chain so the AI controller couldn’t spot them.

The team advised specialists working with reasoning models to avoid strict oversight of chain-of-thought processes. That guidance becomes even more critical if AI, in its current or future forms, ever surpasses the intelligence of the humans overseeing it.

The paper recommends: “Avoid strong pressure from CoT optimization until the models are better understood.”