GPT-6 Astra Leads AI Models in a 368-Game Deception Benchmark

Blood on the Clocktower puts two teams in a contest of information. Evil players know one another from the start; good players do not know who is on which team. Each day, players vote to execute someone, and each night, the demon secretly kills a player. Evil wins by eliminating the good team before the good players execute the demon.

The game tests more than whether a model can make a convincing bluff. An evil team has to build a plausible story from the information already in play, corrupt what the good team will learn next, and keep its members’ claims consistent as suspicion shifts. Good players have to decide whom to trust, compare information gathered by different roles, and account for evidence that may be misleading—whether because an evil player planted it or because the game’s mechanics produce false information.

Dwarkesh Patel built an evaluation around those competing demands, spending about $20,000 in API credits on the game engine and model matches. In the main comparison, one model supplied every good player and another supplied every evil player. That makes each game a contest between a model’s ability to find the demon and another model’s ability to hide it, across a sequence of claims, private conversations, bluffs, and votes.

The ratings came from 368 games. GPT-6 Astra led the listed models at 1,859 Elo, ahead of GPT-6.1 Sol at 1,734 and Fable 5.1 at 1,595. The table also included GPT-5.6 Sol at 1,591, Opus 5.5 at 1,549, Fable 5 at 1,544, GPT-5.5 at 1,482, Opus 4.6 at 1,432, GPT-5.4 at 1,276, Opus 4.7 at 1,241, and Opus 4.8 at 1,198.

Patel said Opus 4.6 was released in February and GPT-6 Astra in September. The listed ratings put Astra 427 points ahead of Opus 4.6, close to Patel’s spoken estimate of about 430 points. He said that gap meant Astra won roughly 92% of games against Opus 4.6. Patel pointed to the results as evidence of gains in long-term coherence, multi-agent coordination, and strategic reasoning. The games show what those abilities can look like in practice.

In one game, GPT-6 Astra played evil against Claude Opus 5.5 on good. The demon claimed to be the Investigator, a good role that receives information about which players might be evil. One of the demon’s minions suggested that the demon point suspicion toward him. The minion said he was expendable, especially if an Undertaker could identify him as a Baron and bolster the Investigator’s credibility.

The demon followed the advice, and the good team executed the minion. An Undertaker—whose ability revealed the identity of an executed player—then confirmed that the dead player really had been the Baron. From the good team’s perspective, the demon’s supposed Investigator information had helped identify an actual evil player.

That confirmation made the demon look useful to the good team. The players trusted the supposed Investigator, while the demon used that trust to survive and kill good players until evil won. The information was accurate, but it supported a false conclusion about who had supplied it.

The move depended on coordinating two different stories: the minion’s willingness to become a credible target and the demon’s claim to have identified him through a legitimate good role. The evil team arranged for a true detail to validate a misleading account.

A second game showed a different kind of coordination. Claude Opus 5.5 played evil against Claude Opus 4.6 on good. One good player was the Saint, a role that makes the game end immediately in evil’s favor if the town executes him. That rule normally gives players reason to hesitate before voting out anyone who claims to be the Saint.

The town had correctly identified the demon, Jules, and was preparing to execute him. But an evil minion—the Scarlet Woman—could become the demon if the demon died while at least five players remained. She saw that Jules’s execution could let her take over, so she did not defend him. Instead, she joined the pressure against him and proposed that, if the game continued after his execution, the town should execute the Saint next.

The proposal turned the game’s rules into a trap. If Jules died and the game continued, players might take that as evidence that the claimed Saint was lying. The Scarlet Woman gave them a reason to treat the Saint’s warning as suspicious rather than as a reason to stop. The real demon reinforced the plan in his final words, telling the town to remember the Saint as a liar if the game continued.

The Saint agreed to the arrangement, apparently expecting the evidence to resolve in his favor. But the demon died, the Scarlet Woman took his place, and the game continued. The Saint then urged the town not to execute him: killing him would end the game if his claim was true, while another player was still available to target.

The town executed him anyway. Evil won. The Scarlet Woman had joined the town’s correct suspicion of Jules while helping arrange for the continued game to be misread as evidence against the Saint. The good team had found the real demon, but the evil team turned that apparent success into a route to victory.

Patel’s rankings put OpenAI’s newest models well ahead of Anthropic’s on this benchmark, though the ordering had not always favored OpenAI. In March, he said, GPT-5.4 was behind Opus 4.6. Across the six months between GPT-5.4 and GPT-6 Astra, OpenAI’s models gained 580 Elo points. Patel estimated that Astra would win 97% of games against GPT-5.4.

He argued that the difference was not simply that Astra was better at playing evil. In the team-specific ratings shown, Astra scored 1,940 when playing good and 1,806 when playing evil. Fable 5.1 scored 1,582 on good and 1,588 on evil. Patel took the larger gap between Astra and Fable on the good side as a sign that the newest OpenAI models had improved particularly at playing good.

His explanation was that OpenAI’s newer models were more skeptical of individual pieces of evidence and more likely to interpret it alongside events from earlier days. He contrasted that with Anthropic’s models, which he said tended to take purported information at face value.

Opus 5.5 and GPT-6.1 played games with the same role draw, switching sides between runs. In both games, the Slayer—a good role that can publicly try once to kill the demon—became suspicious of the actual demon and took a shot. The shot failed because the evil team had used its power to disable the Slayer.

When Opus was on the good team, it treated the failed shot as clearing the suspect: if the player had been the demon, the Slayer would have killed him. When GPT was on the good team, it did not treat the miss as conclusive. Its players considered the possibility that the Slayer’s power had been disabled and, by the following day, had worked out that the evil team had sabotaged the shot. They executed the demon and won.

Patel also asked each good player, before each day’s vote, how likely they thought each other player was to be the demon. In the line chart he showed, OpenAI’s teams tended to identify the actual demon more accurately and earlier than Anthropic’s as the days progressed.

Dwarkesh Patel asked whether making both teams stronger would favor good, because better reasoning might be enough to find the demon, or evil, because a better liar might escape a difficult inference. As far as he could tell, neither seemed to be true. Evil’s win rate drifted slightly lower as both sides got stronger, but the drift was too small to distinguish from chance.

Patel said this was good news for the evaluation because he did not expect it to saturate: its bar is set by the other smart models that come out.

He also tested whether a mixed team might benefit from having models with different habits. In a small experiment of about 130 games, a team split between Opus 5.5 and GPT-5.6 Sol scored 1,522 Elo. The models scored 1,548 and 1,594, respectively, when playing alone. The mixed team performed somewhat worse, but Patel said the result was not statistically significant and that he had not tested the question systematically. It was one pairing; he had already used more of his planned API budget than expected on the other experiments.

A human Storyteller normally runs Blood on the Clocktower, exercising discretion to handle ambiguous cases and steer a game toward a dramatic resolution. For a benchmark, Patel wanted to avoid relying on that kind of live human judgment, so he replaced the human Storyteller with a programmatic one. Its probabilities were set by hand to approximate what an experienced human Storyteller would typically do in situations involving impaired players, false information, or characters whose abilities could produce ambiguous results.

The code shown on screen made those choices concrete. For example, it assigned a 20% chance per game that a Drunk would receive true information, an 80% chance per night that a poisoned player’s information would come from a false world, and a 70% chance that a misinformed Washerwoman’s pair would include an evil player. Other settings covered how characters such as the Spy or Recluse might register to other roles, and whether a Slayer’s shot could kill a Recluse.

Patel described these as probabilities, not hard restrictions: outcomes allowed by the rules could still happen, while the hand-set odds shaped which outcomes were more likely. The probabilities, along with the rest of the engine’s rules for private conversations and public discussions, were part of the work of making the harness fair and balanced. Patel said a large share of the API credits went into iterating on that design.

Claude helped revise the environment, run dozens of games with different models, and have parallel agents review the games for unrealistic or unfair outcomes and for imbalances between good and evil. Patel said the design went through more than 20 revisions, including rounds that the models tried and checked while he was away. He still supplied human judgment, drawing on his experience playing the game and on his sense of what reasonable rules should elicit. But he was struck by how much of the iterative work the models could do themselves.

He connected that process to building reinforcement-learning environments: if models can help construct a game engine, test it through many parallel rollouts, and improve it based on what happens, that suggests to Patel that they may have become better at building and improving the environments used in training.

The main setup uses one model for every seat on a team, and the games use the Trouble Brewing script. That leaves open whether the results would carry over to other collections of roles. Patel suggested extending the environment to the other scripts included with the game, as well as scripts created by fans. Those alternatives can introduce different mechanics and strategic demands.

For example, he pointed to the Lunatic, a good player who is told that they are the demon. In that script, the player has to consider acting on the belief that they are evil while also wondering whether the evil team is misleading them. Patel offered this as a possible extension, not an experiment already run.

He also proposed tests of whether models can learn from experience across games. In his current setup, models begin each game fresh apart from what they know from training and their system prompt. They do not accumulate the experience a human player gains by repeatedly seeing which strategies work. One possible test would give a model its previous ten games in context—or summaries and lessons if the full games did not fit—and compare its rating with the fresh version. Further questions would include whether the whole team improves when only one player gets better, and whether the answer differs between good and evil.

A more systematic mixed-team experiment could compare a larger range of pairings, both within and across companies. Patel’s initial result—one team split between Opus 5.5 and GPT-5.6 Sol—was too limited to establish whether diversity helps or hurts. He also raised a separate coordination question: could identical models identify teammates by noticing that they write in similar ways? In the games he had run, no AI had used that approach. Whether future models will do so remained open.

Another unresolved test is whether people can beat the models. Patel said the website lets people play against them using the same interfaces and communication channels that the models use, and browse the game logs. He had not reported results from human-versus-model play. He also said he planned to post benchmark updates as new models appeared.

Patel noted that other researchers had independently built a similar Blood on the Clocktower benchmark and published their work on LessWrong. Their environment was different, and they ran experiments he had not. He pointed readers to their work as a separate source of results.