People Overestimate Their Ability To Catch Cheaters, Chess Grandmaster And Researcher Reveals

People are not much better than a coin flip at guessing if their opponent is cheating in a chess game. That's what grandmaster and economics researcher David Smerdon concluded after running two experiments to test people's ability to spot cheaters. 

The revelation was revealed in a recent episode of the C-Squared podcast, hosted by GMs Fabiano Caruana and Cristian Chirila. You can watch the full episode below or keep reading for a summary:

Smerdon is an assistant professor of economics at the University of Queensland in Australia. In his own words, part of his research work is dedicated to "help the chess world in some ways." Smerdon, who has a few cheating-related projects, mainly spoke about two of his experiments.

The "Cheating Challenge"

The first is what he called the "Cheating Challenge," an eight-player double round-robin tournament with players of varying strength (1600 to 2300 Elo). Before each round, Smerdon would tell certain players they'd receive engine suggestions at critical moments of the game, elected by Smerdon. 

At the end of the tournament, players could spend the points they earned to accuse other players of cheating. A successful accusation would grant them an extra point, while they'd lose the point invested if they were wrong. Being caught cheating would cost them even more points. According to Smerdon, this setting would encourage players to try to catch cheaters and to cheat without being caught. 

This first experiment led to intriguing conclusions: "In terms of the accusation success rate, it was 69% correct," Smerdon said. "50% would be chance, so it's better than chance but not a whole lot better."

In terms of the accusation success rate, it was 69% correct. 50% would be chance, so it's better than chance but not a whole lot better.
— GM David Smerdon

However, this was not his only interesting finding. The experiment also showed how players are affected by cheating (both the cheater and the victim) and what factors drive suspicion. 

Unsurprisingly, players who thought they had faced a cheater performed worse not only during the game but throughout the rest of the tournament. Most reported a feeling of inevitability and powerlessness that spilled over to subsequent games.

A striking discovery, though, was that cheaters also performed worse on their own when they knew they'd be the ones using engine assistance. "The increase in the quality of the games was coming from the cheating moves, but for the rest of the time, they were just so distracted (...) and also feeling a bit guilty."

This finding puts in question the notion that players could gain a considerable amount of rating points by getting just a few suggestions per game. Smerdon mentioned this idea makes sense if everything else remains equal, but in practice, playing while cheating is not the same as playing clean. 

"That's a ceteris parabus thing, we say, in economics. If everything else is the same (...) then this will happen. The problem is that in a game where you know you're getting moves, not everything's the same because you're not playing your usual strength," Smerdon said. "Maybe cheaters get better at it over time, but I think it's still very difficult to play at your true level knowing that you're cheating and knowing that these moves are coming at some point."

Smerdon's "Cheating Challenge" also highlighted some interesting insights regarding what drives cheating suspicion. Surprisingly, his study showed that players' rating gap and the number of engine moves played by the cheater did not make players more suspicious. 

Instead, the most significant determining factor was how poorly the victim had played. Players who made the most mistakes were more likely to accuse their opponent of cheating, whether they were, in fact, cheating or not. 

"The number one thing that predicted whether you accuse [your opponent of cheating] is if you make more blunders in a game," said Smerdon.

The number one thing that predicted whether you accuse [your opponent of cheating] is if you make more blunders in a game
— GM David Smerdon

The researcher offered his own theory as to why this happens: "My strongest suspicion is that it's kind of an internal way of putting the blame on someone else for your own bad play. (...) I don't want to accept that it's my fault, I know there's a chance my opponent's cheating, I subconsciously, maybe, increase my suspicion."

Can You Catch A Cheater Test

Smerdon also ran another experiment, likely the largest experiment about chess cheating ever conducted in the world. This time, success rates for cheating accusations were even worse: an average of around 54 percent.

The test consisted of a questionnaire containing seven different games. Test-takers could see all the games in full and see the engine evaluation after each move. They could also analyze the game using other engines. They then had to say whether or not one of the players was cheating.

More than 4,000 people took the test and verified their rating (by FIDE ID, Chess.com username, etc). With a much larger sample size, the results were significantly worse.

Smerdon recognizes that this particular test was significantly more difficult: "It's a really difficult assignment because you're given seven independent games, it's not like you're following player X the whole way through."

Still, the results shocked Smerdon. "With a sample that big, you'd still think that you could do better than chance on average," he said. "We really suck at this. We all really suck at this."

We really suck at this. We all really suck at this.
— GM David Smerdon

According to the results, stronger players do have a better chance of catching cheaters. However, chess experience, defined by how long a player has been playing chess, is what makes the most difference. Still, even players with more than 10 years of experience only had a 58% success rate. Caruana, who took the test, scored 3 out of 7.

"I think it's just a very hard task," Smerdon said about correctly assessing whether another player is cheating. "I think it's much harder than people recognize. And there are a few players, very strong players, who overestimate their ability to a significant degree."

You can check Smerdon's slides for the presentation he just delivered at the Saint Louis Chess Club on chess cheating here.