Because doubters appear here, i have decided to make things clear once and forever.
The root of inspiration is methodology used by RedHotPawn game mods, reintroduced by Steve Collyer to the public in 2009 in the infamous Ouachita's case at Chess.com.
Original method uses any modern chess engine (e.g. Fritz10, Fritz11, Rybka etc) running under fixed time condition, usually 30 seconds, showing top choices. Manually feasible with stopwatch in almost any UCI. RHP game mods used this to support their suspicion of engine smell with something more robust than just some uncertain feelings. They analyzed many top OTB and especially pre-computer era correspondence games to obtain values for honest play. After testing many games with many engines, they made an estimation of top amateur human correspondence honest play as top1 max. 60%, top2 max. 75% and top3 max. 85%. To avoid false positives, they added +5% and declared that everyone above 65%, 80% and/or 90% on reasonable big sample (at least 400 moves, recommended 20 games with 20 non-book moves each against solid opposition) is blatant cheater. They created program called Batch Analyzer to perform such analysis automatically.
Well inspired, i downloaded another analyzer
here, paid for full version (weird, isn't it?) and went my own way. The main difference is that i decided to use fixed depth of engine Firebird instead of fixed time.
I analyzed XI. WCCh (1983 - 1988, winner Friedrich Baumbach, 15 players) to obtain benchmarks. I should mention, the main parameter to be watched is Top3.
Obtained results:
Top3 ... 3809 of 4587 moves => 83.04 %
Theoretical calculated binomial standard deviation of the whole sample ... 0.55 %
Calculated residual individual standard deviation by players ... 0.92 % (pythagorean difference between individual total and individual binomial standard deviation)
Total standard deviation of suspect is pythagorean sum of three parts:
constant 0.55 % (binomial benchmark)
constant 0.92 % (residual, considering minor inadequacy of binomial model and differences in playing style)
player's sample binomial (inverse-proportional to square-root of number of analyzed moves)
Forced and obvious moves are used in both, benchmark sample and analyzed sample of suspect. There is no reason to assume that in online chess are these moves more frequent than in ancient correspondence and can create significant bias. Playing styles prefering more forced moves and inadequacy of binomial model are considered in the pythagorean sum of deviations (see above).
The reasonable cut-off line is +2.5 total standard deviations, which is about 99th percentile in the case the player in question is pre-computer correspondence chess grandmaster (most of them obviously aren't and even Docx and Dembo should be considered as much weaker correspondence players than XI. ICCF finalists).
The minimum total standard deviation is 1.07 % for sample with infinite number of moves.
Thus players with Top3 under 85.72 % are considered as undetectable with this method. :o)
The first possible source of bias is opening theory checked & created with computer assistance nowadays. Non-book moves are cut, but so are first 3 nonbook moves to avoid bias from pre-game preparation with computer and private knowledge of a player in his/her pet opening, checked by computer.
Another possible source of bias is the fact that online players today play very often until checkmated, while ancient correspondence players always resigned, because postal cards were somewhat expensive, especially in international competitions. Thus the game is considered determined, when the evaluation exceeds 3 pawn units and rest is cut. However even after these two cuts, there must remain at least 17 moves, shorter games are either show-draws or low quality opposition.
Edit 29th April: Graph of benchmark data, 15 players Top3 statistics. Note: number of moves they played is signicantly lower than number of moves played by suspects, thus the variation is higher.
