Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka: Best move is not the best move?
- - By Tapani [tw] Date 2013-10-09 06:16 Edited 2013-10-09 06:18
Short version: After asking Rybka3 to evaluate a position to certain depth, say 13, it is sometimes possible to find another move that evaluates better than Rybka's move.
Is this right? How can one find the best move within a certain search depth?

Long version: I want to find out how much human moves deviate from Rybka's using a certain search depth. Given a position, first I do a search to depth 13, note the suggested move and its evaluation. Then I perform the human move and evaluate the resulting position to depth 12 (so total 13 from original pos). About one move of 10-15, the evaluation of the human made move is better than the suggested best move!
Of course, I can loop through all legal moves in the position, perform the move and then do a depth 12 search. But this would increase the amount of computation by a factor of 15-20 by evaluating a lot of uneccesary positions... and it is slow already.

So, how do I best measure the human error to Rybka in a reproducable way? (Using x minutes on machine y is too unreproducable for my taste).
Parent - - By user923005 (****) [us] Date 2013-10-09 07:23
Chess engines are not deterministic.
Especially if SMP is involved, you will get different answers for the same problem set if you run it multiple times with the same time control or the same depth.

If a human is using Rybka to cheat, they will search to depths greater than 13 plies, else it is not worth the bother.

I would ask myself some questions:
1.  What are you really trying to calculate and how does it relate to my final goal?
2.  What level of certainty do you wish to achieve?
3.  How much data do you have at your disposal?
4.  What makes you suspicious of cheating?
5.  If you have a finding that appears telling, then what?

You will need mathematical rigor to make any sort of accusation.  I think that the problem is probably harder than you imagine.
Parent - - By Tapani [tw] Date 2013-10-09 08:36
This is not for cheat detection.

It is for evaluating and commenting on my own and my friends' games. Basically for a report generator.

The choice of 13 is because there are a few academic papers and tech reports using that depth, and (initially?) I want to have compatible and comparable results.

Any indeterminism must be placed there intentionally -- is there a way to turn it off in Rybka 3?
Parent - - By user923005 (****) [us] Date 2013-10-09 09:43
Some indeterminism is intentional, some unintentional.
If you use multi-threads, then the task scheduler, which is competing with every other thread on the machine, comes into play.
In single threaded mode, you will get more reproducible results.

My suggestion to accomplish your task is as follows:
Analyze every position in the game at 5 seconds per position.
You will find a few places where there is a sudden change in centipawn evaluation.
Go back to these places, and back up a few half moves.
Analyze again at one minute per position.
After performing this procedure, you will have discovered all the interesting pivot points in the game.
So now, examine them manually, looking to see if you or your friends made a brilliant move or a blunder.
The most valuable thing to find is the blunders.  Examine them carefully.  Is there a pattern?
When you understand what was sprung upon you or your friends, you now have a new technique to spring on your opponents.

I do this by converting the PGN games into Epd records using pgn2fen.  You can find it here:
http://7sun.com/chess/

The most important flags are -e to get Epd records and -l to get the last position from the game (if you want it -- I usually do).
Now that you have a collection of Epd records, you can analyze them easily with a chess GUI.
Aquarium will do it, Arena {free} will do it, and many other choices.

You can also analyze the games as PGN files, but I like Epd because it is already separated into the interesting pieces and I can easily collect all the white records or all the black records.
I am sure you will get other good suggestions.
Parent - By Tapani [tw] Date 2013-10-11 09:16
Thank you for your suggestions. They can be helpful for someone who hasn't already coded a preliminary report generator and just needs to get the tweaking right (if anyone wants to see some preliminary report pdfs, just say so -- or better yet send me your game. Rybka3 @ Depth 13 is not strong enough to critizise top GMs games)

You seem to be right about multiple CPUs. As a programmer I cannot understand how the indeterminism comes in, but maybe the PVS causes them.

Anyway, I found another thread ( http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=12491 ) about how to make Rybka 3 deterministic. Will try if that helps.
Parent - - By turbojuice1122 (Gold) [us] Date 2013-10-09 20:38
You should search for and send an e-mail to Ken Regan, as he has done a lot of work on this.
Parent - - By Tapani [tw] Date 2013-10-11 09:20
Thanks, I will once I have tried hard enough myself. You know: do not meddle in the affairs of professors, they're subtle and quick to anger. :-)
Parent - By turbojuice1122 (Gold) [us] Date 2013-10-11 11:07

> do not meddle in the affairs of wizards, they're subtle and quick to anger.


When the hobbits obeyed that advice, they very nearly found themselves captured by Nazgul. :twisted:
Parent - By siah (***) Date 2013-10-13 05:24
If best move was best move :roll:, ask ulit!
Parent - - By M ANSARI (*****) [kw] Date 2013-10-14 05:58
The best way is to have multiple variations set at 3 or more.  This setting will usually find a "best move" faster in a given position.  This assumes that the "best move" is something that might be not caught so quickly with normal search.  I find that using MV sometimes finds an obscure tactical shot 10x faster than normal search.
Parent - - By Dragon Mist (****) [hr] Date 2013-10-14 09:40
On the other hand, using multi PV can lead to serious trouble. I run DR4.1 for couple of months in 2-PV mode only, and concluded never to use it again, as it can miss some serious threats that otherwise wouldn't be missed. A topic for Rybka development and bug cleaning. If there was any.
Parent - - By M ANSARI (*****) [kw] Date 2013-10-14 16:43
That could be due to the stall bug that although improved with R 4.1, I feel is still there.  I think it depends on the position and if something stalls then there is absolutely no progress in the position as the depth and nps just get stuck.
Parent - - By Dragon Mist (****) [hr] Date 2013-10-14 18:52
There is no stall at all. It was with DR4 and never seen with DR4.1. DR4.1 goes down the depths and just misses the simplest of strikes. I am not allowed to post a position in question, but 2 pv didn't see the blow even after 8 hours of infinite. When the next move (opponents) is played, in single pv it sees it is loosing at depth 15 (!!), i.e. immediately. I have some more examples, but not that radical. I conclude there is something wrong with multi pv analysis of DR4.1.
Parent - By M ANSARI (*****) [kw] Date 2013-10-15 05:51
Very possible, I don't think we tested MPV extensively.  I don't get why you are not allowed to post a position in question?
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka: Best move is not the best move?

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill