It seems to me that two opening books could be compared and the "differences" identified. For example, if a position is selected such that that position occurs in both books, then one book might recommend (or require) one continuation from that position but the other book recommend (or require) a different continuation. Presumably, the optimum book would recommend the optimal continuation (again, allow for the non-uniqueness since two distinct continuations might produce the same result statistically) for the given engine, its settings, its opponent, etc.
Do I have this right?
If not, please enlighten.
Incidentally, and I am sure that there is no easy answer, I wonder what are the primary engine design features that impact book selection.
> My impression is that, once a specific engine's settings have been selected, then there is an opening book (not necessarily unique) which is perfect for that engine with that particular set of engine settings.
I think you mean "openings". There's a set of openings that would be perfect for the engine, and a book could be made that contains them. That such book already exists is rare unless someone built it specifically for the engine.
What the community has been doing is playing games of the top engines of the time, and matching their results to have variations in where the engine has the best performance. This leads to having books that suit the best engines, nobody cares if the weaker engines perform bad with them. When new, stronger engines appear, such top books may not suit the new engine, and they would become obsolete (though they'd become obsolete as their main lines are refuted anyway.)
And this is opponent dependent, the book that is perfect for engine X of today because it has a great performance against top engines will probably not work against top engines of the future. A good book has to be reworked all the time against what is on fashion.
What you say makes sense to me.
Do I have the following right?
(1) I assume engine designers are very interested in winning engine vs engine tournaments and matches, because it affects their sales and hence personal incomes. Hence, I would expect that they would be very particular about which opening books they used for any high-profile engine vs engine encounter.
(2) I also assume that the engine designers do, in practice, perform exhaustive post-mortem analyses of these games, especially the lost games, to determine the causes of the losses or of the bad moves their engines played. In particular they would try to determine whether or not the choice of opening lines was the culprit. If they decided the opening needed changing they they would modify their opening book so that the same opening mistake would not be repeated later against the same opponent.
(3) I also assume that engine designers make trial runs against whatever opponent engine programs are available, using assumptions as to the books that such opponents might use. Perhaps these trial runs could be done in an automated fashion, testing outcomes for all of the opening positions that could occur in the upcoming engine vs engine encounter.
(4) Finally, I assume that it could happen that an engine designer would modify ("tweek") the engine settings or even the engine design based on these pre-event trails.
On the other hand, could it be that engine designers are focussed on the guts of their programs and do not devote much time to opening books?
>> (1) I assume engine designers are very interested in winning engine vs engine tournaments and matches, because it affects their sales and hence personal incomes. Hence, I would expect that they would be very particular about which opening books they used for any high-profile engine vs engine encounter.
Most of them have a special opening book that they built particularly for the engine. They use private books they make themselves so the opponent isn't able to prepare.
>(2) I also assume that the engine designers do, in practice, perform exhaustive post-mortem analyses of these games, especially the lost games, to determine the causes of the losses or of the bad moves their engines played. In particular they would try to determine whether or not the choice of opening lines was the culprit. If they decided the opening needed changing they they would modify their opening book so that the same opening mistake would not be repeated later against the same opponent.
Yes. They also make changes in the engine to address these issues. I remember how changes in Rybka were made to improve her after Zappa defeated her.
>(3) I also assume that engine designers make trial runs against whatever opponent engine programs are available, using assumptions as to the books that such opponents might use. Perhaps these trial runs could be done in an automated fashion, testing outcomes for all of the opening positions that could occur in the upcoming engine vs engine encounter.
It's more effective to play against yourself once you're strong enough. This is more about losing games and fixing the engine's problems than about what openings you play, and if the top engine can't beat itself in some variation the weaker engine doesn't have a chance either. When going against a weaker opponent, the best strategy in an opening book that aims for a complex position or one in where the engine is known to have a high percentage of decided games, this will increase the chance of decided game and you can be confident that if the strategy backfires and the weaker engine gains the initiative, that your engine will be able to save the draw or find counter-play later on.
Against a weaker opponent the biggest concern is its opening book, the longer the game is in book the stronger the weaker engine will play, as it's backed by theory, so a priority would be to get as soon as possible to an out of book position so the opponent's weakness is revealed.
>Finally, I assume that it could happen that an engine designer would modify ("tweek") the engine settings or even the engine design based on these pre-event trails.
This is of the most importance as the game will be decided by what moves the engine chooses when the book ends. The developers focus on changing the engine design, and hire a special person that designs the opening book so they don't have to worry about it.
>On the other hand, could it be that engine designers are focussed on the guts of their programs and do not devote much time to opening books?
Yes, the above things I've been saying are usually done by the person in charge of the opening book, which may not necessarily be the engine's programmer. And in the end, the opening book will only give you a slight edge against the opponent (unless he does a lousy job), and the engines will decide the outcome. That is, once the engines are far apart in strength, if they played with reversed openings (that is, I use your book and you use mine) the outcome would be the same (stronger engine winning.)
Your comment about getting out of the book as soon as "possible" against weaker engines is a point I had failed to think about. This seems analogous to games I have played in over-the-board chess tournaments (in my younger days) in which my opponent would deviate early in the opening. The danger with that was that the deviation often was a weak move, at least in the case of the human vs human games I played in. In fact, as a strong amateur, I once was paired against a strong master. I "knew" that he surely must have a powerful tournament weapon in the form of a well-prepared opening repertoire so, as White, I played 1.a3. Naturally, he eventually won but in our post-mortem analyses he admitted not having done a lot of work preparing an opening for that move. (Much later, I "learned" that 1...g6 is a good response to 1.a3.)
A similar idea is that sometimes book lines tend to become well known. I am sorry I cannot give an example, but I remember some Sicilian openings which seemed to come up all too often in my tournament games. The corollary to this is that the individual assigned to develop a custom book for an upcoming high-profile engine vs engine event might find alternative lines that are, in practice, "just as good" but not seen often. I believe that is what you were suggesting.
Changing the subject slightly: (not necessarily about openings)
In my internet games (15 0 or slower), both I and my opponents make quite a few "really bad" moves in the game, a fact which becomes apparent during post-mortem analysis. Someone commented here that even the strongest human chess players do that, but their "really bad" moves are bad relative to their particular standard. By a GMs standard, I suspect many of my "good" moves are really bad. It's all relative. So, my question is: "Is this not true also for chess engines (middlegame)?" When I see my blunders and weak moves during post-mortem analysis, my primary concern is to answer the question "What was wrong with the way I was thinking?" instead of "What did I miss?" This makes me wonder if there is an engine equivalent to human "bad (counter-productive or non-productive) thinking habits?" In other words, when the engine designer/programmer is searching for ways to improve his/her engine code, perhaps the WAY the engine was "thinking" when the mistake was made would be of most concern. I suspect that it is TYPES of "thinking" errors which would be of most concern.
What do you think about this?
As always, any comments would be appreciated.
> The corollary to this is that the individual assigned to develop a custom book for an upcoming high-profile engine vs engine event might find alternative lines that are, in practice, "just as good" but not seen often. I believe that is what you were suggesting.
Yes. Playing bad lines makes no sense, but the chess space is so huge that finding playable unexplored variations is not that hard.
>Is this not true also for chess engines (middlegame)?
It is true for chess engines and for all chess entities, no matter how strong, as long as they play imperfect chess. It happens on the opening and endgame too, the only part that has been solved is 6men tablebases and a entity is only expected to avoid blunders in them. This is also hidden by the fact of opposition that is unable to exploit the really bad moves that aren't apparent, but a stronger chess entity (or the same entity with a much longer time control) could.
I use chess engines in interactive analysis for hours in my correspondence chess games and still make such blunders, top correspondence chess players of today make such moves that top corr players from the future would punish and show how bad they are, and as long as chess isn't solved, this situation will remain.
>In other words, when the engine designer/programmer is searching for ways to improve his/her engine code, perhaps the WAY the engine was "thinking" when the mistake was made would be of most concern.
These bad moves are usually product of the horizon effect, no matter how good your evaluation is and how much positions you look at, there's always going to be something outside your horizon, the positions that you didn't analyze, and if the best line was pruned and missed, you're going to play a bad move. This is being fixed by stronger and faster hardware, and by engine evaluation improvements so that the bad variations have more chance of being pruned, but searching some positions means you don't search others, so the bad move is played somewhere else instead of here. The trick is to make it so the positions where the blunder is made are minimized.
Now, I just feel like showing pictures about what I'm saying. What an engine does when analyzing a position is building a tree of variations, my move, opponent move, my move, opponent move, in where the tip of the variation has some evaluation, then, the engine goes back to see if any of the sides could improve on this score by using Minimax, and if it doesn't, it goes to the next iteration. Since the engine is time limited for doing this, what it does it cutting some variations and extending others, so instead of the tree ending looking like a circle like in Brute Force, it looks like this:
This is the engine's tree, the horizon is marked with black outline, in red is all analyzed positions, and in green it's all the chess space outside the horizon, all the positions that there wasn't time to analyze (in reality, it would look with a lot more spikes, but this would be a general shape.)
The longest the spike, the deeper the engine looked. The thicker the spike, the more complex the positions are as the engine had to check for more alternatives of the moves. Slim means the variations are more forced so the engine had it easier to look deeper.
So here, the engine considered 9 main moves and looked at them, the flat line above Root indicates that those moves weren't extended at all, they're probably outright blunders like hanging a piece for no compensation or making a move that gets you mated. These moves are pruned so the north of the chess space is almost unexplored.
Now, let's suppose that there are some truly best moves in the position, those that the strongest chess entity would play, like, the fastest mating moves in a 6men TB. And they go to the end of the game. There's no benefit in deviating from them and doing so would just make their line thinner (until the player misses it and he goes to a losing position). Let's suppose this is a critical position and such a perfect path isn't wide.
On a best case scenario, it'd look like this:
Here, Blue marks the optimal path and purple the positions from the optimal path that were analyzed. This is a best case scenario because the tip of the spike, the position that was analyzed the deepest and from where the engine's score is being brought to the root, is inside the optimal path. The engine has great understanding of the position and it's unlikely it'll leave the optimal path in future moves as it gets wider.
A more normal scenario would look like this:
Here, the score of the engine is inaccurate as at some point it has a blunder in the PV (the blunder is played at the point at which Cyan and red intersect, the engine goes wrong and spends a lot of time analyzing irrelevant positions, here the Cyan part is outside the engine's horizon and it's important), so the score comes from a point outside the optimal play. This, however, is normal, the base of the optimal play was analyzed and an inaccurate move hasn't been made yet, this situation can be seen by just playing the PV of the engine out and seeing a fail low, an inaccurate move is played eventually but the moves near the root are accurate. In future moves it's expected that this is rectified and that the tip of the spike would move towards optimal play.
Now, a blunder:
Here, the engine pruned the best move of the position, and spend too few time on it, causing most of the analysis to be irrelevant. The part in where orange and red intersects marks the point in where the engine missed the critical move near the root, and then spent most of its time analyzing variations that it mistook as better. Depending on what side the engine is playing, it means it's going to play a game losing blunder and miss the winning move against it, or that it had a win on its face but missed it due to wrong evaluation. This is why interaction is important, to avoid this a user just needs to force this move to increase this horizon and have the key move inside the search.
Now, the improvements made to the engine in this position could have different effects. More hardware or more time would make all these spikes a little bit longer and thicker, making the tree bigger, and eventually, the small north spike would find the key move and the analysis could focus on the optimal line. Better search would make the spikes sharper, and would make the bad, irrelevant spikes shorter, which would give more time to the ones of the best moves, so the key move is found faster. Better evaluation would make the tree look more like this at earlier depth:
Since the key move would be found faster, most of the analysis focuses on it and the rest of the moves get less time from the engine. However, more knowledge means the engine is slower, so it can't analyze as much positions, so on other positions the tree wouldn't get to grow as big and would miss the optimal line. There's a tradeoff.
So in the end, these kind of blunders aren't of the highest concern, what will improve the engine most will be decreasing blunders overall (no matter what type of blunders they are.)
Your use of those pictures was really impressive!
There is too much material here for me to digest in a single sitting. I will post later if I have more questions.
If this keeps up, I will become a chess engine programmer before you know it. :-) :-) :-)
How does "knowledge" fit into what you are talking about? What does the engine do with "knowledge"? How does it figure into your illustrations?
> How does "knowledge" fit into what you are talking about? What does the engine do with "knowledge"? How does it figure into your illustrations?
The idea here is that with the right knowledge, the engine can have a more accurate score of the position without needing to search it.
Let's put a concrete example, assume the position we're talking about is one in where the optimal line is a Bishop sacrifice on h6. Then, on the "Blunder" image, the engine thinks "oh, that Bishop sacrifice is bad, there's some interesting positions out there" (the orange part) "but they don't work" (wrong) "I'll prune it" (that's why the red spike is so small.)
Now, with the right Knowledge in the King Safety function, the tip of the red spike would have a different score, a "this Bishop sacrifice is better than the other moves I have", so it'd extend the move, and see more of the optimal path (getting a fail high), and then, since this is so much better than anything else, the engine drops all other moves and focuses on it (that's why the last picture looks very different than the other ones.)
So at some point in the tree, the engine knows that the move is best, even if it doesn't know why (the reason is not yet on the analyzed positions).
The tradeoff here, is that now, in all the other positions with a Bishop attacking h6 that don't work, the engine will wrongly think those lines are worth checking, and will waste time checking them, this time is stolen from the main lines and the engine becomes slower.
That's why you can't lose a game, take a look at why you lost it, and plug a fix so it doesn't happen again in such positions, because it may decrease performance in similar positions where the fix isn't necessary and your engine loses time checking if this is a special case.
"Nobody ever said that the life of a chess engine designer/programmer would be easy." :-)
You have answered all the questions I have for now regarding engine-specific opening books.
Thanks for your help!
Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill