Not logged inRybka Chess Community Forum
Up Topic The Rybka Lounge / Computer Chess / How good is the Cerebellum book with Stockfish?
1 2 Previous Next  
- - By Peter Grayson (****) [us] Date 2016-08-07 16:21
The relatively new release of the Stockfish engine as the Brainfish engine paired with a strong book has an appeal , especially if the book is tuned for the engine. I am not sure that is actually the case, however the Cerebellum book is undoubtedly enhancing the engine's performance but the question is, is it any better than another well tuned book would accomplish?

  In carrying out the following match I had intended using my own book but with difficulties getting either of the Fritz GUI 13 or GUI 14 to load two versions of the same engine I managed to allocate the Vesely opening book dating back to January 2011 and unfortunately did not pick up on the mistake until I started analysing the first 80 games. Incidentally, the Fritz engine load issue was down to poor programming where the GUI programmers only compare if the first engine's string appears in the second engine's string taking no account of string length difference. Because I had added the suffix _NB to the engine running the Vesely book, I found the GUI did accept both if I loaded the longer string name as the first engine! Yet another example of sloppy and careless GUI programming!

  Although I had always considered the Vesely book as one of the strongest publicly available at the time, as the game progressed it looked as if the Cerebellum book was highlighting the fact that lines had been updated since 2010 when by game 80 it was running to the good by +14/=65/-1 +58 Elo with the solitary win for the Vesely book looking to be more  as a consequence of two bad moves by the other engine approaching the first time control.

  In contrast, by the time I had to stop the match after 120 games the differential was down to 26 Elo and in the last 40 games the engine + Vesely book scored to its favour +11/=22/-7 +35 Elo so quite possibly the engine + Vesely book may have turned it around if I had been able to run this through to 160 games.

  Initial conclusion though is that the 5.5 year old Vesely book gave the Cerebellum book a good contest and perhaps the Cerebellum book may not look so good in a longer match against an up to date alternative book that can be tuned for results as the match progresses. Unclear that the concept of the Cerebellum book is any improvement to what can already be achieved.

Cumputer: dual Xeon E5-2687w + 64 Gb RAM
Engine1: Brainfish 020816 + Cerebellum_Light book 31.07.2016
Engine2: Brainfish 020816_NB with no Cerebellum_Light book + Vesely .ctg Book, GUI book learning enabled.
Engines' hardware resource = 6 cores + 4 Gb Hash per engine, LP enabled
5-man Syzygy bases,
Ponder=on,
Time Control: 40 moves in 5 minutes repeating until mate or forced draw.

PeterG
Attachment: Brainfish020816vsVeselyBook.zip - The Games (155k)
Parent - By Ozymandias (****) [es] Date 2016-08-07 17:31

> Initial conclusion though is that the 5.5 year old Vesely book gave the Cerebellum book a good contest and perhaps the Cerebellum book may not look so good in a longer match against an up to date alternative book that can be tuned for results as the match progresses. Unclear that the concept of the Cerebellum book is any improvement to what can already be achieved.


I don't think it's better than the ones being updated, either.

The concept (minimaxing) may be an improvement over stats, that remains to be seen, but because it has "Fish" in the name, everyone is running around like headless chicken.
Parent - - By Velmarin (*) [es] Date 2016-08-09 09:45
Vesely, was a great book.
Its author (I believe was my) made a work of Titans.
Even on the net, I am going to download it.
Parent - - By Eelco de Groot (***) Date 2016-08-20 21:33
Is Vesely's book still downloadable? I have a copy, several I believe, but I remember it took very long to download. It wasn't optimized for engines, so I'm not sure it was such a good opponent for the match? It is probably too large to put it here as a file, but maybe on RybkaChess.com, but it would be necessary to have permission from the author.

But it is very quiet in here, and I am afraid the few of you that are here, you are missing the really big news; Jeroen has made an update of his opening book! Check out the subforum, or Talkchess, or Ed's website. It is pretty large also. If I had to choose to spend my money on this or on the Cerebellum, I would go to the former. Because it feels to me as a real book. Of course you can combine any book with Brainfish too...   We currently have 50 guests on Talkchess, I don't know if they are all just downloading Jeroen's openingbook, without saying a word of thanks...

This is the new NCO. (Noomen Chess Openings)
Parent - - By shrapnel (***) [in] Date 2016-08-21 06:47
No, I don't agree, Brainfish/Cerebellum is much stronger.
I tested it
I ran Brainfish/Cerebellum on 1 computer against Asmfish/Noomen.ctg on my 2nd computer, in a French Defence.
When the Engines came out of Book, Brainfish/Cerebellum had a big +0.60 lead over the Noomen Book.
Such a big lead is disastrous in online engine-engine matches.
Cerebellum wins, no contest.
Parent - - By Venator (Silver) [nl] Date 2016-08-21 08:42
That's funny, as Noomen.ctg doesn't play the French defence......
Parent - - By shrapnel (***) [in] Date 2016-08-21 09:00
Funny ? Yes, I forced it to play the French Defence.
I like the French defense, and if this book is so good, it should be able to handle it.
So, sad, I think is the more appropriate word, not 'funny'.
On the other hand, Cerebellum has handled anything I threw at it.
Parent - - By Venator (Silver) [nl] Date 2016-08-21 09:14
Yes, I forced it to play the French Defence.

Well, there you go. The defences in my book are Ruy Lopez, Sicilian and Caro-Kann. It avoids the French and not without reason. So testing a line that is not actively played, is rather useless, because the book will never play that on its own and it is not optimised to use this defence as black.
Parent - - By Dragon Mist (****) [hr] Date 2016-08-21 13:06
Jeroen, just ignore shrapnel comments, he is known troll. Thank you for you newest book.
Parent - - By Venator (Silver) [nl] Date 2016-08-21 13:33
Thank you :-). Currently I am doing a test with my book against Cerebellum and the match is very close: 3-2 in favour of Cerebellum, with many draws. So it cannot be that bad :-).

BTW, I like this Cerebellum book/idea, so I am going to take a look into it. I have forwarded it to my testers as well. I see a new challenge, indirectly thanks to Shrapnel!
Parent - - By Peter Grayson (****) [gb] Date 2016-08-22 00:02

> Currently I am doing a test with my book against Cerebellum and the match is very close


Ditto here. However I have it 3-1 in favour of your book after 50 games using 2 instances of the latest Brainfish engine.

Brainfish 190816 64 POPCNT_CB = engine with Cerebellum book,
Brainfish 190816 64 POPCNT_NB = engine without Cerebellum book using Noomen.ctg (2016) with optimised values including learning
GUI: Deep Fritz 14
6 cores + 4Gb hash per engine, Time control 40 moves in 5 minutes repeating, ponder = on
Machine dual Xeon E5-2687w + 64Gb RAM

46 draws and some of them coming very quickly. Noted game 2 was a book draw without a single move being played by either engine.

Of the 4 wins I have 3 down to engine mistakes with the only immediately identifiable book influence being in game 49 when as Black, the Noomen.ctg book outbooked Cerebellum and pointed the engine in the right direction giving it a -1.78 score in its favour with the indication that  the Cerebellum book may be in error here with Black's 16..0-0 having a good track record. Perhaps an example where Stockfish tuning goes awry ...

[Event "Cerebellum vs Noomen, 5m/40+5m/40+5m/40"]
[Site "TERMINATOR1"]
[Date "2016.08.21"]
[Round "49"]
[White "Brainfish 190816 64 POPCNT_CB"]
[Black "Brainfish 190816 64 POPCNT_NB"]
[Result "0-1"]
[ECO "B33"]
[Annotator "0.21;-1.78"]
[PlyCount "142"]
[TimeControl "40/300:40/300:40/300"]

{Intel(R) Xeon(R) CPU E5-2687W 0 @ 3.10GHz 3092 MHz  W=43.4 plies; 12,142kN/s;
4,070,952 TBAs  B=42.9 plies; 12,382kN/s; 4,781,843 TBAs; Noomen.ctg} 1. e4 {
[%eval 1,0] [%emt 0:00:01] White last book move} c5 {[%eval 0,0] [%emt 0:00:00]
} 2. Nf3 {[%eval 1,0] [%emt 0:00:00]} Nc6 {[%eval 0,0] [%emt 0:00:00]} 3. d4 {
[%eval 1,0] [%emt 0:00:00]} cxd4 {[%eval 0,0] [%emt 0:00:00]} 4. Nxd4 {[%eval
1,0] [%emt 0:00:00]} Nf6 {[%eval 0,0] [%emt 0:00:00]} 5. Nc3 {[%eval 1,0]
[%emt 0:00:00]} e5 {[%eval 0,0] [%emt 0:00:00]} 6. Ndb5 {[%eval 1,0] [%emt 0:
00:00]} d6 {[%eval 0,0] [%emt 0:00:00]} 7. Bg5 {[%eval 1,0] [%emt 0:00:00]} a6
{[%eval 0,0] [%emt 0:00:00]} 8. Na3 {[%eval 1,0] [%emt 0:00:00]} b5 {[%eval 0,
0] [%emt 0:00:00]} 9. Bxf6 {[%eval 1,0] [%emt 0:00:00]} gxf6 {[%eval 0,0]
[%emt 0:00:00]} 10. Nd5 {[%eval 1,0] [%emt 0:00:00]} f5 {[%eval 0,0] [%emt 0:
00:00]} 11. Bd3 {[%eval 1,0] [%emt 0:00:00]} Be6 {[%eval 0,0] [%emt 0:00:00]}
12. O-O {[%eval 1,0] [%emt 0:00:00]} Bxd5 {[%eval 0,0] [%emt 0:00:00]} 13. exd5
{[%eval 1,0] [%emt 0:00:00]} Ne7 {[%eval 0,0] [%emt 0:00:00]} 14. Nxb5 {[%eval
1,0] [%emt 0:00:00]} Bg7 {[%eval 0,0] [%emt 0:00:00]} 15. Nc3 {[%eval 1,0]
[%emt 0:00:00]} e4 {[%eval 0,0] [%emt 0:00:00]} 16. Bc4 {[%eval 1,0] [%emt 0:
00:00]} O-O {[%eval 0,0] [%emt 0:00:00]} 17. Qd2 {[%eval 1,0] [%emt 0:00:00]}
Ng6 {[%eval 0,0] [%emt 0:00:00]} 18. Bb3 {[%eval 1,0] [%emt 0:00:00]} f4 {
[%eval 0,0] [%emt 0:00:00]} 19. Nxe4 {[%eval 21,27] [%emt 0:00:09]} f5 {[%eval
0,0] [%emt 0:00:00]} 20. Nc3 {[%eval 15,26] [%emt 0:00:03]} f3 {[%eval 0,0]
[%emt 0:00:00]} 21. Qe3 {[%eval 11,30] [%emt 0:00:15]} Ne5 {[%eval 0,0] [%emt
0:00:00]} 22. Qf4 {[%eval -51,27] [%emt 0:00:25]} Qe8 {[%eval 0,0] [%emt 0:00:
00] Black last book move} 23. h4 {[%eval -55,27] [%emt 0:00:27]} fxg2 {[%eval
-178,28] [%emt 0:00:31]} 24. Kxg2 {[%eval -191,28] [%emt 0:00:01] (Rfe1)} Qh5 {
[%eval -240,26] [%emt 0:00:08]} 25. Rg1 {[%eval -272,28] [%emt 0:00:20] (Kh1)}
Bh6 {[%eval -287,30] [%emt 0:00:30]} 26. Kf1+ {[%eval -270,32] [%emt 0:00:00]}
Kh8 {[%eval -291,28] [%emt 0:00:03]} 27. Rg5 {[%eval -295,27] [%emt 0:00:02]}
Bxg5 {[%eval -298,30] [%emt 0:00:11]} 28. Qxg5 {[%eval -291,31] [%emt 0:00:00]}
Qf7 {[%eval -284,27] [%emt 0:00:04]} 29. Qf4 {[%eval -309,32] [%emt 0:00:49]}
Rg8 {[%eval -310,32] [%emt 0:00:00]} 30. Re1 {[%eval -323,27] [%emt 0:00:03]}
Rg4 {[%eval -336,29] [%emt 0:00:17]} 31. Qh6 {[%eval -336,32] [%emt 0:00:09]}
Qe7 {[%eval -329,30] [%emt 0:00:00]} 32. Qe6 {[%eval -342,28] [%emt 0:00:04]}
Qg7 {[%eval -346,29] [%emt 0:00:18]} 33. Rxe5 {[%eval -353,29] [%emt 0:00:01]}
dxe5 {[%eval -350,25] [%emt 0:00:03]} 34. d6 {[%eval -357,30] [%emt 0:00:29]}
Rd8 {[%eval -353,30] [%emt 0:00:00]} 35. Bd5 {[%eval -335,28] [%emt 0:00:11]
(d7)} Rg6 {[%eval -353,30] [%emt 0:00:56]} 36. Qxf5 {[%eval -357,32] [%emt 0:
00:00]} Rf8 {[%eval -367,26] [%emt 0:00:04]} 37. Qe4 {[%eval -372,29] [%emt 0:
00:42] (Qh3)} Rf4 {[%eval -374,28] [%emt 0:00:17]} 38. Qe3 {[%eval -372,28]
[%emt 0:00:10]} Rxd6 {[%eval -377,28] [%emt 0:00:07]} 39. a4 {[%eval -369,28]
[%emt 0:00:18] (Bg2)} Rxh4 {[%eval -396,27] [%emt 0:00:22]} 40. Be4 {[%eval
-394,27] [%emt 0:00:00]} Rf6 {[%eval -416,28] [%emt 0:00:33]} 41. b3 {[%eval
-454,30] [%emt 0:00:52] (Ke2)} Rh2 {[%eval -472,29] [%emt 0:00:31]} 42. Nd1 {
[%eval -482,32] [%emt 0:00:00]} Qg4 {[%eval -479,25] [%emt 0:00:02]} 43. Qd3 {
[%eval -477,25] [%emt 0:00:00]} Rf8 {[%eval -490,27] [%emt 0:00:12]} 44. Bb7 {
[%eval -526,30] [%emt 0:00:49] (a5)} Rh4 {[%eval -529,29] [%emt 0:00:32]} 45.
a5 {[%eval -542,30] [%emt 0:00:00] (Bg2)} Qg7 {[%eval -564,29] [%emt 0:00:15]}
46. Bg2 {[%eval -605,29] [%emt 0:00:16]} Qg5 {[%eval -568,29] [%emt 0:00:02]}
47. Qc3 {[%eval -619,30] [%emt 0:00:22]} Rhf4 {[%eval -654,27] [%emt 0:00:00]}
48. b4 {[%eval -659,28] [%emt 0:00:32]} h5 {[%eval -739,30] [%emt 0:00:09]
(Qf5)} 49. b5 {[%eval -613,24] [%emt 0:00:06]} axb5 {[%eval -777,28] [%emt 0:
00:06]} 50. a6 {[%eval -694,27] [%emt 0:00:08]} Rd4 {[%eval -815,29] [%emt 0:
00:00]} 51. Qe1 {[%eval -715,31] [%emt 0:00:16] (Ne3)} h4 {[%eval -944,27]
[%emt 0:00:10] (Ra4)} 52. Ne3 {[%eval -1045,28] [%emt 0:00:35] (a7)} Ra4 {
[%eval -1145,26] [%emt 0:00:19]} 53. a7 {[%eval -1275,26] [%emt 0:00:06] (Qd2)}
Rxa7 {[%eval -1753,25] [%emt 0:00:28]} 54. Bd5 {[%eval -1549,27] [%emt 0:00:00]
} Ra4 {[%eval -1959,23] [%emt 0:00:12] (Ra6)} 55. Bg2 {[%eval -1384,23] [%emt
0:00:04] (Be6)} Raf4 {[%eval -3002,22] [%emt 0:00:05] (Ra6)} 56. Bb7 {[%eval
-12341,30] [%emt 0:00:10] (Ke2)} h3 {[%eval -12345,32] [%emt 0:00:14] (Rxf2+)}
57. Nd1 {[%eval -12344,30] [%emt 0:00:03] (Ke2)} h2 {[%eval -32735,32] [%emt 0:
00:05]} 58. Ke2 {[%eval -12345,32] [%emt 0:00:00]} Qh5+ {[%eval -32739,32]
[%emt 0:00:06]} 59. Kd2 {[%eval -32741,36] [%emt 0:00:00]} Rd8+ {[%eval -32741,
39] [%emt 0:00:09]} 60. Kc1 {[%eval -32743,43] [%emt 0:00:00]} Qxd1+ {[%eval
-32743,42] [%emt 0:00:08] (Rxd1+)} 61. Qxd1 {[%eval -32745,57] [%emt 0:00:03]}
Rxd1+ {[%eval -32745,51] [%emt 0:00:00]} 62. Kxd1 {[%eval -32747,58] [%emt 0:
00:01]} e4 {[%eval -32747,60] [%emt 0:00:05]} 63. Kd2 {[%eval -32749,64] [%emt
0:00:00]} h1=Q {[%eval -32749,62] [%emt 0:00:01]} 64. Ke3 {[%eval -32751,66]
[%emt 0:00:00]} Qf3+ {[%eval -32751,77] [%emt 0:00:06]} 65. Kd4 {[%eval -32753,
92] [%emt 0:00:00]} Qxf2+ {[%eval -32753,85] [%emt 0:00:01]} 66. Kd5 {[%eval
-32755,99] [%emt 0:00:00]} Rf5+ {[%eval -32755,121] [%emt 0:00:06]} 67. Kxe4 {
[%eval -32757,127] [%emt 0:00:00]} Rf4+ {[%eval -32757,127] [%emt 0:00:00]} 68.
Kd5 {[%eval -32759,127] [%emt 0:00:00]} Qd4+ {[%eval -32759,127] [%emt 0:00:00]
} 69. Ke6 {[%eval -32761,127] [%emt 0:00:00]} Rf6+ {[%eval -32761,127] [%emt 0:
00:00]} 70. Ke7 {[%eval -32763,127] [%emt 0:00:00]} Qd6+ {[%eval -32763,127]
[%emt 0:00:00]} 71. Ke8 {[%eval -32765,127] [%emt 0:00:00]} Rf8# {[%eval
-32765,127] [%emt 0:00:00]} 0-1

PeterG
Attachment: CerebellumvsNoomenBookG1-50.zip - Games 1-50 (52k)
Parent - By Venator (Silver) [nl] Date 2016-08-22 14:41
Hi Peter,

Thanks for posting. Interestingly, in my own test my book had wins in the exact same Sicilian Sveshnikov line as the game you posted here :-).

Indeed there are many draws that come right out of book. Some lines just seem to head to that result, no matter what alternative you try.
Parent - - By Peter Grayson (****) [gb] Date 2016-08-22 15:22
Games 51-100 saw 4 wins without reply for the Cerebellum book. As per previous comment there seemed nothing immediately winning at the book departure points but there may be more subtle attributes in the positions that lead to wins much later in the game. However with the very low win rate being maintained at just 8%, apart from game 49 the wins may just be the result of SMP variability. The high number of draws suggest there is not too much difference in the books' capabilities but the question is; is it possible to tune the books to provide more winning opportunities without weakening?

I'll run this through to game 150 and run the same match using Komodo 10.1 with the Noomen book. I suspect Komodo 10.1 is a little way behind the latest Stockfish engines but it will be an interesting control to check out if there any issues with the Stockfish book tuning. It would perhaps be of more interest if there was an up to date book tuned specifically for the Komodo engine that would really test the Cerebellum tuning method.

PeterG
Attachment: CerebellumvsNoomenBookG51-100.zip - Games 51-100 (44k)
Parent - - By shrapnel (***) [in] Date 2016-08-22 15:36
Seems strange that the "SMP variablity" seems to favor the much-criticized  Cerebellum.
No offence, but I can't help but wondering if the factor of SMP variabilty would have been raised if the " 4 wins without reply" had been in Noomen's favour !
Parent - - By Peter Grayson (****) [gb] Date 2016-08-22 16:33
You are reading too much into it. The main point of this exercise was to identify if the books offered any killer lines. The results speak for themselves; they do not but show themselves to be well balanced but tending towards draws.

With bigger hardware resource allocated to each engine and longer time controls the draw rate would likely increase further when SMP variability would be of less influence.

As with any non-GUI dependant book, there is some merit in the Cerebellum book because it can be used in any GUI supporting UCI engines. However there is no flexibility because the end user is stuck with what it provides and currently it cannot be used for other engines. I do not see this as a new concept at all.

The Chessbase book system gives the end user many more options; for example the facility to modify the book lines, introduce and delete moves without having to build a new book as well as marking moves specifically to be played with weightings and the option of different settings for learning and variety. The Chessbase book can be used with any engine but the main drawback is use outside of the Chessbase GUI's is limited and in most cases not possible.

PeterG
Parent - - By shrapnel (***) [in] Date 2016-08-23 04:12
I talked to the Author of Cerebellum and he confirmed that Cerebellum can be used with any Engine in the final, commercial version which will be released in November.
So that takes care of that particular point.
Parent - By Peter Grayson (****) [gb] Date 2016-08-23 11:42

> I talked to the Author of Cerebellum and he confirmed that Cerebellum can be used with any Engine in the final, commercial version which will be released in November.


Will that be in .ctg format or will the engine's code have to accommodate the .bin book? Is there merit in a different engine using a book tuned by and for Stockfish?
Parent - - By Venator (Silver) [nl] Date 2016-08-22 16:29
As per previous comment there seemed nothing immediately winning at the book departure points but there may be more subtle attributes in the positions that lead to wins much later in the game.

I notice that the Cerebellum book sometimes stays in book for over 40-45 moves, when the position is still +0.15 according to SF, but objectively totally drawn. IMO it is pointless to have such enormous long lines in the book. I prefer to have shorter lines, accepting a -0.15 with black and having the opportunity to find something new later on.

The high number of draws suggest there is not too much difference in the books' capabilities but the question is; is it possible to tune the books to provide more winning opportunities without weakening?

After some testing and tweaking my second match ended 2-2 with the rest draws. The problem is that the Cerebellum book doesn't seem to have much variety. The lines it plays are very, very solid and can hardly be beaten without taking too much risk. It gets a bit boring, actually. I see countless times the same Italian game, the same Najdorf line, the same Sveshnikov line and the same line against the Caro Kann. I am not very keen on a rat race doing countless improvements at move 40 to 45 to avoid a loss. And I prefer to test against books with a lot more variety.

I'll run this through to game 150 and run the same match using Komodo 10.1 with the Noomen book. I suspect Komodo 10.1 is a little way behind the latest Stockfish engines but it will be an interesting control to check out if there any issues with the Stockfish book tuning.

Thanks for testing. Indeed it will be interesting to see how a match vs Komodo will go. Looking forward to this match.
Parent - By Peter Grayson (****) [gb] Date 2016-08-22 17:00
Thanks for adding your Comments Jeroen.

The Cerebellum book does seem very narrow and very deep. It would be worse if the 2nd move value was turned down to zero. I selected a value of 50 that I hoped would give reasonable variability but it does seem to be mostly down to the opponent to provide the variety. One benefit of going so deep is that in faster time control games it can gain a significant time advantage but here, using the same engine for both books there was no measurable benefit for either set up when one outbooked the other.

Given the Cerebellum book is labelled _Light it leads one to believe there is a _Full version and having whetted people's appetites I wonder if the intention is for the _Full version to be commercial. It may have some appeal for some but I suspect the serious users prefer their own books.

Peter
Parent - - By Peter Grayson (****) [gb] Date 2016-08-22 23:10
Games 101-150 saw just 3 more wins; 2-1 in favour of the Cerebellum book giving a final score of 7-4 to the Cerebellum book with 139 draws.

Game 103 was a Giuoco Piano with the Noomen book as Black exhausted with last move as early as move 8 and the Cerebellum book continued to move 14.cxd4 when after Blacks reply of ..c5, White scored an  advantage of 0.62 for 15.a5 and with still much work to do it nevertheless did not look back. This could be down to book advantage but unclear.

The second win as White, this time for the Noomen Book came in game 130 and was a consequence of an old Stockfish problem rearing its head again. A sequence of 0.00 scores, here for both engines, but after Black's 28..Ra5 0.00/28, White's 29.h6 scored it 0.59/30 and went on to win. This 0.00 issue with the Stockfish engine seems to surround possible draw scenarios that cause it to miss key moves. Checkout Black's moves 21 to 28 all scoring 0.00.

The third win for White and the second for the Cerebellum book saw the Noomen Book out booked by 7 moves. Despite the engines being the same check out the score differential after Black's 45..Kg7. Again something looks to be wrong here with the engine as Black being 1.7 at odds with White's score despite indicating 11 ply deeper search.

After 150 games, other than game 49 and possibly game 103, there was little measurable evidence to suggest the books had much influence in the wins with some indicators that all was not well with BrainFish (Stockfish) scoring in some of the games that suggested it was going astray.

I just checked the Book Move2 Probability and confirm the setting was 50 because ...

of the 35 games beginning 1.e4 where the Cerebellum book had Black it only responded with 1..e5. Not one Sicilian at all!

Of the 28 games opening 1.d4, as Black Cerebellum played 3 replies with Nf6 and the remainder with d5.

PeterG
Attachment: CerebellumvsNoomenBookG101-150.zip - Games 101 to 150 (47k)
Parent - - By Venator (Silver) [nl] Date 2016-08-23 07:58
Thanks a lot for the test, Peter. I have addressed the early book leave issues, I also had seen them in my own test. In a sense the Cerebellum book is excellent to find 'black holes' in ones own book. So I will do more tweaking and testing until all losses are gone. The 'bad' thing will be, that future matches between those two books might end in 100% draws....
Parent - - By Peter Grayson (****) [gb] Date 2016-08-23 10:36

> The 'bad' thing will be, that future matches between those two books might end in 100% draws....


Unfortunately the effort put in to find wins is rarely rewarded by that success lasting too long once the game(s) become public. The problem then however becomes the same as seen in today's correspondence chess with the high number of draws as a consequence of players using computers and opening databases.

The prediciton of the death of chess when all games end in a draw comes ever closer but there is still mileage perhaps using different engines and perhaps real rather than artificial intelligence? The general trend with engines has been towards moving away from positions creating tension to early simplification and I have always seen the opening book as a vehicle to force engines into those more interesting positions but it will require the artistic nature of the human mind to continue to create those lines leading to those positions.

I'll keep the BF + Cerebellum vs Komodo 10.1 + Noomen Book in this post for completeness.

Peter
Parent - - By Venator (Silver) [nl] Date 2016-08-23 18:13
The prediciton of the death of chess when all games end in a draw comes ever closer

In engine-engine games this might be possible. I am not afraid for us humans, though. Our tendency to make mistakes keeps the game interesting.

BTW, there is more in chess computer life than Cerebellum. My book is not intended for quick SF-SF games only. I refuse to skip the Ruy Lopez because that always leads to a draw. And I will refrain from playing lesser known lines only, to boost the results in engine matches. My target is a broader audience and thus the book should simply play all the popular openings.
Parent - By Ozymandias (****) [es] Date 2016-08-23 19:22

>In engine-engine games this might be possible.


First chink in the when-an-engine-that-pushes-for-the-win-arrives-the-draw-rate-will-plummet amor.

Next thing we'll know: there's no second Messiah.
Parent - - By Thomas Zipproth [de] Date 2016-08-23 19:49
Hi all,

thanks for all the tests, I will try to comment on the remarks and questions raised in this topic, reading all this was very interesting for me.
I was not sure where to put this, but as Jeroen seems to be the one challenged, I will answer him directly.

First refering to your comment "there is more in chess computer life than Cerebellum", that is only true for Cerebellum_light. The full version covers the same broader audience as your book, which I hope I'm able to explain in my post here.

The Cerebellum_light book is an export of the unpublished "Cerebellum full" book, which is mainly a tool for the deep analysis of chess postions and opening lines, intended to be used for example in Correspondence Chess and in the opening preparation of grandmasters. The full book offers most of the functions available in standard opening books, like statistics and manually selecting preferred lines. And of course all the additional Cerebellum functions. It can handle two engine caclulated graphs/trees together. You can see a part of his data on the "Demo" tab of the Brainfish website, which tooks some time to load. It is also part of a complete new chess Gui.

It can be very interesting to dig around in the Cerebellum book, searching for new lines and connections, calulating and adding new positions, recalculating the whole graph and look what has changed. It is possible and already occured several times, that adding an endgame position changed what is played in the first moves. The library is very broad and covers every aspect of chess openings, also rarely played Gambits.

Some time ago I tried to use Cerebellum for playing in engine chess too and developed a proxy which can connect the full book with any engine. The playing part was not so easy as it seems, because you have to handle many things like repeated postions, moves with equal scores but different depth in the graph and so on.

After succesfully playing for about a year on sites likes Infinity and Playchess, with several tournament victories and a lot of second and third places against strong opponents, I decided to develop Brainfish as a free implementation of the Cerebellum library intended for the use in engine chess, but without the analysing and statistics functions of the full library. The main reason for Brainfish is to get feedback from users to improve the library, to see how it behaves in competition, and of course a bit advertising for the full library. So far I'm very satisfied with the results.

Now to some of the issues mentioned here:

1. The Cerebellum book repeats loosing lines

With the concept of adding new positions only through the calculation of that positions and then performing a recalculation the whole graph, it is not possibly to do that on the fly. In engine chess that is not really a weakness, because practical experience shows that you normally don't loose two times in the same way in a sequence of about let's say 80 games, in fact you normally loose only about 1-2 games out of 100. So with the full library you can always recalculate your book with all the games you played, including the lost ones of course. For now some users of Cerebellum are sending me their games, and they are very satisfied. The book is updated about 3 times a week. Recent reports showed results in practical playing like for example +12 =120 -2.

2. The very narrow and very deep book.

It was a decision to let the book play only the strongest moves for the first release. I will keep that some time till I made some experience and then try to broaden to book moves, leading to a more flexible play. In the full version you can configure the book play exactly how you want, like in a standard opening book. In Cerebellum light I manually ruled out only the Marshal Gambit, which is a bit overestimated for white by Stockfish, and temporarily in the last 3 versions Sicilian, till I finished some bigger calculations.

3.) Book Matches

I think one single book match covers only a part of a books strength. A main aspect is how fast a book can be adapted to new lines, lost games or in general the progress of opening theory in chess and engine chess. With Cerebellum (full) that is very easy, new games are added automatically with all their calculated positions, and then you really know why a line is loosing, maybe the line was not bad at all, and only one move was faulty. In general I need about 200 games to "absorb" a good book, from that on it only wins occasional, and the weaknesses it may have exposed in Cerebellum are gone.

4.) Many draws and long lines without winning chances.

The concept for the Cerebellum_light release was to have a book which is completely calculated by Stockfish itself, without human intervention. So the name Brainfish is justified because Stockfish alone is playing, with a "Brain" which created new knowledge out of Stockfish calculated positions only with the graph recalculation of all scores. With that concept I was successfully playing engine tournaments over a year, and other users are successfull too. But with the improvement of Stockfish in the last month I saw an increase of the drawing rate not only in the book, but in Stockfish vs Stockfish games too. That seems to be a general problem, maybe we are really facing a draw problem regarding the best engines playing with good books, endgame tables and on fast computers. Of course I can manually change some lines, but I would like to do that automatically, and there already some ideas, like using Komodo evaluations in addition to the Stockfish ones. And finally using statistics deciding between moves with a similar score is a second possibility.

If you have questions please simply ask, my english is not the very best, so sometimes I'm not as clear in what I try to explain as I would like to be.

Thomas
Parent - By Venator (Silver) [nl] Date 2016-08-24 04:09 Edited 2016-08-24 04:12
Hi Thomas,

Thank you very much for your detailed reply. By the way, your English is very good :-).

Concerning my reaction that "there is more in chess computer life than Cerebellum" I was referring to my own book. What I meant to say was, that I don't want to concentrate on one book only, to make sure my own book scores well against that book. F.e. I won't skip the Ruy Lopez, because my book only score draws with it against the Cerebellum book. When I see an interesting idea played by a top GM, I might include it in my book, even if objectively speaking it could lead to a draw. IMO they will be interested to see this new idea in a book, it gives them the opportunity to see what the top engines think of this line. That is more important for me than the fact that it might hurt performance a little bit. Finally, I am not that interested in optimising for engine-engine blitz matches. That has never been my main objective.

In any case, I very much like the Cerebellum book concept. I know that in checkers the strongest books are completely computer generated, using the biggest endgame databases. That approach leads to unbeatable books of superb quality. In chess, however, I am of the opinion that the view of a strong human player is necessary to avoid certain lines to be played. I have seen quite a lot of lines where Stockfish thinks white has a small/clear advantage, but I simply disagree with that evaluation.

Kind regards, Jeroen
Parent - - By Peter Grayson (****) [gb] Date 2016-08-23 11:54
Match conditions as previous. Komodo 10.1 contempt parameter set = 0. There was no date stamp change to the Brainfish engine or the Cerebellum book after the previous match indicating no consequential changes therefore I saw no need to overwrite them.

The previous match had established there were very  maybe a couple if any killer lines. Both books are strong in their own right therefore the purpose of this match was to identify whether the Komodo 10.1 engine with Noomen Book would provide a different performance.

Games 1 to 50 highlighted some more salient points regarding narrow books.

Komodo 10.1 + Noomen Book went into an early 2-0 lead after winning game 10 but by game 20 BrainFish + Cerebellum had pulled it back including a win with Black. The early 20% win rate looked promising but had dropped back to 14% by game 50.

Komodo's first win came in game 6 in a Nimzo Indian with Cerebellum exhausted after 9..Qa5 with the Noomen book continuing to move 16. The scores looked marginally in favour for Komodo until BrainFish played 29..f4 after which the game moved in Komodo's favour.

Black wins are always interesting and BrainFish' win in game 16 came with early departures from both books after the Noomen Book's 8.Na3 that looked to be an added move to the book that may not be best when the scores indicated BrainFish quickly pulling the score through equal to advantage for itself. Perhaps 8.Na3 works with deep analysis but not so under these match time controls and allocated engine resources. Potential risk of further losses with no book alternative.

[Event "Cerebellum vs Noomen, 5m/40+5m/40+5m/40"]
[Site "TERMINATOR1"]
[Date "2016.08.23"]
[Round "16"]
[White "Komodo 10.1 64-bit"]
[Black "Brainfish 190816 64 POPCNT_CB"]
[Result "0-1"]
[ECO "C54"]
[Annotator "0.11;0.13"]
[PlyCount "176"]
[TimeControl "40/300:40/300:40/300"]

{Intel(R) Xeon(R) CPU E5-2687W 0 @ 3.10GHz 3092 MHz  W=32.3 plies; 10,890kN/s;
91,822,683 TBAs; Noomen.ctg  B=33.9 plies; 12,236kN/s; 11,009,486 TBAs} 1. e4 {
[%eval 0,0] [%emt 0:00:00] Black last book move} e5 {[%eval 1,0] [%emt 0:00:01]
} 2. Nf3 {[%eval 0,0] [%emt 0:00:00]} Nc6 {[%eval 1,0] [%emt 0:00:00]} 3. Bc4 {
[%eval 0,0] [%emt 0:00:00]} Nf6 {[%eval 1,0] [%emt 0:00:00]} 4. d3 {[%eval 0,0]
[%emt 0:00:00]} Bc5 {[%eval 1,0] [%emt 0:00:00]} 5. O-O {[%eval 0,0] [%emt 0:
00:00]} O-O {[%eval 1,0] [%emt 0:00:00]} 6. a4 {[%eval 0,0] [%emt 0:00:00]} d6
{[%eval 1,0] [%emt 0:00:00]} 7. c3 {[%eval 0,0] [%emt 0:00:00]} a6 {[%eval 1,0]
[%emt 0:00:00]} 8. Na3 {[%eval 0,0] [%emt 0:00:00] White last book move} Na5 {
[%eval 13,24] [%emt 0:00:07]} 9. b4 {[%eval 11,26] [%emt 0:00:19]} Nxc4 {
[%eval 13,28] [%emt 0:00:00]} 10. bxc5 {[%eval 6,27] [%emt 0:00:20] (Nxc4)} Na5
{[%eval 0,23] [%emt 0:00:05]} 11. h3 {[%eval 0,25] [%emt 0:00:01]} Re8 {[%eval
0,26] [%emt 0:00:11] (Be6)} 12. cxd6 {[%eval 0,26] [%emt 0:00:10]} Qxd6 {
[%eval -6,27] [%emt 0:00:00]} 13. Qc2 {[%eval 0,26] [%emt 0:00:04]} c5 {[%eval
-7,26] [%emt 0:00:10] (Be6)} 14. Rb1 {[%eval 0,25] [%emt 0:00:19] (Bd2)} Bd7 {
[%eval -13,24] [%emt 0:00:06] (h6)} 15. Nd2 {[%eval -10,26] [%emt 0:00:49] (d4)
} Qc6 {[%eval -12,27] [%emt 0:00:26] (Qe6)} 16. Ndc4 {[%eval 1,26] [%emt 0:00:
10]} Nxc4 {[%eval -9,28] [%emt 0:00:06]} 17. Nxc4 {[%eval 0,24] [%emt 0:00:00]}
b5 {[%eval -14,29] [%emt 0:00:05]} 18. Ne3 {[%eval 0,23] [%emt 0:00:03]} bxa4 {
[%eval -9,29] [%emt 0:00:07] (h6)} 19. c4 {[%eval 2,25] [%emt 0:00:08]} Rab8 {
[%eval -13,28] [%emt 0:00:13]} 20. Bd2 {[%eval 0,27] [%emt 0:00:00] (Ba3)} Qd6
{[%eval -44,24] [%emt 0:00:05]} 21. Nd5 {[%eval 0,25] [%emt 0:00:00]} Nxd5 {
[%eval -37,23] [%emt 0:00:02]} 22. cxd5 {[%eval 0,27] [%emt 0:00:02]} Rb5 {
[%eval -32,25] [%emt 0:00:21] (Bb5)} 23. Rbc1 {[%eval 0,25] [%emt 0:00:10]
(Be3)} Qg6 {[%eval -26,24] [%emt 0:00:06] (Rb3)} 24. Kh2 {[%eval 0,24] [%emt 0:
00:05]} Rb3 {[%eval -23,24] [%emt 0:00:00]} 25. Be3 {[%eval 0,24] [%emt 0:00:
06]} f5 {[%eval -22,26] [%emt 0:00:12]} 26. Bxc5 {[%eval 0,28] [%emt 0:00:00]}
h6 {[%eval -21,25] [%emt 0:00:14] (Rc8)} 27. Rfe1 {[%eval 0,24] [%emt 0:00:16]
(Rfd1)} Bb5 {[%eval -32,27] [%emt 0:00:36] (Rc8)} 28. Red1 {[%eval -7,27]
[%emt 0:00:32]} Rc8 {[%eval -30,28] [%emt 0:00:00]} 29. d6 {[%eval -3,27]
[%emt 0:00:11]} Kh7 {[%eval -31,28] [%emt 0:00:00]} 30. f3 {[%eval -10,27]
[%emt 0:00:13] (Qe2)} Bd7 {[%eval -45,25] [%emt 0:00:10] (Qg5)} 31. Qf2 {
[%eval 0,26] [%emt 0:00:04] (d4)} a3 {[%eval -51,24] [%emt 0:00:05]} 32. Rd2 {
[%eval 0,27] [%emt 0:00:00] (Rc2)} Qg5 {[%eval -63,26] [%emt 0:00:09] (f4)} 33.
Rcd1 {[%eval -12,28] [%emt 0:00:23] (h4)} Qf4+ {[%eval -82,25] [%emt 0:00:07]
(Ba4)} 34. Kh1 {[%eval 0,29] [%emt 0:00:05]} fxe4 {[%eval -91,27] [%emt 0:00:
03]} 35. dxe4 {[%eval -34,29] [%emt 0:00:12]} Rcb8 {[%eval -98,27] [%emt 0:00:
02]} 36. Qe2 {[%eval -10,22] [%emt 0:00:00] (Re2)} a5 {[%eval -123,29] [%emt 0:
00:13] (R8b7)} 37. Qf2 {[%eval -67,22] [%emt 0:00:04] (Qe1)} R8b5 {[%eval -129,
29] [%emt 0:00:08]} 38. Kg1 {[%eval -66,25] [%emt 0:00:00]} Rc3 {[%eval -131,
28] [%emt 0:00:03]} 39. Bb6 {[%eval -101,26] [%emt 0:00:04] (Ba7)} Qf7 {[%eval
-175,29] [%emt 0:00:13]} 40. Bd8 {[%eval -144,27] [%emt 0:00:00]} Rcb3 {[%eval
-163,32] [%emt 0:00:06]} 41. Bh4 {[%eval -169,30] [%emt 0:00:19]} Rb2 {[%eval
-184,35] [%emt 0:00:07] (a4)} 42. Kh2 {[%eval -173,31] [%emt 0:00:29] (Bg3)}
Qa2 {[%eval -220,30] [%emt 0:00:10] (R5b4)} 43. Qa7 {[%eval -193,28] [%emt 0:
00:13] (Bd8)} Rxd2 {[%eval -259,30] [%emt 0:00:08]} 44. Rxd2 {[%eval -209,30]
[%emt 0:00:00]} Qxd2 {[%eval -275,28] [%emt 0:00:04]} 45. Qxd7 {[%eval -214,31]
[%emt 0:00:01]} Rb8 {[%eval -294,29] [%emt 0:00:06]} 46. Qf5+ {[%eval -225,30]
[%emt 0:00:04]} Kh8 {[%eval -297,30] [%emt 0:00:00]} 47. d7 {[%eval -235,30]
[%emt 0:00:11]} a2 {[%eval -307,33] [%emt 0:00:11]} 48. Qxe5 {[%eval -245,30]
[%emt 0:00:00]} Rf8 {[%eval -318,31] [%emt 0:00:24]} 49. Qa1 {[%eval -256,30]
[%emt 0:00:00]} Qxd7 {[%eval -321,26] [%emt 0:00:04]} 50. Qxa2 {[%eval -254,28]
[%emt 0:00:01]} a4 {[%eval -332,31] [%emt 0:00:29]} 51. Bg3 {[%eval -273,29]
[%emt 0:00:00] (Qa3)} Ra8 {[%eval -345,32] [%emt 0:00:41]} 52. Qa3 {[%eval
-283,32] [%emt 0:00:00]} Qe6 {[%eval -362,25] [%emt 0:00:03]} 53. h4 {[%eval
-291,28] [%emt 0:00:09] (Bd6)} Kh7 {[%eval -383,28] [%emt 0:00:08]} 54. Bf4 {
[%eval -305,31] [%emt 0:00:13]} Qb3 {[%eval -404,29] [%emt 0:00:00]} 55. Qe7 {
[%eval -315,30] [%emt 0:00:12]} a3 {[%eval -418,32] [%emt 0:00:00]} 56. Be5 {
[%eval -329,31] [%emt 0:00:23] (Qd7)} Qg8 {[%eval -467,31] [%emt 0:00:39]} 57.
Ba1 {[%eval -337,34] [%emt 0:00:00]} a2 {[%eval -477,24] [%emt 0:00:01]} 58. h5
{[%eval -362,31] [%emt 0:00:29] (Qd7)} Qb8+ {[%eval -534,25] [%emt 0:00:05]
(Rf8)} 59. Kh3 {[%eval -373,31] [%emt 0:00:07]} Qf8 {[%eval -559,28] [%emt 0:
00:00]} 60. Qe6 {[%eval -476,30] [%emt 0:00:24] (Qe5)} Qc8 {[%eval -1579,28]
[%emt 0:00:08]} 61. Qf5+ {[%eval -652,32] [%emt 0:00:14]} Qxf5+ {[%eval -12337,
36] [%emt 0:00:07]} 62. exf5 {[%eval -1614,28] [%emt 0:00:00]} Ra3 {[%eval
-12338,38] [%emt 0:00:02]} 63. Kg4 {[%eval -25000,37] [%emt 0:00:04] (Kh2)} Rb3
{[%eval -12339,45] [%emt 0:00:08] (Rd3)} 64. Be5 {[%eval -25000,37] [%emt 0:00:
03] (Kf4)} Rb1 {[%eval -12341,47] [%emt 0:00:20] (Rb5)} 65. g3 {[%eval -25000,
39] [%emt 0:00:05] (Kf4)} a1=Q {[%eval -12343,49] [%emt 0:00:31]} 66. Bxa1 {
[%eval -25000,43] [%emt 0:00:00]} Rxa1 {[%eval -12344,44] [%emt 0:00:01]} 67.
Kh3 {[%eval -25000,37] [%emt 0:00:02]} Rh1+ {[%eval -12344,47] [%emt 0:00:01]}
68. Kg2 {[%eval -25000,39] [%emt 0:00:02]} Rxh5 {[%eval -12345,48] [%emt 0:00:
18]} 69. g4 {[%eval -25000,42] [%emt 0:00:00]} Rg5 {[%eval -12346,48] [%emt 0:
00:13]} 70. Kh3 {[%eval -25000,44] [%emt 0:00:00] (Kg3)} h5 {[%eval -12347,48]
[%emt 0:00:02]} 71. Kg3 {[%eval -25000,43] [%emt 0:00:02]} Kh6 {[%eval -12348,
52] [%emt 0:00:00]} 72. Kf4 {[%eval -25000,42] [%emt 0:00:10] (Kh4)} hxg4 {
[%eval -12348,39] [%emt 0:00:02]} 73. fxg4 {[%eval -25000,53] [%emt 0:00:01]}
g6 {[%eval -12349,66] [%emt 0:00:00]} 74. f6 {[%eval -25000,57] [%emt 0:00:03]}
Ra5 {[%eval -12350,74] [%emt 0:00:00] (Rb5)} 75. Ke4 {[%eval -25000,62] [%emt
0:00:04]} g5 {[%eval -12351,83] [%emt 0:00:00]} 76. Kf3 {[%eval -25000,73]
[%emt 0:00:05] (f7)} Kg6 {[%eval -12351,85] [%emt 0:00:03]} 77. Ke2 {[%eval
-25000,82] [%emt 0:00:02] (Kg3)} Kxf6 {[%eval -12352,56] [%emt 0:00:02]} 78.
Ke3 {[%eval -32745,0] [%emt 0:00:00]} Ke5 {[%eval -32745,0] [%emt 0:00:00]} 79.
Kd3 {[%eval -32747,1] [%emt 0:00:00]} Ra4 {[%eval -32747,1] [%emt 0:00:00]} 80.
Kd2 {[%eval -32749,0] [%emt 0:00:00]} Rxg4 {[%eval -32749,0] [%emt 0:00:00]}
81. Kd3 {[%eval -32751,0] [%emt 0:00:00]} Rf4 {[%eval -32751,0] [%emt 0:00:00]}
82. Kc2 {[%eval -32753,0] [%emt 0:00:00]} g4 {[%eval -32753,0] [%emt 0:00:00]}
83. Kb2 {[%eval -32755,0] [%emt 0:00:00]} g3 {[%eval -32755,1] [%emt 0:00:00]}
84. Ka2 {[%eval -32757,0] [%emt 0:00:00]} Rf3 {[%eval -32757,0] [%emt 0:00:00]}
85. Kb2 {[%eval -32759,1] [%emt 0:00:00]} g2 {[%eval -32759,1] [%emt 0:00:00]}
86. Ka2 {[%eval -32761,0] [%emt 0:00:00]} g1=Q {[%eval -32761,0] [%emt 0:00:00]
} 87. Kb2 {[%eval -32763,1] [%emt 0:00:00]} Qh2+ {[%eval -32763,0] [%emt 0:00:
00]} 88. Ka1 {[%eval -32765,0] [%emt 0:00:00]} Rf1# {[%eval -32765,1] [%emt 0:
00:00]} 0-1

In these first 50 games there were 8 games exactly the same. Games 20, 22, 30, 36, 40, 42, 44 and 48 all giving the same 26 move draws in a C90 Ruy Lopez opening with the White line being played by the Komodo engine. The BrainFish engine did not play any moves with them all coming from the Cerebellum book. The Noomen Book managed to move away from this in game 50 and obtained a win to pull the score back to 4-3 in favour of BrainFish + Cerebellum book. It will be interesting to see if the Cerebellum book has an answer to this in the remaining games to be played.

PeterG
Attachment: CerebellumvsKomodowithNMNBookG1-50.zip - Games 1 to 50 (55k)
Parent - By Venator (Silver) [nl] Date 2016-08-23 13:31
Not a bad result by Komodo sofar!

Some comments: this Ruy line that was repeated several times, can lead to interesting play if black plays differently, but Cerebellum's line is just a draw. So I changed this line. The Na3!? novelty in the Giuoco Piano is inspired by Anand, who used these Na3 ideas in the Candidates tournament. His idea, however, didn't get any followers, so I guess it was a one time inspiration. In later games GM's simply kept on playing Nbd2, which seems more logical and better.

In any case, your matches give me a lot of room for improvements. Very nice :-).
Parent - - By Peter Grayson (****) [gb] Date 2016-08-23 23:36
Games 51 to 100 saw BrainFish + Cerebellum increase its lead over Komodo 10.1 + Noomen.ctg with a score of 7-3 with 40 draws.

The immediate point of interest here was whether the Cerebellum book had any answer to the line played in game 50. It did not, losing games 52, 54 and 56 with Black when as expected the Noomen Book continued to play the same line. So what happened in game 58? Move 13 is what happened when already out of book, Komodo played 13.Bc4 instead of the previously played 13.h3 and went on to lose. It had to be move 13!

In case Mr. Shrapnel reads this, this is a prime example of  the consequence of  SMP variability that in this case completely reversed the result. Further consequence was that after 2 drawn games with Queens Pawn openings the book reverted back to the C90 26 move draw after Which it changed to the alternative 8.c3 giving a C89 line and that draw kicked it out of 1.e4 until game 76. The winning C90 line seems out of favour for now because of that loss.

After that initial flurry of winning activity, two further wins came from Brainfish with the Cerebellum book as White against the Caro-Kaan in games 69 and 83. After two earlier draws with 9.Qe2, the Cerebellum book changed to 9.0-0 but as can be seen it looked to be just a transposition but was sufficient to throw the Noomen book off guard gaining the extra half point.

Just as the Komodo 10.1 + Noomen book combination had a 4 win flourish from game 50, the BrainFish + Cerebellum combination had 3 succesive wins with White in games 95, 97 and 99 with game 99 being another Caro-Kaan loss for the Komodo + Noomen book.

Game 95 was a QGD that saw Komodo go downhill fairly quickly after exiting book after move 12.

The game 97 win suggested Komodo was lost for the correct move with the early departure from book in a Giuoco Pianissimo when perhaps 8..Bg4 should have received a green move mark to avoid the dubious 8..g5 played by Komodo.

BrainFish + Cerebellum now has a 5 game advantage.

PeterG
Attachment: CerebellumvsKomodowithNMNBookG51-100.zip - Games 51 to 100 (62k)
Parent - - By Venator (Silver) [nl] Date 2016-08-24 06:56
Interestingly, Komodo often disagrees with the SF eval. Where SF says "clear advantage", Komodo's eval is close to 0.00.

When I have fixed the weaker lines in my book, it would be interesting to repeat the match.

BTW, what puzzles me, is that I have 3.Bc4 as an alternative to the Ruy, but the book never seems to play this, even after a huge number of draws in the Marshall gambit. I don't know why the CB software doesn't switch to 3.Bc4 at some point.
Parent - - By Peter Grayson (****) [gb] Date 2016-08-24 09:41

> When I have fixed the weaker lines in my book, it would be interesting to repeat the match.


I'll certainly consider running that if you wish although the weather has warmed up here again so I was glad the match finished this morning because the dual Xeon machine does generate much heat.

Please also satisfy yourself that the wins and losses were not just consequential to the faster time control used that is close to 40 moves in 15 minutes for CCRL and CEGT references.

I used the match games as the database for the book to learn from Komodo 10.1 64-bit. The "Prob" column shows that the likelihood of 3.Bc4 being played was very slim from the outset and further reduced based on the match results that is indicated in the [%] column. Check game 16 when it lost with 3.Bc4 and later drew in game 80. The probability of the line being played looks consistent with 2 games in the 75 played as White here. Bear in mind I used a fresh copy of the book for the second match so there was no learning influence from the previous match.

Peter
Parent - By Venator (Silver) [nl] Date 2016-08-24 09:48
Hi Peter,

Indeed it is getting a lot hotter, same over here. The coming 5 days we have 30 degrees centigrade or more.

One of my testers asked me to do a retest, he is currently running it (albeit on a much slower machine). I'll post the result later.

The most interesting fact from the Komodo match is that you'll often see a clear discrepancy in evaluation. A Brainfish-SF match might not put the finger on the weak spots in SF's eval, hence this second match was very interesting. Furthermore, Komodo might play some lines better than SF and vice versa. An example is the Caro-Kann, where Komodo badly goes astray. SF finds the correct line of play quite quickly.

Anyway, once again thanks for your time and testing my book!

Kind regards, Jeroen
Parent - - By Peter Grayson (****) [gb] Date 2016-08-24 09:12
In games 101 to 150 BrainFish + Cerebellum increased its lead with 5 wins, 2 losses with 43 draws for an overall margin of 8 wins with a score of  +16 =128 -8. Win rate = 16%

Game 103 saw BrainFish win in a Giuoco Piano with both books exiting by move 9 and BrainFish gaining a spatial advantage allowing better mobility of pieces behind its pawn wall to achieve a very nice attack on Komodo's King defences.

Game 107 and 141 was the fourth and fifth loss by Komodo in the same Caro-Kaan line.

Game 122 gave Komodo a win in a Nimzo-Indian, Samisch variation that both sides scored even around the first time control and with opposite coloured Bishops but with both rooks still on the board it looked to be heading towards a draw but here it is worth looking at the subtlety of Komodo's play to get the win. Still not sure I fully understand it!

Game 128 gave Komodo its last win in a QGD decline 5.Bf4 variation. The Noomen book provided Komodo with an extra 6 moves. There was not much in it after books were exhausted but perhaps the sequence of 0.00 scores by BrainFish to 29..Re7 suggested it may have missed something that Komodo was able to work on.

Game 143 saw a rare Sicilian and it appears the addition of 12..Be7 to the Noomen book may not be so good when 12..Nc5 may be better. BrainFish showed a score of +1.13 out of book for 13.a3 and Komodo did not look comfortable after the Noomen book's 14..d5. BrainFish went on to win.

Both matches provided some interesting games and the high number of draws indicated there is not too much between the books. Important to point out that different engines are likely to get different results but it does not necessarily mean the losing lines were bad when perhaps it is just the case under these match conditions the engines are unable to provide better play with the lines.

There has obviously been much work put into these books for our enjoyment so it just leaves me to thank Jeroen Noomen and Stephan and Thomas Zipproth for their work when I hope they see any comments I have made as positive rather than negative criticism.

PeterG
Parent - By Venator (Silver) [nl] Date 2016-08-24 09:41
Hi Peter,

Thanks a lot for the effort. All losses are in lines that were indeed not optimally checked in my book. And I also found a few unintended priorities, like the quick Sicilian loss. The Caro-Kann losses are rather strange, it appears that Komodo doesn't know the correct line of play. In my own tests this Caro-Kann line now always leads to a draw.

The past three days I have been patching the suboptimal lines and I have asked one of my testers to do another test in the same setting.

Jeroen
Parent - By Thomas Zipproth [de] Date 2016-08-24 12:41
Hi Peter,

thanks a lot for your efforts too. Without testing and feedback progress would not be possible.
I'm updating the Cerebellum book too, let's see how things develop.
I will reactivate Sicilian when the calculations are finished, it still needs some days I think.

Thomas
Parent - - By Venator (Silver) [nl] Date 2016-08-24 14:17 Edited 2016-08-24 14:31
What a little bit of tweaking can do :-). I just received this from one of my testers:

Komodo 10.1 (+NoomenBeta.ctg) - Brainfish 190816 (+Cerebellum_light)  26-24  (+5-3=42, 52%)

Setup: Corei7, tempo game in 5', no TB's.

Games below.
Attachment: Noomen_Cerebellum.pgn (249k)
Parent - - By Peter Grayson (****) [gb] Date 2016-08-24 20:19
A remarkable match score turnaround in such a short time!

Looking at some of the games I wonder if Komodo has more SMP variability than other engines? After a draw and 2 wins in the C54 Giuoco line in games 8,16 and 20 then came the loss in game 24 when it changed first move out of book from 16.Qe1 to Qe2 similar to what happened in the B90 Ruy Lopez games in my match. In analysis on my older O/C'd quad with 4 Gb hash and 256 Mb Table memory, it also considers 16.Ba4. I also noted the 0.00 evaluation my have had some influence on both engines after move 16 when for example in game 24 Komodo had 26.c4 at 0.00 but BrainFish' reply was scored at 26..Rg5 -0.96/24.

Stockfish 190816 shows less variability sticking with 16.Qe2 albeit with a 0.00 score up to 28 ply but then has a very long "think" before changing to 16.d4 at 29 ply but the strange thing is the score stays at 0.00 so no indication why it changed.

Cannot help but wonder then on the reliability of the engines' analysis!?

Need to have a look at what is going on in deeper detail.

Peter

24: Komodo 10.1 64-bit - Brainfish 190816 64 POPCNT, Blitz 5' Noomen.ctg test 2016
r3r1k1/1pp1qpp1/p2pbnnp/P3p3/2N1P3/1BPPP2P/1P1N2P1/R2Q1RK1 w - - 0 1


Analysis by Komodo 10.1 64-bit:

16.Na3 Rad8
  =  (-0.22 --)   Depth: 7   00:00:00  17kN
16.Na3 Rad8 17.Nf3 Bd7 18.Nc2 Qf8 19.Nd2
  =  (-0.22)   Depth: 7   00:00:00  20kN
16.Na3 Rad8
  =  (-0.30 --)   Depth: 8   00:00:00  23kN
16.Nf3 Rf8 17.Ncd2
  =  (-0.14 ++)   Depth: 8   00:00:00  33kN
16.Nf3 Rf8 17.Ncd2
  =  (-0.06 ++)   Depth: 8   00:00:00  35kN
16.Nf3 Rad8
  =  (-0.14 --)   Depth: 8   00:00:00  40kN
16.Nf3 Rad8 17.Ncd2 d5 18.exd5 Bxd5 19.Bxd5 Rxd5 20.Qb1
  =  (-0.14)   Depth: 8   00:00:00  41kN
16.Nf3 Rad8
  =  (-0.22 --)   Depth: 9   00:00:00  43kN
16.Nf3 Rad8
  =  (-0.30 --)   Depth: 9   00:00:00  47kN
16.Na3 Rad8 17.Bxe6
  =  (-0.14 ++)   Depth: 9   00:00:00  66kN
16.Na3 Bxb3
  =  (-0.22 --)   Depth: 9   00:00:00  72kN
16.Na3 Bxb3 17.Qxb3 c6 18.Nc2 Rad8 19.Rad1 Nf8 20.Qa3 Qc7
  =  (-0.22)   Depth: 9   00:00:00  75kN
16.Na3 Rad8 17.Bxe6 Qxe6 18.c4 Nd7 19.b4 Nf6 20.Nc2 Nh4 21.Qe1
  =  (-0.19)   Depth: 10   00:00:00  98kN
16.Na3 Rad8 17.Bxe6
  =  (-0.12 ++)   Depth: 11   00:00:00  139kN
16.Na3 Rad8
  =  (-0.19 --)   Depth: 11   00:00:00  152kN
16.Na3 Rad8 17.Bxe6 Qxe6 18.c4 c6 19.Qf3 d5 20.cxd5 cxd5 21.Nc2 Nf8 22.exd5
  =  (-0.22)   Depth: 11   00:00:00  176kN
16.Na3 Rad8
  =  (-0.29 --)   Depth: 12   00:00:00  234kN
16.Nf3 Rad8 17.Qc2 Bxc4 18.Bxc4 d5 19.Bxd5 Nxd5 20.exd5 Qc5 21.c4 Qxe3+ 22.Rf2 c6 23.Re1 Qc5 24.dxc6 bxc6
  =  (-0.21)   Depth: 12   00:00:00  391kN
16.Nf3 Rad8
  =  (-0.28 --)   Depth: 13   00:00:00  478kN
16.Qe1 Rad8 17.Ba4 Bd7 18.Bxd7 Qxd7 19.Rc1 Qc6 20.Qf2 d5 21.exd5 Qxd5 22.d4 e4
  =  (-0.21)   Depth: 13   00:00:00  790kN
16.Qe1 Rad8 17.Ba4 Bd7 18.Bxd7 Qxd7 19.Rf3 d5 20.exd5 Qxd5 21.e4 Qc5+ 22.Kh1 Qc6 23.b4 Nf4 24.d4
  =  (-0.24)   Depth: 14   00:00:00  1077kN
16.Qe1 Rab8 17.Bc2 Rbd8 18.Ba4 Bd7 19.Bc2 Bc6 20.Rd1 Qe6 21.Qf2 d5 22.exd5 Bxd5 23.Ba4 Rf8
  =  (-0.21)   Depth: 15   00:00:01  2607kN
16.Qe1 Rad8 17.Ba4 Bd7 18.Bc2 Bb5 19.Qf2 Qd7 20.b4 d5 21.exd5 Qxd5 22.Rad1 Qe6 23.Qf3 Nd5 24.Ne4
  =  (-0.22)   Depth: 16   00:00:01  3651kN
16.Qe1 Rad8 17.Ba4 Bd7 18.Bc2 Bb5 19.Rd1 d5 20.exd5 Rxd5 21.Qf2 Rdd8 22.d4 e4 23.Bb3 Qd7 24.Qg3 Kh7
  =  (-0.18)   Depth: 17   00:00:02  5747kN
16.Qe1 Rad8
  =  (-0.24 --)   Depth: 18   00:00:02  6969kN
16.Ba4 Red8 17.Qe1 Rac8 18.Bc2 Nh7 19.Qf2 Nh4 20.d4 Qg5 21.Kh1 Nf6 22.d5 Bd7 23.b4 Ng6 24.Bd3 Bb5 25.Nb2 Bxd3 26.Nxd3
  =  (-0.19)   Depth: 18   00:00:03  10538kN
16.Ba4 Red8 17.Qe1 Rac8 18.Bc2 Nd7 19.d4 Nf6 20.Rc1 Bd7 21.Na3 Qf8 22.Rd1 Kh7 23.Bd3 Re8 24.b4 Red8 25.d5
  =  (-0.19)   Depth: 19   00:00:03  12082kN
16.Ba4 Red8 17.Qe2 Rac8 18.Bc2 Nh4 19.Na3 Kh7 20.Rad1 Ng6 21.Qe1 Kg8 22.Bb3 Bxb3 23.Nxb3 d5 24.exd5 Nxd5 25.Nc2 Nf6 26.d4
  =  (-0.18)   Depth: 20   00:00:10  35924kN
16.Ba4 Red8 17.Bc2 Nh4 18.Qe2 Nh7 19.Kh1 Qg5 20.Qf2 Nf6 21.b4 Re8 22.Bb3 Rad8 23.Nb2 Nh5 24.g4 Nf6 25.Bxe6 Rxe6 26.Rae1 Ree8
  =  (-0.15)   Depth: 21   00:00:13  46918kN
16.Ba4 Red8
  =  (-0.21 --)   Depth: 22   00:00:16  58909kN
16.Qe1 Rad8 17.Ba4 Bd7 18.Bc2 Bb5 19.Na3 Bc6 20.Qf2 Nf8 21.Nac4 Bd7 22.b4 Ng6 23.Bb3 Be6 24.Bc2 d5 25.exd5 Bxd5 26.d4 exd4 27.exd4 Qe2
  =  (-0.13)   Depth: 22   00:00:21  77245kN
16.Qe1 Rad8 17.Ba4 Bd7 18.Bc2 Bb5 19.Na3 Bc6 20.Qf2 Nf8 21.Nac4 Bd7 22.b4 Ng6 23.Qe1 Be6 24.Ba4 Nd7 25.Qe2 Nh4 26.Bb3 Qg5 27.Nf3
  =  (-0.13)   Depth: 23   00:00:22  83197kN
16.Qe1 Rad8 17.Ba4 Bd7 18.Bc2 Bb5 19.Na3 Bc6 20.Qf2 Qd7 21.Bb3 Ba4 22.Bxa4 Qxa4 23.Nac4 Qd7 24.Qf3 Qe6 25.b4 d5 26.exd5 Qxd5 27.Qxd5 Rxd5 28.Nb2 e4
  =  (-0.17)   Depth: 24   00:00:25  93175kN
16.Ba4 Rf8 17.Qe1 Rad8 18.Bc2 Nh7 19.b4 Nf6 20.Rb1 Rc8 21.Ra1 Rfd8 22.Rc1 Qe8 23.Ra1 Qf8 24.Rc1 Qe7 25.Ra1 Re8 26.Rd1 Rcd8 27.Ba4 Rf8
  =  (-0.13)   Depth: 24   00:00:28  104MN
16.Ba4 Rf8 17.Qe1 Rad8 18.Bc2 Qd7 19.b4 Qe7 20.Rb1 Rc8 21.Ra1 Rfd8 22.Rc1 Qe8 23.Ra1 Qf8 24.Nb2 Qe7 25.c4 Nh7 26.d4 Nh4 27.Nf3 Nxf3+ 28.Rxf3 Qg5
  =  (-0.12)   Depth: 25   00:00:43  163MN
16.Ba4 Rf8 17.Qe1 Rad8 18.Bc2 Rc8 19.b4 Rfd8 20.Rc1 Re8 21.Rd1 Qd7 22.Ra1 Red8 23.Qf2 Qe7 24.Bb3 Qe8 25.Qe1 d5 26.exd5 Bxd5 27.e4 Bxc4 28.Bxc4 Nf4 29.Qe3
  =  (-0.11)   Depth: 26   00:01:05  245MN
16.Ba4 Rf8 17.Qe1 Rfc8 18.Bc2 Rd8 19.b4 Rac8 20.Rc1 Rf8 21.Ra1 Rfe8 22.Qf2 Rcd8 23.Qe1 d5 24.exd5 Bxd5 25.Qf2 Be6 26.Rac1 c5 27.Rb1 cxb4 28.Rxb4 e4 29.d4 Qc7 30.Rfb1
  =  (-0.10)   Depth: 27   00:01:15  285MN
16.Ba4 Rf8 17.Qe1 Rfc8 18.Bc2 Rd8 19.b4 Rac8 20.Rc1 Rf8 21.Ra1 Rfe8 22.Qf2 Rcd8 23.Qg3 Bd7 24.Qe1 Bc6 25.Qf2 Rf8 26.Qe1 Kh7 27.Rd1 d5 28.exd5 Nxd5 29.Ne4 f5 30.Nc5
  =  (-0.10)   Depth: 28   00:01:31  348MN
16.Ba4 Rf8 17.Qe1 Rfc8 18.Bc2 Rd8 19.b4 Rac8 20.Rc1 Qe8 21.Ra1 Qd7 22.Qf2 Qe7 23.Bb3 Re8 24.Bc2 Rb8 25.Bb3 Rbd8 26.Bc2 Rf8 27.Qe1 Nh7 28.Qg3 Nh4 29.d4 Rfe8 30.Nb2
  =  (-0.11)   Depth: 29   00:01:52  426MN
16.Ba4 Bd7 17.Bxd7 Qxd7 18.Qf3 Rad8 19.b4 Re6 20.Rf2 Ne7 21.d4 Ng6 22.d5 Ree8 23.Qe2 Rc8 24.Qd3 Nh4 25.Qe2 Kh7 26.Qd3 Qe7 27.Qc2 Qd8 28.Qd3 Rf8 29.Qc2 Kg8 30.Nf3 Nxf3+ 31.Rxf3 Re8 32.Qd3
  =  (-0.09)   Depth: 30   00:05:07  1202MN
16.Ba4 Rf8 17.Bc2 Nh4 18.Kh2 Nd7 19.b4 Nf6 20.Qe1 Ng6 21.Kg1 Rad8 22.Rc1 Nd7 23.d4 Nf6 24.d5 Bd7 25.Bd3 Bb5 26.Qe2 Rfe8 27.Ra1 Rc8 28.Rf2 Nf8 29.Raf1 Qd8 30.Qd1 Ng6 31.Qe2
  =  (-0.07)   Depth: 31   00:06:07  1451MN
16.Ba4 Rf8 17.Bc2 Rad8 18.Qe1 Rfe8 19.Qg3 Bd7 20.Qf2 Bc6 21.Rac1 Rf8 22.Rce1 Bd7 23.Qe2 Nh4 24.Qf2 Nh5 25.Kh2 Ng6 26.Ra1 Nf6 27.Kg1 Rfe8 28.Rac1 Qe6 29.Na3 Bc6 30.Bb3 d5 31.exd5 Bxd5 32.Bc2
  =  (-0.09)   Depth: 32   00:07:06  1693MN

(Grayson, Newport, South Wales 24.08.2016)

24: Komodo 10.1 64-bit - Brainfish 190816 64 POPCNT, Blitz 5' Noomen.ctg test 2016
r3r1k1/1pp1qpp1/p2pbnnp/P3p3/2N1P3/1BPPP2P/1P1N2P1/R2Q1RK1 w - - 0 1


Analysis by Stockfish 190816 64 :

16.d4 Qd8 17.Ba4 Rf8 18.d5 Bd7 19.Bxd7 Qxd7
  =  (0.03)   Depth: 7/7   00:00:00  18kN
16.d4 d5 17.exd5 Bxd5 18.Bc2 exd4 19.cxd4 Ne4
  =  (-0.01)   Depth: 8/8   00:00:00  31kN
16.Ba4 Rf8 17.Qe2 Rad8 18.Bc2 Bd7 19.Qf3 Bb5 20.Qd1
  =  (-0.02)   Depth: 9/12   00:00:00  78kN
16.Ba4 Rf8 17.Qe2 Rad8 18.Bc2 Rde8 19.Ba4 Rd8
  =  (0.00)   Depth: 10/12   00:00:00  92kN
16.Ba4 Red8 17.Bc2 Re8 18.Ba4
  =  (0.00)   Depth: 11/12   00:00:00  141kN
16.Ba4 Bd7 17.Bxd7 Qxd7 18.Qf3 Rad8 19.b4 d5 20.exd5 Qxd5 21.Qxd5 Nxd5 22.Ra3
  =  (-0.03)   Depth: 12/14   00:00:00  187kN
16.Ba4 Bd7 17.Bxd7 Qxd7 18.Qf3 Rad8 19.Rad1 Nh4 20.Qf2 Ng6 21.Qf3
  =  (0.00)   Depth: 13/16   00:00:00  242kN
16.Ba4 Rf8 17.Qe2 Rad8 18.Bc2 Bd7 19.d4 Rfe8 20.Qd1 Bb5 21.Bd3 Bc6 22.d5 Bb5 23.b4
  =  (-0.10)   Depth: 14/19   00:00:00  675kN
16.Ba4 Red8 17.Qe2 Rf8 18.Rae1 Rad8 19.Bb3 Rfe8 20.Ba4 Rf8
  =  (0.00)   Depth: 15/19   00:00:00  954kN
16.Ba4 Red8 17.Qe2 Rf8 18.Rae1 Rad8 19.Bb3 Rfe8 20.Ba4 Rf8
  =  (0.00)   Depth: 16/20   00:00:01  1461kN
16.Ba4 Rf8 17.b4 Rad8 18.Bb3 d5 19.exd5 Bxd5 20.Qe2 e4 21.d4 Bc6 22.Rac1 Bb5 23.Rf5 Rd5 24.Rcf1 Rfd8 25.R1f2
  =  (-0.08)   Depth: 17/23   00:00:01  5129kN
16.d4 Rad8 17.d5 Bd7 18.Ba4 Bb5 19.Bxb5 axb5 20.Na3 Qd7 21.Nc2 Ra8 22.Nb4 Qc8 23.Qe2 Qd7 24.a6 bxa6 25.Nxa6 Ne7 26.Nb4 Rxa1
  =  (-0.01)   Depth: 18/25   00:00:02  10206kN
16.Qe2 Nh7 17.Nf3 Rad8 18.Ba4 Rf8 19.Ncd2 Nf6 20.Bc2 c5 21.Qf2 Qc7 22.d4 cxd4 23.exd4 Nf4 24.Qe3 Bxh3 25.gxh3
  =  (-0.04)   Depth: 19/27   00:00:04  18432kN
16.Qe2 Nh7 17.Nf3 Rab8 18.Rae1 Nf6 19.Nfd2 Rbd8 20.Ba4 Bd7 21.Bc2 Nh4 22.d4 Ng6 23.b4 Bc6 24.Nb2 d5 25.exd5 exd4
  =  (-0.03)   Depth: 20/27   00:00:04  21264kN
16.Qe2 Nh7 17.Qf2 Rad8 18.Qg3 Qg5 19.Qxg5 Nxg5 20.Kh2 Rf8 21.Bc2 f6 22.Bd1 Ne7 23.Be2 Rfe8 24.d4 d5 25.exd5 Nxd5 26.Bh5 Rf8
  =  (-0.01)   Depth: 21/29   00:00:07  39521kN
16.Qe2 Nh7 17.Qf2 Rad8 18.Bc2 Ng5 19.Kh1 Rf8 20.b4 Nh7 21.d4 Nf6 22.d5 Bd7 23.Qe2 Rfe8 24.Bd3 Bb5 25.Kg1 Nh4 26.Rf2 Ng6 27.Raf1
  =  (0.00)   Depth: 22/29   00:00:09  48280kN
16.Qe2 Nh7 17.Qf2 Rad8 18.Bc2 Ng5 19.Kh1 Nh7 20.Kg1
  =  (0.00)   Depth: 23/29   00:00:10  51601kN
16.Qe2 Nh7 17.Qf2 Rad8 18.d4 Nf6 19.Bc2 Bd7 20.Rae1 Bc6 21.b4 Bb5 22.Qg3 Nh5 23.Qf2 Nf6
  =  (0.00)   Depth: 24/33   00:00:16  83707kN
16.Qe2 Nh7 17.Qf2 Rad8 18.d4 Nf6 19.Rae1 Bd7 20.Bc2 Nh5 21.Qf3 Nf6 22.Qg3 Bb5 23.b4 Nh5 24.Qf2 Nf6 25.Qg3
  =  (0.00)   Depth: 25/33   00:00:16  85878kN
16.Qe2 Nh7 17.Qf2 Rad8 18.d4 Nf6 19.Rae1 Bd7 20.Bc2 Nh5 21.Qf3 Nf6 22.Qg3 Bb5 23.b4 Nh5 24.Qf2 Nf6 25.Qg3
  =  (0.00)   Depth: 26/33   00:00:19  103MN
16.Qe2 Nh7 17.Qf2 Rad8 18.Bc2 Rf8 19.Qg3 Qg5 20.Qxg5 Nxg5 21.Kh2 Nh7 22.Bb3 Ng5 23.Bc2
  =  (0.00)   Depth: 27/39   00:00:34  183MN
16.Qe2 Nh7 17.Qf2 Rad8 18.Bc2 Rf8 19.Rae1 Ng5 20.Qg3 h5 21.Rf5 Nh7 22.Rxh5 Nf6 23.Rg5 Nh7 24.Rh5
  =  (0.00)   Depth: 28/39   00:00:38  206MN
16.d4 Rad8 17.d5 Bd7 18.Ba4 Bxa4 19.Qxa4 Nh5 20.Qd1 Qh4 21.Qe1 Qxe1 22.Raxe1 Ng3 23.Rf3 Nh5 24.Rff1
  =  (0.00)   Depth: 29/46   00:03:48  1241MN
16.d4 Rad8 17.d5 Bd7 18.Ba4 Bxa4 19.Qxa4 Nh5 20.Qd1 Qh4 21.Qe1 Qxe1 22.Raxe1 Ng3 23.Rf3 Nh5 24.Rff1
  =  (0.00)   Depth: 30/46   00:04:04  1327MN
16.d4 Rad8 17.d5 Bd7 18.Ba4 Bxa4 19.Qxa4 Nh5 20.Qd1 Qh4 21.Qe1 Qxe1 22.Raxe1 Ng3 23.Rf3 Nh5 24.Rff1
  =  (0.00)   Depth: 31/46   00:04:15  1387MN
16.d4 Rad8 17.d5 Bd7 18.Ba4 Bxa4 19.Qxa4 Nf8 20.b4 N8d7 21.Rf5 Nh7 22.Qc2 Qh4 23.Nf3 Qe7 24.Nfd2
  =  (0.00)   Depth: 32/46   00:04:52  1593MN
16.d4 Rad8 17.d5 Bd7 18.Ba4 Bxa4 19.Qxa4 Nf8 20.b4 N8d7 21.Rf5 Nh7 22.Qc2 Qh4 23.Nf3 Qe7 24.Nfd2
  =  (0.00)   Depth: 33/46   00:05:15  1721MN
16.d4 Rad8 17.d5 Bd7 18.Ba4 Bxa4 19.Qxa4 Nf8 20.b4 N8d7 21.Rf5 Nh7 22.Qc2 Qh4 23.Nf3 Qe7 24.Nfd2
  =  (0.00)   Depth: 34/49   00:06:40  2185MN
16.d4 Rad8 17.d5 Bd7 18.Ba4 Bxa4 19.Qxa4 Nf8 20.b4 N8d7 21.Rf5 Nh7 22.Qc2 Qh4 23.Nf3 Qe7 24.Nfd2
  =  (0.00)   Depth: 35/49   00:08:00  2628MN

(Grayson, Newport, South Wales 24.08.2016)
Parent - By Venator (Silver) [nl] Date 2016-08-25 15:15
I am quite surprised that Komodo scores so well with my book, playing against Brainfish + Cerebellum. After some more tweaks, a second match ended 5-2 in favour of Komodo, with 43 draws. The total score after 100 games is thus 10-5, with 85 draws (52.5-47.5).

My move selections were made by using SF as an analysis engine, so I thought it would be more or less a "Stockfish book". But Komodo seems to like it as well :-).
Parent - - By Venator (Silver) [nl] Date 2016-08-26 04:22 Edited 2016-08-26 04:53
A third match was aborted after 13 games, when Komodo+Noomenbeta.ctg was leading 6-0 (with 7 draws) against Brainfish. The latter stumbled into a bad line and kept repeating it 5 times, losing all 6 games. It seems to me that the light version of Cerebellum sometimes doesn't have alternatives available when a line fails.

It is getting time I buy Komodo. It seems excellent in pointing out errors in SF's eval.
Parent - - By Thomas Zipproth [de] Date 2016-08-26 09:52 Edited 2016-08-26 09:58
That's interesting, I never saw such results in practical play, but as I pointed out, Cerebellum can only learn with calculating the lost games and adding it to the graph. It cannot adjust himself during play, because it does not use statistics on the fly.

That means of course, if you detect a loosing line in Cerebellum, and let your book adjust to it, you can also score 1000-0 against Cerebellum, but that does not mean a serious issue, except of course, that the variability of the played moves of Cerebellum is a bit too low at the moment. The next step in Cerebellum would be to play more different lines, which means to select also moves with a lower score than the best one, and to find a good selection method for that moves, perhaps a combination of score and statistics.

The main issue here is I think, that you cannot change or view Cerebellum light yourself, I hope to make progress with the full version soon.

Recent results from Playchess still showed results like +12, = 80, -1, so that issue seems not to happen in real competitions, where the library is updated about every second day.

I also never saw such a result when testing against other books, but I normally recalculate Cerebellum after each run of about 100 games.

Of course that is an ongoing process, and the updated book I released today may show different results. At the moment I have no free computer for making tests with your book, but it would be an interesting competition to see how the different methods  behave. Perhaps you can release your updated book sometimes, so that someone can make new tests.

Thomas
Parent - - By Peter Grayson (****) [gb] Date 2016-08-26 10:14

> That's interesting, I never saw such results in practical play,


> I also never saw such a result when testing against other books


That was one of the draw backs that I picked up in the very first match I posted in this thread. Everything seemed to be going well for the BrainFish + Cerebellum combination against the same BrainFish engine without Cerebellum but paired with the 5 year old Vesely.ctg openings book.

Game 83 saw BrainFish + Cerebellum lose with Black in a B97 Sicilian Poisoned Pawn variation and having identified that, the Vesely.ctg book registered 11.5 - 0.5 as White with that opening in the latter part of the game. Therefore in that match it took just 83 games for exactly that situation to arise. I commented that  the Chessbase book learning capability to reduce the probability of poor scoring lines in favour of better scoring lines during a match is a significant feature that gave it an advantage over the Cerebellum_Light book in that particular match.

PeterG
Parent - By Thomas Zipproth [de] Date 2016-08-26 13:05
Yes, I see that this can be a problem in testing. What I do for now is that I calculate and add every lost game that is published to the Cerebellum book. That is an automated process with not much work involved. So for example the problem with the Vesely book should have gone, but I had not the time to test it.

Every new book release is recalculated with the test results and games I'm aware of and with the user feedback, so it continuously improves.
A loosing line is of course still possible, but it should become more and more rare.

There are also different types of bad lines I think. For example many of the positions in Cerebellum are calculated with a time of about 70 seconds and endgame tablebases on a fast quadcore. so in a 5+1 match without endgame tablebases it may happen that the engine playing is not able to continue with the correct moves, even if they exist, because the thinking time is too short.

Thomas
Parent - By Peter Grayson (****) [gb] Date 2016-08-26 09:53
According to Thomas Zipproth's comments here, the full Cerebellum package will afford a high degree of flexibility and I will be curious to see what it offers compared to that currently available.

> It is getting time I buy Komodo. It seems excellent in pointing out errors in SF's eval.


Just my perception but I would describe Stockfish as the more aggressive engine capable of finding some very deep key moves whereas Komodo plays with a more positional, subtle style that may be beneficial in closed positions. From my observations Komodo used to have a better endgame technique but more recently that may have swung slightly in Stockfish' favour when it can pick up the win in the endgame too.

Houdini looks very competitive in the current TCEC and with another 2 to 3 months of improvement it will be interesting to see where that lies in style compared to the big two. I'm hoping the Tactical Mode will still be provided and that capability will not be sacrificed.

PeterG
Parent - - By shrapnel (***) [in] Date 2016-08-22 04:57
Excellent ! That suits me just fine !
You just exposed the weakness in your Book.
Just last night I won three games on c2p (www.come2play.com) with Black using the French Defense.
I do hope more and more players use your Book.
Heh heh.
Parent - - By Master Om (Bronze) [in] Date 2016-08-22 13:47
Won with french ?
LOLZ
You play with kids. :yell:
Parent - - By shrapnel (***) [in] Date 2016-08-22 14:43
Hi Om
        Why don't you play me sometime and show this noob how its done ?
Not the first time I challenged you either.
I'll PM you my UserId and you PM me yours, if you're upto the challenge, GrandPa.
Let me know.
Regards
Parent - - By Master Om (Bronze) [in] Date 2016-08-22 15:36
Yah Sure. Where do U play ? Let Us play at Lichess.org. The user interface is sluggish.
Just send me the link and u set the time controls. Am Ready.
Parent - - By shrapnel (***) [in] Date 2016-08-23 04:15
Hi Om
        I play at c2p.
I don't know about Lichess.org....will have to check it out to see if it suits my style of play.
I'll let you know later, at work now.
Parent - By Master Om (Bronze) [in] Date 2016-08-23 15:49
Waiting....
Parent - By Venator (Silver) [nl] Date 2016-08-22 14:47
Hi Shrapnel,

No more need for me to engage in discussions with you. You are on ignore.
Up Topic The Rybka Lounge / Computer Chess / How good is the Cerebellum book with Stockfish?
1 2 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill