Not logged inRybka Chess Community Forum
Forum RybkaChess Help Search Register Login
Up Topic The Rybka Lounge / Opening Books / ABK Book Testing Underway...
- - By Dark_wizzie (**) [us] Date 2011-11-16 00:27 Edited 2011-11-17 02:20
Hello. A mini-tournament is underway to test the new book, Tria 1.5a Varied by Saldom, and another book, ZBook. Now, my new way of testing books is to bring any new books being tested, and add the top 5 books I have into the "tournament" to test the book's ability to go against lines of other books. This "top 5" excludes books under the same name, and usually the same author, to ensure diversity in openings.

Time control: Five minutes

Engine                                 Score                
1: Tria 1.4L Houdini               61.0/93 <-- New contender
2: Tria 1.5L Varied Houdini     56.0/92
3: Aaricia 11.11 Houdini 2      48.5/92
4: Perfect 2012 Houdini         38.5/92
5: ZBook Houdini 2                38.0/92  <-- New contender
6: Morphius Houdini Wizzie      35.0/93

Important note:
Unfortunately, I accidently changed the book settings for the Arena Book. I made Tria 1.5 Varied the Arena book, and this gave  Tria 1.5L Varied an advantage over other books. I have fixed this halfway in the tournament, and since then, Tria 1.5L Varied has fallen into second place. I WILL update this score sheet later on, as I feel 92 games isn't enough to give a clear, defined picture of ratings considering my mistake with the book settings.

I have decided not to test Lange and MrQ unless somebody requests I do so. Looking at the size of the books, I don't think it would have any possiblity of being a top book. Saldom's tests at http://www.tpp89.org/p-abk-2.htm for the two books prove my guess to be correct.
Parent - By Saldom (**) [ru] Date 2011-11-16 06:36
In my humble opinion, if possible, it is necessary to test all the new books. Meanwhile, more books by authors who produce not the first book.

And focus on my tests are not worth it. Different computers, different time control.
Parent - - By Saldom (**) [ru] Date 2011-11-16 06:52
And the strength of the game on the size of the book can not be judged. Very often great books are much worse than the carefully crafted book.

Do not forget about the depth of the book.

Lange_2011.abk
Filesize: 39mb
Book depth: 20 halfmoves

TriA + JK 1.4L +. Abk
Filesize: 246mb
Book depth: 200 halfmoves

... If Evgeny Timoshchuk make his book a depth of 200 halfmoves his book will become much more of my books.
Parent - - By Dark_wizzie (**) [us] Date 2011-11-16 07:31 Edited 2011-11-16 07:38
Then the question is, how is a book with that few moves going to get anywhere close to the top?

Alrite, if you want me to test Lango and MrQ, I will test them...

Results will take a few days, unfortunately. I need to redo and change some stuff up to ensure my results are accurate.

Quick question: What exactly is a "halfmove"?
Parent - By Saldom (**) [ru] Date 2011-11-16 07:42
Dear Dark_wizzie,

You should decide what books are and how to test, I only expressed my opinion.

Best regards,
Aleksandr
Parent - - By Saldom (**) [ru] Date 2011-11-16 07:46
halfmove - deep plies in the book. 200 halfmove - means that the book has information for 100 of the black and white.
Parent - - By Dark_wizzie (**) [us] Date 2011-11-17 02:18 Edited 2011-11-17 02:31
Engine                              Score               
1: Tria 1.4L Houdini              60.5/98
2: Tria 1.5L Varied Houdini    56.0/99
3: Aaricia 11.11 Houdini        51.0/98
4: Perfect 2012 Houdini        43.5/99
5: ZBook Houdini                 38.5/98
6: Quinox 4 Houdini             31.0/56
7: MrQ1 Houdini                  29.0/56        
8: Lange_2011 Houdini         20.5/56

Standings after ZBook and Lange are eliminated for being the last and second to last place:

   Engine                            Score      
1: Tria 1.4L Houdini             42.0/72
2: Tria 1.5L Varied Houdini    40.0/72
3: Aaricia 11.11 Houdini        35.5/72
4: Perfect 2012 Houdini         26.5/73
5: Quinox 4 Houdini              21.5/40
6: MrQ1 Houdini                   19.5/41

Now adding TriA+JK 1.6.1L+.abk although I was hoping for TriA+JK 1.6.1A40.abk...
Parent - - By Sekos (****) [pl] Date 2011-11-17 12:25 Edited 2011-11-17 12:51
Hello, it a little strange result, because in Saldom site is a little different :

1 Aaricia 11.11.ABK  18.0/23 78.2 ··· 2-1-2 2-0-3 5-0-3 5-0-0  142,25 
2 TriA+JK 1.6.1A40.abk  8.5/18 47.2 1-2-2 ··· 1-1-2 2-1-2 0-1-3  87,50 
3 Lange 2011.abk  8.5/19 44.7 0-2-3 1-1-2 ··· 0-0-5 0-0-5  81,50 
4 MrQ1.abk  8.0/22 36.3 0-5-3 1-2-2 0-0-5 ··· 2-2-0  79,25 
5 TriA+JK 1.6.1 Varied.abk 7.0/18 38.8 0-5-0 1-0-3 0-0-5 2-2-0 ···  58,50 

http://www.tpp89.org/p-abk-2.htm

S.
Parent - - By Dark_wizzie (**) [us] Date 2011-11-17 14:31 Edited 2011-11-17 14:48
A difference in place in a testing scene like that isn't surprising at all. Don't forget that Saldom and I probrably use different time controls and hardware. We also have different setting for Arena Mainbook weights. I've also done more games of chess total than in Saldom with Aaricia/Tria, etc. (Looks like Saldom only did 23 games for testing Aaricia, while I've done 100+ for this testing alone, not forgetting the other 100+ games per engine for ABK Book Bash.) Also, there is also the factor of pure luck. A little of this is up to chance. Saldom's earlier tests (also with Aairicia, I think) show Tria leading, from what I remember. Finally, we have different books up for testing, that might affect the results a little bit. Some books are better against others than some. By the way, Aaricia is closer to Tria 1.4L as ever, as it's only 4.5 points off.

Don't worry, more tests are on the way, just give me a day or two and I'll post the results.
For now, here are the results I have currently.

   Engine                            Score                  
1: Tria 1.4L Houdini             48.0/85
2: Tria 1.5L Varied Houdini    47.0/86
3: Aaricia 11.11 Houdini 2      43.5/86
4: Perfect 2012 Houdini         33.5/87
5: Quinox 4 Houdini               31.5/60
6: Tria 1.6.1L+ Houdini 2       31.0/60
7: MrQ1 Houdini 2.0              27.5/60
Parent - - By Saldom (**) [ru] Date 2011-11-17 16:29
Here you have the perfect proof that the little book may be stronger than large.
My experiment has failed.
The fact that increased Polyglot book - weakened the book for the Arena. I'm sorry.
Parent - - By Dark_wizzie (**) [us] Date 2011-11-18 02:19 Edited 2011-11-18 02:24
Not really, Saldom. You are too quick to put your own books down. Look at this:

   Engine                              Score             
1: Tria 1.4L Houdini              55.5/101   .549     2nd place    -Same author as other Tria books;all tria books count as one book; Book contestant 1
2: Aaricia 11.11 Houdini 2      52.0/101  .514      3rd place    - Book contestant 2
3: Tria 1.5L Varied Houdini     51.0/102  .5          5th place  -Same author as other Tria books
4: Tria 1.6.1L+ Houdini 2       50.5/91    .554     1st place  -Same author as other Tria books
5: Quinox 4 Houdini              47.0/92  .51         4th place  - Book contestant 3
6: MrQ1 Houdini 2.0              42.5/92  .461       6th place   -Book contestant 4
7: Perfect 2012 Houdini         42.5/103  .412      7th place   -Book contestant 5

If you want to find generally who would be in first place if all books had the same amount of games, divide won games by games played. Tria 1.6.1 is BEATING 1.4L if it keeps up this ratio of won games.

Now:

1) Very curious thing happening: I saw how Perfect 2012 is clearly the losing book here. So if I remove it from the list, this happens:

   Engine                             Score                
1: Tria 1.6.1L+ Houdini 2       44.0/77     .571
2: Tria 1.4L Houdini            41.0/83       .493
3: Tria 1.5L Varied Houdini    39.5/83
4: Aaricia 11.11 Houdini 2      39.5/82 
5: Quinox 4 Houdini             39.0/77
6: MrQ1 Houdini 2.0             36.0/76

Aaricia drops from third place to fourth place! Also, Tria 1.6.1 gains a huge lead! It seems like Tria 1.6.1 is unable to effectively beat Perfect 2012, while Aaricia was able to. However, according to by rules, the books being tested all count as one book in my "tournament of five", so Perfect 2012 belongs on that list since it is the strongest book not counting the other books in this test. (I have quite a few other ABK books that I do not use to test Tria since they are too weak.)

I will continue to test. As of now, Aaricia is still in 3rd place.

Also, Saldom:
I would appreciate immensely if you focus a lot on your 20 minute time control tournaments, since I'm focusing on the 5 minute time controls.
Parent - By Dark_wizzie (**) [us] Date 2011-11-19 00:55
Engine                             Score                 
1: Tria 1.6.1L+ Houdini 2       76.0/133   .571
2: Tria 1.4L Houdini              75.0/133   .563
3: Aaricia 11.11 Houdini 2      69.5/132   .526
4: Quinox 4 Houdini              64.5/132    .488
5: Tria 1.5L Varied Houdini     63.0/133    .379
6: MrQ1 Houdini 2.0              59.0/133   .443
7: Perfect 2012 Houdini         57.0/132    .431
Results fit my exact predictions.
...But, the testing isn't over. Tria 1.6.1L+ and 1.4l+ are too close to count.
I will continue to run more tests.
Parent - - By Dark_wizzie (**) [us] Date 2011-11-17 14:47
HI Saldom.

If I adjust the Arena Mainbook setting for Testo book, will it affect the way another book will use their moves? Or does each book have their own weights for moves played, priority, etc?

What time controls are you using for your website to test Aaricia and Tria 1.6?
Thanks
Parent - - By Saldom (**) [ru] Date 2011-11-17 16:36
If the engine configuration is not set "main book isprolzovat Arena" - that will not be affected.

I use the control 20 minutes per batch + 10 seconds per move
Parent - - By Dark_wizzie (**) [us] Date 2011-11-18 02:21
Hmm, then what weights for "priority", "won games", and "games played" will be set for those books not set as Arena Mainbook? Default? Or same as Mainbook?
Parent - - By Saldom (**) [ru] Date 2011-11-19 03:56
To test the books were accurate and honest, you must:
- Do not modify the author of the book.
- Do not change the settings of the books.
- Do not include the engine configuration option "Use the basic book of the Arena"
Parent - - By Dark_wizzie (**) [us] Date 2011-11-19 05:15 Edited 2011-11-19 06:12
Ok Saldom. I want to set up an accurate, honest tournament. I'd appreciate if you would elaborate on what you mean.
Here is my understanding, correct me if I'm wrong, please.

All books have settings that apply whether they are mainbook or not.
All books have settings that may only be changed if they are the mainbook.

The book system is confusing for Arena. I'd really like to provide accurate testing, and testing all the games in the world means nothing if my settings are whacko.

I have double checked everything, and the only book that got modified in Arena settings is Tria 1.4L+. I have fixed the error.

Thank you.
Parent - By Saldom (**) [ru] Date 2011-11-19 08:42
Dark_wizzie,

To test books must meet the rules of my previous posts.
Sorry for my English.

Best regards
Parent - By Sekos (****) [pl] Date 2011-12-07 18:49
Hello Dark_wizzie, can You test my new book, please,
Later please share results in PGN,

Sekos
Up Topic The Rybka Lounge / Opening Books / ABK Book Testing Underway...

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill