Not logged inRybka Chess Community Forum
Up Topic The Rybka Lounge / Computer Chess / import book
- - By manos (**) Date 2010-01-30 06:34
hello,
I am downloading books and games which I importing them to rybka book by the command from the menu in fritz. So far the book is 18Gb, but the last days I see that is remain is the same size although I import games and I see that new positions are added. why is that?
Parent - - By Fulcrum2000 (****) Date 2010-01-30 12:45
There are two possibilties.
1. Almost all moves/games you are adding are already in the book, so the size will not increase much as the book has only to update the statistics not the actual moves (apart from a few new moves).
2. You reached the limit of the .ctg book format. Check with chessbase.
Parent - By manos (**) Date 2010-02-01 06:35
and how can I set it up in chessbase the .ctg format to make it bigger?
Parent - - By BrandonSi (***) Date 2010-01-30 14:33
Why are you doing this? I'm just curious?

Importing many books does not necessarily make the book stronger, in fact, unless you are importing the correct lines, it can very easily make the book weaker.
Parent - - By manos (**) Date 2010-01-30 19:02
All the books that I import are very good, I searched them through tests and from the lists of various chess sites.
I thought that importing games and books I am going to create a stronger book, and as far I can see, its strong enough, I have been playing a few months a large number of games in internet and its winning.
Please tell me if I am made something wrong or am I wrong and stupid?
Parent - - By BrandonSi (***) Date 2010-01-31 02:34
You're not stupid! :) As long as it is working for you, it's good. :)
Parent - - By Mark Mason (***) Date 2010-01-31 08:49
Manos,

I agree with Brandon that you should not assume that merging several strong books will result in a stronger one . The problem with merging whole books together is the effect it has on the move weightings (ie. the figure in the .ctg "FACT" column that determines the probability that a move will be chosen by the engine) of each book when they are combined.  A lot of these books have been carefully hand tuned and weightings assigned .  When you merge the books these weightings clash and get messed up and corrupted which can lead to bad results eg. a particular move in a particular position may have for example a weighting of (+ 50) in one book and  (- 20) in another - what happens when you merge the books ?   The only way around this is to reset the weights after merging the books together, but this completley loses all the work that the book authors have put into hand tuning their books.

Mark
Parent - - By Uly (Gold) Date 2010-01-31 20:28
Yes, however, that's theory, in practice the other day we had someone that merged a ton of books and the new monstrosity was able to beat the best book of the mix, no tweaking needed. I think more experimentation is needed.
Parent - - By Nelson Hernandez (Gold) Date 2010-02-05 15:47
Monstrosities can be very effective.
Parent - - By manos (**) Date 2010-02-05 17:32
in my case the book is great.
Parent - - By Uly (Gold) Date 2010-02-05 23:42
So, two out of two times we got a counterintuitive result.
Parent - - By BrandonSi (***) Date 2010-02-06 00:08
Hardly a valid sample population, but it is interesting. :)
Parent - By Uly (Gold) Date 2010-02-06 00:51
Since theory says that even after optimization the merging a bunch of books idea shouldn't work and the examples seen show that you can do it carelessly and get good results, the chance that the theory is flawed is quite high, even with a small sample.

Oh, a sample of 3 out of 3 (1.- the guy that did it at first 2.- the guy that tested if the results of the first guy could hold 3.- the current case)
Parent - - By pawnhook (**) Date 2010-02-06 05:35
Yes merging of books works very fine I have played many games with merged books but sometimes it fails too?
Parent - - By Mark Mason (***) Date 2010-02-07 08:50
Pawnhook/Manos,

When you merge the books do you reset the weights each time or just import it and leave it as is ?
Parent - By pawnhook (**) Date 2010-02-07 19:41
I reset the weight and make it learn from database.
Parent - - By manos (**) Date 2010-02-07 20:21
just learn from database each time that I importing books,games.
I never reset the weights.
Parent - - By manos (**) Date 2010-02-08 12:00
and always set it up to optimize  minimum 1-max 100 games,
Parent - - By Mark Mason (***) Date 2010-02-08 14:26
Hi,

I understand about applying learn from database when you add games - but how can you do a learn from database when you merge a whole book ?  If you merge a book then unless you also have the games that book was made from, how do you do a learn from database on the merged book as you will not have the source game database?  Which database do you point at ?
Parent - By BrandonSi (***) Date 2010-02-08 15:07 Edited 2010-02-08 15:14
I've done this before as a test, and it does work (though I'm sure one could get negative results from doing this as well). For example, I created a book with playchess games from Oct-Dec,  lets call it book A. I then imported Book A into Book B. I selected Book B and had it learn from a database with about 800k games, while leaving book A alone.

In a tournament with Rybka 3 + BookA vs Rybka 3 + Book B (1+1 if I remember correctly) Book B easily defeated Book A.

The learning feature doesn't require the same source games, it just promotes or demotes lines in the book based on the results of the game database. My understanding (and I'm not an expert by any means) is that "learn from database" will add to the statistics of a line, increasing or decreasing the frequency that a specific line is played. If you took a book that consisted of only D46 (1 d4 d5 2 c4 e6) lines, and used "learn from database" against a database of say, B08 games (1.e4 d6 2.d4..), no values would change.

I guess my point is that you do not need an exact match for a game for the statistics to change. "Learn from database" will adjust the statistics for sets of contiguous *moves*, not entire lines  or games.

Obviously a good "learning" database is required. If the learn database was all from human players with ELO ~800 the book that learned from the database would not do very well.

Apologies if you knew that already, you know much more than I do Mark. :)
Up Topic The Rybka Lounge / Computer Chess / import book

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill