Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka TB test Rarely v Normally
- - By Shaun (****) Date 2007-12-18 20:41
I ran a TB test with Rybka 2.3.2a 64-bit 4CPU (default - NalimovUsage Rarely) v Rybka 2.3.2a 64-bit 4CPU (NalimovUsage set to Normally)

I have the full 5 piece TBs on a fast USB flash drive and have previously confirmed this runs dramatically faster than my 7200 SATA drive.

I was expecting a measurable improvement for the higher TB usage however:

1024 games played 511:513 - the time control was CCRL 40/4 (this was 40/2 on my overclocked Quad)...

Shaun

P.S. I will be running Rarely v No TB's soon...
Parent - - By turbojuice1122 (Gold) Date 2007-12-18 20:45
This is an interesting test.  I tend to think that in your next test, it might be better to test normally versus never, since this will provide a bit of a larger difference than rarely versus never.

Will you eventually try to test with some of the 6-piece tablebases?  Any improvement would be noted more in this case.
Parent - - By Shaun (****) Date 2007-12-18 23:20
There are several tests I want to run - the problem is you need a lot of games for confidence... if fast TB access does not benefit my assumption is that TBs on a standard 7200 SATA disk will hurt performance.

I am not sure about the 6 piece TB's obviously flash memory is out of the question so you would need a fast raid array - also I have seen some very odd behaviour with part TB's but going for the full 6 men is 1.2TB download. I would need to know I would be getting a significant performance boost - which I doubt...

If you are really into difficult position analysis and finding the shortest mate then the TBs are usefull, but for me if it could have been mate in 10 but the computer took 12 moves I am not concerned.
Parent - - By Permanent Brain (*****) Date 2007-12-18 23:45 Edited 2007-12-18 23:48
I think, if you try to test this with normal games, most of the time is wasted and the results are too much influenced by book and what happened in the middle game, where tbs. access issues do not interfere with. Also, I think it would be better to compare the performance against another 'standard' opponent instead of Rybka vs. Rybka.

Proposal:

Select or create some balanced early endgame positions. Not too simple, different from each other, and the more the better. Put them into a database and use this database as openings database. Play test matches with switching colors (=switching sides, 2 games per position), always versus the same good test opponent (maybe Zappa or whatever you have), oppo always in default engine settings... Or maybe it will be even better to set the opponent engine to NOT use tablebases, if possible, because Rybka's behaviour may influence also the other engine's tbs. access because sometimes more, sometimes less tbs. data will be in the windows file cache.

1. Rybka/no tbs.
2. Rybka/rarely
3. Rybka/normally
4. Rybka/frequently

(I suggest to set hash table and tbs. cache to 'medium' sizes, for that)
Parent - By Shaun (****) Date 2007-12-19 00:14
Normally I am not a fan of playing an engine v itself or a similar version but I think with just a TB usage changes then this is probably okay...

Oh and each of the position out of the book is played twice once for white and once for black.

I agree that a lot of the games will have been decided before TB's are accessed however I do not know of a suit of early endgame positions that I could just use and I would want it to be a little before any TB access occurs.

If anyone knows of such a suite please let me know...

Shaun
Parent - - By NATIONAL12 (Gold) Date 2007-12-18 21:34
i have found tablebases, like you appear to also have found make very little difference.what you gain is offset by the slowing down of your computer to acess them,at most i think 5 elo but more likely 2-3 elo.are they worth all the hassle of downloading them for free (days for 6 man tbs) or worth paying £40 for.
Parent - - By Shaun (****) Date 2007-12-18 23:22
Probably not...
Parent - - By NATIONAL12 (Gold) Date 2007-12-18 23:31
even vas himself dous not use them, i think that speaks for itselfs.
Parent - - By Shaun (****) Date 2007-12-19 00:21
If someone has the full 6 men TBs on a super fast raid array with shed loads of disk cache I would love to see the results...

Oh looking at Shredder it does seem to benefit from the Shredder bases

See here

This is comparing the 'fast' Shredder bases to TB access on a fast 7200 SATA2 disk.

Now is this showing that the Shredder bases are helping or the TB's on disk are handicapping I am not sure - I suspect a bit of both but I will find out ;)

Shaun
Parent - - By BB (****) Date 2007-12-19 00:47
Oh looking at Shredder it does seem to benefit from the Shredder bases

I have a version of Shredderbases in my programme, and the anecdotal evidence is that they definitely do make an ELO difference. At least 1 point per 100 games (which is usually the number of overnight games I look at per day). These games are at 2m+2s on a 3Ghz P4, and the lessened search time here might make a difference. (One of the most bizarre games was where Toga had BNPP vs BPPPP, my programme trades B for 2P, both programmes erroneously think the 4 pawns are better for a few moves, then Toga wipes out all four, but fiddles at mate-in-25 in KBN vs K for 50 moves). I will probably quantify the observed bitbase gain sometime next year.
Parent - By BB (****) Date 2007-12-19 23:52
Just to support my ancedote, here are three(!) very explicit examples taken from a sample of 100 2m+2s games against Glaurung 2-epsilon/2 last night (there are many other cases where having bitbases might have changed a result, but in these three, it is fairly clear). [Of course, some endgame logic might obviate the need for TBs here, and having three such examples was quite surprising to me].

6k1/6p1/5p1p/1R6/4P1PP/r4P2/6K1/8 b - - 0 49

This should certainly be a draw of course. After nearly 60 moves of shuffling, we get
8/8/5p1k/5P2/4P3/8/4K3/r2R4 b - - 32 108

Glaurung plays 108.. Rxd1?? 109 Kxd1 and #27

8/2r5/1R5P/8/5P2/2k3K1/1p6/8 b - - 0 61

Glaurung plays 61.. Kc2?? 62 Kg4 b1Q 63 Rxb1 Kxb1 and #42

8/4k3/4q3/8/3Q2p1/6P1/5P2/6K1 b - - 7 91

Glaurung plays 91.. Qe2?? 92 Qxe3+ and #27

This one also came up:
8/3k4/4n3/B2K2n1/6B1/6P1/8/8 b - - 0 68

Bourzutschky and Konoval have solved such, but I don't think the TB is public. [Glaurung ended up winning here - my programme had a slight logic error in aiming for this position; it knows that RmP vs Rm is likely drawn due to trading the minor piece for the pawn, but I didn't think to note that BBP vs NN is not the same].
Parent - - By Mark (****) Date 2007-12-18 22:44
Wow, 511:513.  I wouldn't have expected it to be that close even with identical settings!
Parent - By Shaun (****) Date 2007-12-18 23:27
Hi Mark,

As I watched the match it bounced about a little at the beginning but then stayed fairly even for the last few hundred games...

I tended to check the score a couple of times a day over the 4 days for the test.

Shaun
Parent - - By Vasik Rajlich (Silver) Date 2007-12-20 13:49
You see, Rybka's comprehensive endgame knowledge makes TBs obsolete.

Vas
Parent - By Roland Rösler (****) Date 2007-12-20 15:21
No :-) from your side! That´s cool, but no good bluff!
Parent - - By Lukas Cimiotti (Bronze) Date 2007-12-20 16:44
You are right in many cases - but not in all.
Rybka doesn't win KBNK and KNNKP without TBs. KQPPKQ is difficult, too.
I made some tests about TBs as i like that stuff very much. I was very disappointed about my results, TBs only added ~5 Elos for 5 man on a SSD.
And i saw Rybka win some games without TBs that i think are difficult to win - like KQKR.

Regards,
Lukas
Parent - - By turbojuice1122 (Gold) Date 2007-12-20 18:40
What time controls?

The tablebases for both engines definitely made differences in the Mexico match--noticeable differences just in ten games--this tends to happen more often at the highest levels and in longer games.
Parent - - By Lukas Cimiotti (Bronze) Date 2007-12-20 19:02
I made my tests at bullet time control (1+0).
I am not shure, if the effect is better in longer games. But I'd like to hear some arguments pro and con.
In the Mexico match i had a good set of TBs - all 5 man and 54 GB of 6 man (incl. KRPKNP - the missing TB in Amsterdam).
I didn't have KRRBKQ (and i still haven't got it). Maybe that one could have save Rybka from loosing - but i don't think so.

Regards,
Lukas
Parent - - By turbojuice1122 (Gold) Date 2007-12-20 19:47
I think the main thing should be that in long games, the evaluation would be a bit more stable, and with better moves being played and more tablebase hits per move, there will be an increasing effect.  On the con side, this would still mean a slowdown in search, but I think this would be avoided after a certain point when most of the relevant tablebase positions have been evaluated.  However, I don't know if the "relevant tablebase tree" will grow as quickly as the search tree--my feeling is that it won't.
Parent - - By Lukas Cimiotti (Bronze) Date 2007-12-20 20:06
These are good thoughts.
I use 16 MB for TB cache and a SSD with 0.1 ms of access time.
With this setup, I don't see any slowdown (with an ordinary hd you can see this slowdown when looking at the Task Manager - with a 7200 rpm disk, cpu usage goes down significantly when TBs are used - with that SSD cpu usage is always at 100%).
So i think, my results should be better when using TBs. But they aren't.
I'll make more tests about this, but i don't know if the money spent for these SSDs was a good investment.

Regards,
Lukas
Parent - - By Vasik Rajlich (Silver) Date 2007-12-25 15:07
FWIW - I'd bet that the Elo gains from tablebases are similiar in blitz and in longer time controls.

5 Elo is nothing to scoff at. Somewhere down the line, that will be the difference between winning and not winning.

Vas
Parent - By Mark (****) Date 2007-12-26 04:40

> 5 Elo is nothing to scoff at


Doesn't sound like much at first, but it's probably equivalent to five weeks or so of either yours or Larry's time!
Parent - - By peter carvalho (*) Date 2007-12-26 11:45
My experience has been that the Elo gains depend on the interface used there are no gains with chessassistant interface or fritz interface but there is a three to five percent more wins when shredder UCI interface is used. I came to this conclusion accidentally while analyzing endgames in DEEP RYBKA AND chess openings 2007. In a computer where the commercially available six piece endgame tablebases are loaded      go to game     Fiori De azevedo pio        v/s  Pinto Paiva Jose      Sao paulo  1973   if you search for blunders using the function given in chess assistant you will find no errors found  but if you do infinite analysis using tablebases you will find that77.Kd2 is a blunder  If you change the engine to Rybka the evaluation is not replaced by the tablebases evaluation even when the engine uses tablebases but comes as (-5.91) which reveals nothing . When i used to conduct engine tournaments using Fritz interface i found that th results were poorer when tablebases were used that there was a decrease in percentage scores ( if i recall correctly) So when i discovered this new information I set up two tournaments on two computer one a quadcore . On the quad core (64bit) i noticed that with tablebases used normally in shreeder UCI interface the score percentage against Really shredder 9 increased from 91% to 96 % using tableses and on the dual core (32 bit) the score increased from  78.5 % to 82 %. A total of 400 games 1200 in each set May or may not be statiscally significant
Any comments. These gains were in blitz games of 5 minutes each side

Dr. Peter arvalho
Parent - - By Vasik Rajlich (Silver) Date 2007-12-27 12:42
Perhaps different resignation thresholds were used in the different interfaces, this might affect tablebase results. Aside from this, the interface shouldn't have any effect on tablebase usage.

Vas
Parent - - By peter carvalho (*) Date 2007-12-28 05:37
different resignation threshold does answer why Rybka evaluates a six piece endgame ( -5.31) in chesassistant where the tablebase analysis shows mate in 26. This does not happen in shredder UCI interface. What led me to suspect the fritz interface was , when the same game was analysed using fritz the evaluation using a blunder range of 300 centipawns was different from the tablebase evaluation for certain six piece endings the ealuation shows mate in 28 in fritz interface whereas the tablebase evaluation is a draw. It probably happens many times like this when it comes to six piece endgames

Dr.Peter Carvalho
Parent - - By Permanent Brain (*****) Date 2007-12-28 05:49
I'm not sure if this is involved here, but: There was (and always will be) confusion about tablebase access in different interfaces, because in some - like Arena - only the Engines access them, while i.e. in Fritz, the interface takes contol as soon as there is a tablebase position on the board, in a game (this is the default but can it be switched off though).

For example, if you analyse a five piece position in Fritz, AND you have the complete five piece tablebases, the GUI will simply show the list of all legal moves and their evals from the tbs., each - most of the time, there seem to be rare exceptions where the engine will still calculate although the tablebase for the position is available (at least I noticed that in Fritz 8's GUI; not sure about newer ones).
Parent - By Vasik Rajlich (Silver) Date 2007-12-30 11:49
Rybka herself will display mate (or 0.00) scores in tablebase positions. Perhaps the tablebases are not being set properly in that interface.

Vas
Parent - - By Shaun (****) Date 2007-12-20 17:42
Well I will soon have the results No TB's v TB's ;).....

Watch this space

Shaun
Parent - By Shaun (****) Date 2007-12-20 17:48 Edited 2007-12-20 18:28
I do think that TB's will give a small improvement if the access is fast - however there may be a danger that TB's on a standard HD will hurt performance. However if you are looking at the best move rather than game play TB's are probably a must - but I don't really care if Rybka takes 10 moves or 15 to mate the result is the same to me.

Over time as processor improvements have dramatically out paced HD improvements the benefit of TBs will have decreased (or at least that is the theory I believe).

Shaun
Parent - - By Banned for Life (Gold) Date 2007-12-20 18:08
I will believe this when you stop using tablebases in engine tournaments (and I won't be holding my breath waiting for this to happen).

Alan
Parent - By BB (****) Date 2007-12-20 23:44
I will believe this when you stop using tablebases in engine tournaments

There was a movement (led by Bruce Moreland, writer of Ferret, which was 2nd place in the WCCC/WMCCC a few times) at one point (seems to be from 2000) to allow only Nalimov (or only one entrant, named by him, for a given tournament) to use his TBs.

>Personally I think that Nalimov should be allowed to designate one program as"his", at which point he'd become a co-author of that program, and nobody could use the tables in official competition except that program.


His position did not gain much support. :)
Parent - By Vasik Rajlich (Silver) Date 2007-12-25 15:09
Actually, there were a few tournaments on ICC where we used my hardware and played without TBs. Whether this is laziness or design, I'll leave to your imagination :)

Vas
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka TB test Rarely v Normally

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill