Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Tournament: R3 no EGTB vs. R3 EGTB
1 2 Previous Next  
- - Date 2009-07-20 10:40
Parent - - By bnc (***) Date 2009-07-10 02:31
Final Report
----------------

I have finished the EGTB test of 10,000 games . Rybka 3 with EGTB's 3-4-5 and 180gb 6man has defeated Rybka 3 without EGTB's in a single CPU fixed depth 8ply match:                
                 
   Rybka 3  EGTB       +2064/=6237/-1699   51.83%   5182.5/10000
   Rybka 3  no EGTB  +1699/=6237/-2064   48.18%   4817.5/10000

This match favored the EGTB engine by 13 ELO

For the next 10,000 game test , the setup will be the same except that the GUI tablebases option will be enabled.

Setup
--------
Time control - fixed depth 8 ply
System - Intel i7-920 @2.67ghz , hyperthreading disabled
RAM - 6gb
CPUs - 1 thread per engine
Engine - Rybka 3
Engine hash - 512mb
EGTB hash - 32mb
Engine tablebases - 3-4-5-6 man (180gb 6man)
GUI tablebases - none
Nalimov usage - "Never" for one engine and "Normal" for the other
Opening book - large private book, learning functions disabled, variety at 75%
Engine match options - alternating colors
Game options - Resign : never and Draw : never
OS - Vista x64
GUI - Chessbase Rybka 3 (version May 4 2009)
Parent - By Mark (****) Date 2009-07-10 02:51
I ran my test at different depths:

D=7  R3 with EGTBs wins  1023/2000 = 8 elo
D=8  R3 with EGTBs wins  1019/2000 = 6.5 elo
D=9  R3 with EGTBs wins  1039/2000 = 13.5 elo

overall R3 with EGTBs wins  3081/6000 = 9 elo

Probably not enough games at each depth to show a pattern...

Setup
-------
R3 1 cpu with ~30GB EGTBs vs. R3 1-cpu without EGTBs
widebook (without playing each opening twice)
Engine hash - 128MB
EGTB hash - 32mb, Normal usage
Parent - - By Vasik Rajlich (Silver) Date 2009-07-10 05:29
Thanks - that's quite interesting! It's more than I expected.

Vas
Parent - - By Vasik Rajlich (Silver) Date 2009-07-10 05:30
BTW - any chance to get the game pairs (ie. white game and black game) where the EGTB version got a better result?

Vas
Parent - - By bnc (***) Date 2009-07-10 14:25

> BTW - any chance to get the game pairs (ie. white game and black game) where the EGTB version got a better result?
>


Ask and ye shall receive
CB10 and a small script did the trick  :)
Attachment: EGTBpairedwinsfinal.cbv - Game pairs where the EGTB engine got a better result (3336k)
Parent - - By Vasik Rajlich (Silver) Date 2009-07-11 06:05
Interesting, thanks!

I browsed the first 25 game pairs and classified the cause of the discrepancy. Here is what I found:

no apparent cause: 15
P v -: 2
QP v Q: 1
P v B: 1 (game #12 - really insane eval by Rybka)
BN v -: 1
NPP v B: 1
NP v BP: 1
BP v - (wrong bishop): 1
RPP v R: 1
RP v R: 1

Two interesting things here:

I) About 60% of the differences are due to random effects early in the game. I assume that there is a corresponding # of games where the no-EGTB version won for the same reason. Would it be possible to eliminate this noise somehow?

One possible solution is to get the subset from this set of games where the divergence came late in the game (let's say 10 pieces or less) and/or where it was accompanied by a significant score difference.

I wouldn't mind to classify the true differences a bit more without having to wade through this noise.

II) 80% (8/10) of the true differences came from 5-man positions, and only 20% from 6-man. This may be a statistical fluke - it's certainly not what I would expect.

Vas
Parent - - By Mark (****) Date 2009-07-11 12:26

> About 60% of the differences are due to random effects early in the game.


Could this be due to positions in the hash, or is the hash cleared before each game?

> 80% (8/10) of the true differences came from 5-man positions, and only 20% from 6-man.


Maybe because he only had a partial set of 6-man tables?
Parent - - By bnc (***) Date 2009-07-11 15:01
I think I have most of the "common" and/or "important" 6man EGTB's  :) :)   I have posted below what I currently am using.
For your reference, Nelson did some nice analysis of this issue,  http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?t=59

KBBpKQ
KBBpKR
KBNKpp
KBNpKR
KBpKBp
KBpKNp
KBpKpp
KBppKB
KBppKN
KBppKR
KNpKNp
KNpKpp
KNppKB
KNppKN
KNppKR
KppKpp
KpppKB
KpppKN
KpppKR
KpppKp
KQQKQQ
KQpKQp
KQpKRR
KQpKRp
KQppKQ
KRBKNN
KRBpKQ
KRRpKQ
KRpKBN
KRpKBp
KRpKNp
KRpKRp
KRpKpp
KRppKQ
KRppKR
Parent - By Mark (****) Date 2009-07-11 16:00
Thanks.  I use a similar subset that works well, but mostly I use a smaller subset for everyday use.  I used to check the CCRL Engame Tablebase forum often, but it hasn't been very active lately.
Parent - - By bnc (***) Date 2009-07-11 15:11

> One possible solution is to get the subset from this set of games where the divergence came late in the game (let's say 10 pieces or less) and/or where it was accompanied by a significant score difference.
> I wouldn't mind to classify the true differences a bit more without having to wade through this noise.


I didn't think I could accomplish this in any sort of automated way until I stumbled upon the CB "Compare Notations" command. Very useful ! 
Attachment: EGTBpairedwinsdivergence10pieces.cbv - Game pairs where the EGTB engine got a better result and divergence with 10 pieces or less remaining (1407k)
Parent - - By Roland Rösler (****) Date 2009-07-12 04:00

>Game pairs where the EGTB engine got a better result and divergence with 10 pieces or less remaining (1407.1k)


Okay, here we have 536 games (5.36%), where R3 EGTB makes 403.5/536 (+271 =265 -0; this means, we have 3 games R3 EGTB makes 2-0!). But we need 365 (not 271) winning points for R3 EGTB. So where are the remaining 94 winning points?

My questions:
1. Are there game pairs where the no EGTB engine got a better result at all? If yes, how many?
2. Are there game pairs where the EGTB engine got a better result and divergence with 14 (or finally 20?) pieces or less remaining? How many?

It would be fine, if you can show us, how deep EGTB goes with depth=8 and main 6-man (with depth=8 you can kill theoretical 8 pieces bothsides :-)). # of 2. - # of 1. should be ~730 (365*2)! If not, I have to think more about this issue (okay, I have already some ideas).

PS: I know, I´m a pain in the neck. :-) But I´m really interested in the answers of the questions!
Parent - By Vasik Rajlich (Silver) Date 2009-07-12 06:11
If the statistics are significant, then there must have been a handful of positions which missed the 10-piece cutoff where the tablebases still helped. That's not surprising.

Vas
Parent - - By bnc (***) Date 2009-07-12 15:26

> My questions:
> 1. Are there game pairs where the no EGTB engine got a better result at all? If yes, how many?
> 2. Are there game pairs where the EGTB engine got a better result and divergence with 14 (or finally 20?) pieces or less remaining? How many?
>


1. The no EGTB engine got a better result in 372 pairs (744 games)
    The EGTB engine got a better result in 718 pairs  (1436 games)

2. The EGTB engine got a better result in 329 pairs (658 games) where the divergence was 14 pieces or less
    The EGTB engine got a better result in 268 pairs (536 games) where the divergence was 10 pieces or less
Parent - - By Roland Rösler (****) Date 2009-07-13 01:42
Many thanks for the data.

>The no EGTB engine got a better result in 372 pairs (744 games)


Okay, this Vas called noise. Let us assume it is +372 =372 (no double point).

>The EGTB engine got a better result in 718 pairs  (1436 games)


We know the end result +2064 =6237 -1699
Therefore the result is +737 =699 (this means we have 19 pairs (38 games), EGTB engine made double points).

So we have 2583 pairs (5166 games) ending with two draws.
So we have 1327 pairs (2654 games) ending with a win bothsides (maybe bad opening line?).
So we have 1090 pairs (2180 games) ending with mixed results.
We can break the mixed results in:
- 744 pairs (2x 372; 1488 games) with noise (68%!)
- 346 pairs (692 games; +365 =327) ending in favour of EGTB engine

>The EGTB engine got a better result in 329 pairs (658 games) where the divergence was 14 pieces or less


Okay, 329 pairs isn´t far away from 346 pairs! :-)

Question: Do you know the number of pairs (games) overall where the divergence was 14 pieces or less? Less than 1000 pairs (20%)?
Less than 2500 pairs (50%) where the divergence was 20 pieces or less?
Parent - By Vasik Rajlich (Silver) Date 2009-07-13 05:10

> - 346 pairs (692 games; +365 =327) ending in favour of EGTB engine
> Okay, 329 pairs isn´t far away from 346 pairs! :-)


14 pieces would have been a better cutoff if you wanted to "shoot down the middle". Any cutoff point however will give some noise. I just wanted to get as dense and noise-free a data sample to browse as possible. The 10-man cutoff gave a sample where in roughly 10% of the pairs, the influence of EGTBs was impossible to determine.

Vas
Parent - - By bnc (***) Date 2009-07-14 16:24

> Question: Do you know the number of pairs (games) overall where the divergence was 14 pieces or less? Less than 1000 pairs (20%)?
> Less than 2500 pairs (50%) where the divergence was 20 pieces or less?


From the 5,000 pairs (10,000 games) database :

  2,892 pairs (5,784 games) diverged at 14 pieces or less
  3,447 pairs (6,894 games) diverged at 20 pieces or less
Parent - - By Roland Rösler (****) Date 2009-07-14 22:40

>2,892 pairs (5,784 games) diverged at 14 pieces or less


WOW!! I didn´t expect such high number of pairs (games).
It´s clear, there must be pairs (games), where the no EGTB engine has a better result! Reason: When we assume there are no such games, we get +329 =5455 (no double point pair). This is 52,84% (~20 Elo). More likely is 349 =5435 (20 pairs with double point). This is 53,01% (~21 Elo points). I can´t argue these results. :-(
After some thinking I´m rather sure, with cutoff 10 we have pairs (games) where no EGTB engine has a better result.
It would be fine we could calculate exactly the gain of EGTB at depth=8 for 10 man endgames and 14 man endgames.
Cutoff 10: Therefore we need the # pairs diverged at 10 pieces or less (~2000?) and the # pairs where no EGTB has a better result (and # of pairs, where no EGTB engine makes 2-0; I´m sure this #=0). And it would be fine to see the # pairs draw with win bothsides (or draw bothsides).
Cutoff 14: Therefore we need the # pairs where no EGTB engine gets better results at 14 pieces or less and # pairs, where no EGTB engine makes 2-0. And we need the # of pairs where EGTB engine makes 2-0. And it would be fine to see the # pairs draw with win bothsides (or draw bothsides).
Maybe it´s easier for you two create two databases (cutoff10 and cutoff14) and show us the results (tables with score) created by CB10! :-)

When we have the results we can speculate about depths >8. :-)
My estimation: The score of cutoff10 at depth=8 (>53%?) will become relevant for cutoff14 at higher depths. The higher the depths the nearer to the score.
Parent - - By Vasik Rajlich (Silver) Date 2009-07-15 05:55

> After some thinking I´m rather sure, with cutoff 10 we have pairs (games) where no EGTB engine has a better result.


Yes, of course. If you look at my classification, you'll see that 5/50 (10%) of the improved results had no apparent cause. Some of these probably had an EGTB-related cause which I couldn't pick out, and some of them were probably just noise.

The point of generating the cutoff 10 was just to browse the results and classify them. For Elo results, the overall result is perfectly sufficient.

Vas
Parent - - By Roland Rösler (****) Date 2009-07-15 09:33

>For Elo results, the overall result is perfectly sufficient.


Maybe, but I want to see the real Elo difference in endgames (here 10 man at depth=8) for R3 sp with EGTB vs. R3 without EGTB. With data from bnc we will see the noise (10%?) in the cuttoff 10 pairs won by no EBTB engine and with # of pairs in cutoff 10 we can determine exactly the gain in Elo. Maybe there is different (more?) noise in the drawn pairs, but this doesn´t matter now.
My thesis: When you will replay the cutoff 10 positions at depth=9, 10, 12, ... 16 the noise will decrease and score for the EGTB engine will increase all the time! The same will happen (maybe even faster) when you replay these positions with tc 1', 2' ... 6'. You may say this isn´t allowed! Okay, to see the effect of EGTB in given 10 man positions, it is allowed. With even more time you will see this effect also in 12 man or 14 man positions.
Parent - By Vasik Rajlich (Silver) Date 2009-07-16 04:08

> Maybe, but I want to see the real Elo difference in endgames (here 10 man at depth=8) for R3 sp with EGTB vs. R3 without EGTB.


Why? Of course, you can measure just about anything you like. :)

> My thesis: When you will replay the cutoff 10 positions at depth=9, 10, 12, ... 16 the noise will decrease and score for the EGTB engine will increase all the time!


I would bet good money on the opposite. The tablebase play remains the same at all time controls (perfect), while the non-tablebase play gets better as the time increases increases.

Vas
Parent - - By bnc (***) Date 2009-07-16 15:25

>Maybe it´s easier for you two create two databases (cutoff10 and cutoff14) and show us the results (tables with score) created by CB10!


Analysis of the 10,000 games EGTB test database : http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?pid=170727;hl=

  All games :    ELO +13
                
     1   Rybka 3  EGTB        +2064/=6237/-1699   51.83%  5182.5/10000
     2   Rybka 3  no EGTB   +1699/=6237/-2064   48.18%  4817.5/10000

  Games with divergence at 20 pieces or less :    ELO +14
                
     1   Rybka 3  EGTB       +1495/=4184/-1215   52.03%  3587.0/6894
     2   Rybka 3  no EGTB  +1215/=4184/-1495   47.97%  3307.0/6894

  Games with divergence at 14 pieces or less :   ELO +16
                
     1   Rybka 3  EGTB       +1427/=3202/-1155   52.35%  3028.0/5784
     2   Rybka 3  no EGTB  +1155/=3202/-1427   47.65%  2756.0/5784

  Games with divergence at 10 pieces or less :   ELO +20
              
     1   Rybka 3  EGTB       +1123/=2019/-890    52.89%  2132.5/4032
     2   Rybka 3  no EGTB  +890/=2019/-1123    47.11%  1899.5/4032
Parent - - By keoki010 (Silver) Date 2009-07-16 17:11
Durn, even Roland ought to be happy with that!
Parent - By Roland Rösler (****) Date 2009-07-17 01:39
I really hoped it´s something like >53% in cutoff 10 games. But who really knows what depth=8 in R3 really means. :-)
Parent - - By Roland Rösler (****) Date 2009-07-17 01:30 Edited 2009-07-17 01:47

>Games with divergence at 10 pieces or less :   ELO +20


>   1   Rybka 3  EGTB       +1123/=2019/-890    52.89%  2132.5/4032
>   2   Rybka 3  no EGTB  +890/=2019/-1123    47.11%  1899.5/4032


Okay, I was right, ~ 2000 pairs in cutoff10. Let´s have a deeper look to cutoff10 results.

We have the mixed results:

- 268 pairs, where    EGTB engine is better: +271 =265 (3 pairs won by 2-0)
-  38 pairs, where no EGTB engine is better:  +38 = 38  (assumtion: no pairs won by 2-0)

     1   Rybka 3  EGTB        +271/=303/-38   69.04%  422.5/612
     2   Rybka 3  no EGTB   + 38/=303/-271   30.96%  189.5/612

Let´s look to the drawn pairs. Here we have

- 858 pairs, where the engines make 2 draws
- 852 pairs, where the engines make wins bothsides

- Result:                       +852/=1716/-852   50.00% 1710/3420

Upshot: We see, there is still some noise in cuttoff 10 seen by 38 pairs won by R3 no EGTB at depth 10. And we can´t calculate the noise in the drawn pairs (overwhelming number!). I suggest, we can forget the drawn pairs won bothsides, because of clear won positions. But I still think, cutoff 10 (2016 pairs) is a good place for further investigations (other depths or tc).
Only to have a better feeling for this issue (I was surprised so many times here in this thread) it would be fine to see the results for cutoff 9 and 8 and 7 and even 6. Okay, cutoff 8 is especial interesting (~1000 pairs?), but it would be also intersting to see if there is a number of pairs in cutoff 6 (I can´t imagine, but this wouldn´t be the first time I would be surprised; maybe you still see a number in cutoff 0-5, then the no EGTB engine played perfectly! at depth=8). And cutoff 9 or 7 would be also interesting to see any anomaly.

Conclusion: This test is really great! I´m not a big fan of EGTB, but I´m sure EGTB can help R3 in endgame very much. I´m not so interested in the current test with GUI EGTB=on. This is a bluff package. Here we will see the no EGTB engine playing like God (R3 EGTB) in cutoff 6 games. Okay, you can look to same! cutoff 6 games in both tournaments (if there any) and can say how many points no EGTB engine lose in these games. Not very interesting!
I´m interested in cutoff 10 (or 9 or 8) in replay at the position with depth=9 or =10 or tc 1'+1" (increment is important here). Only to show that EGTB works (for engines which are not perfect in endgames :-)!
Parent - - By bnc (***) Date 2009-07-17 04:23

> I´m interested in cutoff 10 (or 9 or 8) in replay at the position with depth=9 or =10 or tc 1'+1" (increment is important here). Only to show that EGTB works (for engines which are not perfect in endgames


I can create an openings database based on the position at the time of divergence with 10 or less pieces remaining. Additional matches can then be run with different fixed depths or time controls. Be advised that even a game at time control 0'+1" can take several minutes since each book move is awared the 1 sec initial bonus, thus making a match of 4032 games a potentially very lengthy process.  Another option is to create the openings database with positions only (Fen) instead of complete games. This would speed up the testing at time control 0'+1" as there would be no bonus time awarded for book moves. However this setup might lead to a small number of game "distortions" due to 3 fold draw repetitions being possibly handled differently.
Parent - - By Vasik Rajlich (Silver) Date 2009-07-17 05:32

> I can create an openings database based on the position at the time of divergence with 10 or less pieces remaining.


This would make classification faster - I could then probably do a couple hundred pairs instead of 50. Anyway, I think we have the general picture now.

Vas
Parent - - By bnc (***) Date 2009-07-17 16:08 Edited 2009-07-17 16:13

> (bnc)  I can create an openings database based on the position at the time of divergence with 10 or less pieces remaining.


> (Vas)  This would make classification faster - I could then probably do a couple hundred pairs instead of 50. Anyway, I think we have the general picture now.


OK ..I have created an openings database based on the position at the time of divergence with 10 pieces or less remaining.  :)  The number of games in the 10 piece database was 4032 so there are 2016 games in the opening database. I made the result in each game equal to the overall result of each pair. A result of  +:-   means the EGTB engine had the advantage. A result of  -:+  means the no EGTB engine had the better result. A result of  =:=  means a drawn paired result occurred. If you want to create a mini opening book of just the games where the EGTB engine had the better paired result, just sort on the result field and pick off the 268 games.
Attachment: EGTB_book_diverge10.cbv - EGTB book based on position of divergence with 10 pieces or less (2849k)
Parent - - By Roland Rösler (****) Date 2009-07-18 02:10
Many thanks for posting the best opening book I ever saw! :-)

Some shorts comments after a short time of investigation from my side:
1. I discarded 22 games because of duplicates (very rare indeed in 2016 games (pairs)).
1.1 One game -:+ was double (1 discarded)
1.2 Five games +:- were double and 1 game was quad (8 discarded)
1.3 Eight games =:= were double and 1 game was sixfold (13 discarded).

2. It´s intersting to see that no EGTB engine can win even 7 man against EGTB engine (with main 6 man EGTB; #485)!

Questions:
1. Is there any chance to eliminate the wins bothsides in the =:= games automatically (or maybe eval > 6)?
2. Is there any chance to eliminate the real draws (i.e by repetition, stalemate or something else) in the =:= games automatically?
Parent - - By bnc (***) Date 2009-07-18 11:45

> Questions:
> 1. Is there any chance to eliminate the wins bothsides in the =:= games automatically (or maybe eval > 6)?
> 2. Is there any chance to eliminate the real draws (i.e by repetition, stalemate or something else) in the =:= games automatically?


Thanks for the positive feedback and good catch on finding the doubles.  :)

#1 can be accomplished. I think trimming based on a high eval makes sense. However, I am not sure its wise to eliminate the "wins bothsides in the =:="
#2 is very problematic at best.

I created a newer trimmed down book based on the one I created yesterday by eliminating all but the last 2 moves in each game. If you want to know why it has to be the last 2 moves, I can explain that there is a bug in the ChessBase GUI thats been there for years. If you create an openings book based on just the last move and use that book in an alternating colors engine match, then the first move out of book in each alternate game is always the same as in the prior game . Confused ? LOL ..I can offer examples if anyone is interested.  In testing with the trimmed book,  I find that matches at 0'+1" are more practical.  So I will work on suggestion #1 and then consider using that book in a match at time control 0'+1".
Parent - - By Roland Rösler (****) Date 2009-07-19 01:48

>So I will work on suggestion #1 and then consider using that book in a match at time control 0'+1".


I like it very much to see this statement! :-)
When we have ~1500 "fair" positions in cutoff 10, then we will have 3000 games at tc 0'+1". After this tournament we will be more clever (wise)!
Vas is a smart guy and he has his own opinion (he has seen enough!). Maybe he is right, but I´m not convinced. It would be fine, you can show him, he is wrong.
To show tc 0'+1" gets better results than depth=8 is the first step.
If there are any discussions about the (positive! :-)) results, we will see further. Maybe we will go further and show that we can get >+n Elo at 0'+2" (in comparison to 0'+1)! :-)
If tc 0'+1" is worse than depth=8 we are lost! :-(
Parent - - By Vasik Rajlich (Silver) Date 2009-07-19 07:25
Ok, I see the point now - the divergence positions can be the starting point for the next batch of games. Yes, this sounds good.

When playing the 0' + 1" games, keep watch over the CPU usage times for both engines.

Vas
Parent - - By bnc (***) Date 2009-07-19 22:16

> Ok, I see the point now - the divergence positions can be the starting point for the next batch of games. Yes, this sounds good.


I have modified the opening book based on the EGTB test with divergence of 10 pieces or less. I reduced each postiion to just the last 2 moves to make matches at time control 0'+1" even faster. I have also added a Rybka 3 evaluation to the annotator  field. I let the Rybka 3 engine (1cpu and EGTB's) run for 5 seconds for each opening position. If a mate was found during the search, an eval of +99 or -99 was used. If the actual position was already a checkmate, an eval of "MATE" was used.  You can sort on the annotator  field and prune whatever opening positions you dont want.  As a reminder, the results field for each position are 1-0 if the EGTB engine had a better result in the 2 game pair, 0-1 if the no EGTB had a better result, and 1/2-1/2 if neither engine gained an advantage.
Attachment: EGTB_book_diverge10_evals.cbv - EGTB book divergence with 10 pieces or less reduced to last 2 moves and evals added (106k)
Parent - By Vasik Rajlich (Silver) Date 2009-07-20 06:20
Ok, thanks, I'll keep this around. :)

Vas
Parent - - By bnc (***) Date 2009-07-19 11:27

> If tc 0'+1" is worse than depth=8 we are lost!


Why are we "lost" ?!  I tend to think that the EGTB engine has a greater advantage in fixed depth tests than in a time control match, but that is just my guess at this point.
Let the results speak for themselves !
Parent - - By Vasik Rajlich (Silver) Date 2009-07-20 06:21
This is true for your combination of hardware and settings, since in your case the CPU usage times are higher for the EGTB version.

Vas
Parent - - By bnc (***) Date 2009-07-20 10:40

> (bnc)  I tend to think that the EGTB engine has a greater advantage in fixed depth tests than in a time control match, but that is just my guess at this point.
>
> (Vas) This is true for your combination of hardware and settings, since in your case the CPU usage times are higher for the EGTB version.


I must admit some confusion on my part at this point. In the original test, the no EGTB engine used significantly more CPU time, likely due to the handling of root positions. In the Tablebase GUI option enabled test, the EGTB engine used slightly more CPU time, which you classified as "the effect of 2% more speed is negligible" .  I drew no conclusions from the CPU data at fixed depth and I have no data yet from any time controlled matches which is why my initial statement was a just a guess.

Please advise  :)

P.S. The best defense if the NFL does not reside in the NFC conference .. I think a team in western PA can make claim to that  :)
Parent - By Vasik Rajlich (Silver) Date 2009-07-21 05:16

> In the Tablebase GUI option enabled test, the EGTB engine used slightly more CPU time


This just means that in practice the benefit of tablebases is slightly overestimated in your test (and for your settings & hardware). It's no big deal.

The Steelers defense will be pretty good next year, that's true. They carried that team ..

Vas
Parent - By Vasik Rajlich (Silver) Date 2009-07-18 06:43
Thanks. I thought that this would be easier to classify, but unfortunately it's harder - the divergence points are too soon. Here are the first five examples:

8/8/5p1r/2kb4/5R1P/P4P2/5K2/8 w - - 0 58

8/2k3R1/8/1p6/2rPK1P1/8/8/8 b - - 0 59

1R6/8/2k2p2/5PpP/4K3/1p4r1/8/8 w - - 0 69

1r6/5pp1/p3k3/8/8/8/P1P1R3/2K5 b - - 0 46

8/1k3p1b/3K1P2/1pB2P2/r7/R7/8/8 w - - 0 91


Vas
Parent - By Vasik Rajlich (Silver) Date 2009-07-17 05:31

> - 268 pairs, where    EGTB engine is better: +271 =265 (3 pairs won by 2-0)
> -  38 pairs, where no EGTB engine is better:  +38 = 38  (assumtion: no pairs won by 2-0)


This is in fact very consistent with my classification. 5/50 (10%) of the EGTB-is-better pairs had no obvious EGTB-related cause. It now seems like there was no hidden EGTB-related cause either.

I'm still not sure what you want to show. Even a "cutoff-6" sample will still yield a limited (but higher) Elo gain, for obvious reasons.

Vas
Parent - - By Vasik Rajlich (Silver) Date 2009-07-12 06:10
Thanks! This worked quite well. I have classified the game pairs from games 51-150:

still no apparent cause: 5

PP v PP: 1
RP v RP: 4

P v -: 9
RP v R: 11
NP v N: 1
PP v P: 1
NP v B: 1
BP v B: 1

RPP v R: 5

RB v BP: 1

BP v -: 3
BPP v R: 1
Q v PP: 1
NP v R: 1
Q v RB: 1
R v PPP: 1
BP v P: 1
Q v RRP: 1

By broad category:

no apparent cause: 5
equal material: 5
up a pawn: 24
up two pawns: 5
up an exchange: 1
other: 10

The most surprising thing is the lack of up-an-exchange positions. Up two pawns is also surprisingly rare. I guess endgames are pretty boring, it all comes down to being up a pawn. :)

Vas
Parent - - By Roland Rösler (****) Date 2009-07-13 03:14

>I guess endgames are pretty boring, it all comes down to being up a pawn.


Yes! :-)
This you learn in kindergarten. But this alone isn´t sufficient. What R3 without EGTB shows in game 12 isn´t good enough! Maybe she wasn´t watchful in kindergarten. :-)
Parent - By Vasik Rajlich (Silver) Date 2009-07-13 05:11
Are you talking about the P v B from the first game set? There must be some sort of bug there ..

Vas
Parent - By Bouddha (****) Date 2009-07-23 10:07
That was in the figures I expected.

We have seen more than 1 game in official Rybka tournament where the EGTB bug affected the result.

Therefore it was mostlikey possible that this can be seen in more games than what were thinking
Parent - - By Roland Rösler (****) Date 2009-07-10 06:55

>For the next 10,000 game test , the setup will be the same except that the GUI tablebases option will be enabled.


What do you want to show by this test?
Parent - - By bnc (***) Date 2009-07-10 10:35

> (bnc) Rybka 3 EGTB       2:08:27
> (bnc) Rybka 3 no EGTB  2:30:22


>> (Vas) Such a massive difference can only be caused by the handling of root positions, as opposed to probing inside the search. Would it be possible to later re-run the experiment, but allow the GUI to take over in EGTB positions for both engines. This data will help tell me whether Rybka probes too often or not enough.


>>> (Roland) What do you want to show by this test?


The match request came from Vas. This setup should reduce the advantage the EGTB engine has since the GUI will take over for both sides when a tablebase position is reached.
Parent - - By Vasik Rajlich (Silver) Date 2009-07-11 06:06
It will also give us meaningful CPU usage times. (Currently, they are obviously dominated by the time needed to play out tablebase positions.)

Vas
Parent - - By bnc (***) Date 2009-07-11 08:56 Edited 2009-07-11 09:09

> It will also give us meaningful CPU usage times. (Currently, they are obviously dominated by the time needed to play out tablebase positions.)
>


I am not actively tracking CPU usage time with the next 10,000 game test (GUI EGTB option active). Everytime I have used Task Manager to observe it, the CPU usage times of both engines have been very close to equal.

(The problem is that once the match ends, the Chessbase GUI terminates the 2 processes associated with the UCI engines. I usually run the match in 500 or 1000 game subsets. There might be a way to automatically track it using some other tool like the performance monitor)
Parent - - By keoki010 (Silver) Date 2009-07-11 14:12
bnc I have a problem with my log on screen saver blanking my screen after about 18 mins. I have an asus p6t mob i7-920 @2.67 GB
desktop.  I turned off the power saver options.  But something is still blanking the screen if I don't move the mouse or a key. Any help or pointer to where I can find info for this is appreciated.  Oh I have Vista Ultimate x64 OS.
Parent - - By Akbarfan (***) Date 2009-07-11 19:09
maybe try this: right mouseclick on desktop, properties, Tab Screensaver,  
click energy management, Monitor off: should be set to -> Never
 
My Os is Windows XP
Up Topic Rybka Support & Discussion / Rybka Discussion / Tournament: R3 no EGTB vs. R3 EGTB
1 2 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill