Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Opening Analysis----reliability of CAP scores and +/= evals
- - By HartRook (*) Date 2007-12-28 20:37
Hi,

    Does anyone know how good the CAP evaluations are that are shown on the Rybka 2.3 Chess Openings 2007? They have an indicator for the relaibility score of "6" of "10" and the +/= type-scores are shown as "10" of "10".
     I have been building my openings based on the CAP scores shown but I now have major concerns that they are not that good. I wonder who determines that a particular position would get a (+0.15) for example, and if I ran my Rybka in infinite mode with the very best computer available, would I GET very, very close to the publised CAP scores?
    Do the people sellng Rybka itself confirm with that THEIR program will yield the CAP scores shown on the disk?
    Have good people in this forum tried to get together and combine resouces so that the primary trees are "correctly" scored, or shared their scores? I have had Fritz 10 running in infinite mode for the last 13 months almost non-stop to assess some 300-400 positions, and have built powerpoint files with the results
    Can someone help me? If you wish to start sharing info on opening evaluations--write me at mickeymike_2005@yahoo.com
   
      This is a great forum
               Mike
Parent - - By Permanent Brain (*****) Date 2007-12-28 21:25
I am not sure how popular the Chess Openings 2007 program is or how much attention it gets. For example, I have bought Rybka without it, engine only.

13 month infinite mode of Fritz 10, for 300 to 400 positions sound good (basically), that would be about one day per position. But how did you select these positions and what is the hardware you used? Also, if the result - I guess - is only an evaluation and a primary variation (pv), what did you put into PowerPoint?

I don't know if one Fritz day can replace 100+ years of master chess experience, nevertheless, the results shouldn't be too far away from the truth. - Of course you should have use Rybka instead! :-D
Parent - By Uly (Gold) Date 2007-12-28 23:31

> Of course you should have use Rybka instead!


This is very important, as Fritz's evaluations are more likely to be wrong, than Rybka's, on most positions.
Parent - By Felix Kling (Gold) Date 2007-12-29 00:14
"I am not sure how popular the Chess Openings 2007 program is or how much attention it gets."
It's quite similar to Chess Assistant 9 afaik, so it definetly got some attention :)
Parent - By Felix Kling (Gold) Date 2007-12-29 00:13
Those scores are as far as I know a mixture of the evaluations of different programs.
Parent - - By Victor Zakharov (*****) Date 2007-12-29 22:04 Edited 2007-12-29 22:07
The main problem of CAP is its irregularity. Evaluations went mainly from 4 egines: Crafty, Tiger, Shredder and Rybka. All engines have different scale of evaluations and different point of view on different classes of positions.

Moreover analysis time is irregular too.

So some variations are covered very deep (sometimes extremly deep) and another ones - not. And there is no way for common user to know what evaluations are adequite. Currently we are trying to go completely to Rybka evaluations. Number of positions will be less but people will not doubt what engine gave it. Complete CAP can be an option.

As for the quality I can say the following. CAP is sufficient for having common impression about a variation. CAP is not sufficient to play corresopondence games where very deep analysis is esential. Anyway it can give preview what directions should be searhed first.

In any case CAP quality grows quickly with increasing of engines strength and CPUs power and in a year or two it can become "must have" data source.

Currently 10-15 computers are busy with CAP calculations (mainly at nights) and other "idle" time.
Parent - - By Felix Kling (Gold) Date 2007-12-29 23:39
I think someday we will have Rybka randomizer scores, which really can tell us more about a position. Btw., if enough games are available, you could also calculate a "human score" calculated from the elo difference between player's rating and performance... this would be better than any engine evaluation i guess.
Parent - - By Vasik Rajlich (Silver) Date 2007-12-30 12:28
Indeed, we're going to work on this a bit more in 2008. If randomizer scores are better, then a sort of randomizer could be built into Rybka's search. This will require a lot of tinkering.

Vas
Parent - - By Banned for Life (Gold) Date 2008-01-02 01:26
The CAP score database would be a good place to test how useful this really is. If you had a large number of scores for both normal analysis and randomizer analysis side by side along with game statistics, you might be able to identify positional features where one is better than the other.

Alan
Parent - By Vasik Rajlich (Silver) Date 2008-01-03 14:46
We'll have to think about this.

The most important thing is moves (and relative scores). Absolute scores are secondary.

Vas
Parent - By ozziejoe (**) Date 2007-12-30 11:49
on a related question......Do the Cap scores get updated when you do a web update? Or are they static
Parent - - By HartRook (*) Date 2008-01-07 01:45
Victor and Others,

     Thank you for the useful information! I think as the CAP's become more consistently evaluted (examples: Rybka-only, at least 2.4 GHz operating for a minimum of 8 hours on an opening position, evalutions of later positions of openings to "build-back and confirm the correct CAP score from a given position" could and should be a major goal the entire Rybka community can contribute to. I would think starting with only ONE opening and focusing on ONE will show its powerful potential, and it will really catch on so that people all over the world will work together to get the major lines of each opening properly evaluated (recognizing that as later versions of Rybka comes out, futher refinements can be done)
    As I mentioned, I have studied a couple openings on Fritz by evaluating over 300-400 positions (probably more like 600-700) by running my single 2.8GHz CPU (1 G Ram) for roughly 12-24 hours on each postion by keeping my computer running 24 hours a day for the last 13 months. By using copy and paste, I have put the results in Powerpoint files I study from
    I just purchased a Intel Core 2 Quad Processor (Q6600); each core operates at 2.4 GHz with 3G of RAM to continue my efforts, only this time with Rybka 2.3a. The evaluations of Rybka already appear to be of higher quality than Fritz 10 or 11 with much less time needed, but I need to study this more to make sure of it; Rybka seems to score the positional features better while Fritz seems to over-emphasize the tactical possibilities. I think that in time, I hope that Chess Openings 2008 (when it comes out) will have links that will SHOW how a particular CAP was determined (what machine and speed and hash etc)
Parent - By JhorAVi (***) Date 2008-01-07 02:35
Atleast additional info should be included in each CAP position such as Machine, Analysis Duration and Engine name. This way CAP generators can incrementaly update only those positions with outdated specs..
Up Topic Rybka Support & Discussion / Rybka Discussion / Opening Analysis----reliability of CAP scores and +/= evals

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill