Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka 3 -- a hundred elo!
1 2 3 4 5 Previous Next  
Parent - - By Banned for Life (Gold) Date 2008-07-11 06:16
The correlation between engine tournament play with generic books and analysis is pretty tenuous in my opinion. When doing analysis, you are generally looking for the best move (which may be impossible to find) rather than something that's good enough. For this task, its not clear that the smallest possible branching factor is what you want.

Another issue is how well the engine does analysis working backward through a game. This is not a skill that is used during an engine tournament. There are also skills that are important for engine matches, but much less so for analysis, like time management.

So much as we might like there to be a strong correlation between the two, it may not really exist.

Alan
Parent - - By BB (****) Date 2008-07-11 06:20

>you are generally looking for the best move (which may be impossible to find)


Especially if it's a bishop underpromotion. :-P
Parent - - By Banned for Life (Gold) Date 2008-07-11 06:27
ARGGGGGGG!!!!
Parent - - By BB (****) Date 2008-07-11 06:42

>the best move (which may be impossible to find)


Surely you recall the Harrwitz-Staunton repartee (from Bird's Chess History and Reminiscences):

Staunton pretended sometimes not to see Harrwitz, and would look round the room and even under the chairs for him when he was sitting at his elbow, which greatly annoyed Harrwitz, who, however, sometimes got a turn, and was not slow to retaliate. In a game one day, Staunton materially damaged his own prospects by playing very tamely and feebly, and testily complained - "I have lost a move." Harrwitz told the waiter to stop his work, and search the room until he had found Staunton's lost move, and his manner of saying it caused a degree of merriment by no means pleasing to the English Champion.
Parent - By Banned for Life (Gold) Date 2008-07-11 07:34
:-D
Parent - - By Uri Blass (*****) Date 2008-07-11 15:14
I disagree and the correlation is certainly high.
It does not mean that always the better engine in games is better for analysis but when there is a big gap it is clear that the better engine is better.

3000 engine may be better than 3100 engine for analysis but
I do not believe that 2700 engine can be better than 3100 engine for analysis.

branching factor is clearly unimportant because branching factor tell you nothing about playing strength.
smaller branching factor can be result of better order of moves or better pruning that prunes only additional bad moves and in this case it helps and smaller branching factor may be result of unsound pruning that prunes good moves and in this case it may be counter productive also for games.

In correspondence games the target is also to win against the opponent and not to find the best move and it is better to avoid serious blunders.

If you cannot trust non rybka engines in the task of avoiding serious blunders then you certainly need rybka to avoid them.

Uri
Parent - - By Banned for Life (Gold) Date 2008-07-12 01:17
I think you're to some degree nit-picking so I'll be more specific. According to the rating charts, Rybka is 50-100 Elo better than Zappa. Does that mean that one should spend all their time analyzing with Rybka? Vas would probably say Yes! But: I disagree! Although with 100% probability, Rybka is better than Zappa in tournament play with a generic book, there are a quite a large number of positions where Zappa and Rybka disagree by a good margin on the evaluation. Rybka is right more often in general, but less often in positions with certain identifiable characteristics. I believe this is exploitable both in engine-engine play and in analysis.

Zappa also has some intangible advantages when it comes to doing analysis:

- It is much better at remembering the results of multiple variations when doing backward analysis. We all know how frustrating it is to check out a sideline and then come back to the mainline and see that Rybka has forgotten why the position is so good or so bad.
- Zappa allows more flexibility in controlling the engine. Zugzwang in an endgame position? Turn off null move. Think there is a potential mating sequence? Turn on mate extensions. etc.

Of course there are limits to this. You can't expect to compete doing analysis with an engine that is 300 Elo in the hole.

Rybka certainly has her uses. There is no engine better at doing overnight forward searches from the root to try to find deep opportunities or traps. But saying branching factor is unimportant is, I think, a mistake when looking at an engine's suitability for analysis. Rybka seems to have the lowest branching factor for a top engine. Zappa is on the other end of the scale (Anthony didn't get around to adding futility pruning until after the match in Mexico). In my opinion, the fact that these two approaches give results that are at least in the same ballpark Elo wise means it makes sense to look at both of them.

As far as correspondence games go, if you're happy getting mostly draws and a few wins from opponent blunders, avoiding bad moves is a reasonable strategy. In tournaments when you need a high score to win, this may not be the best strategy.

Regards,
Alan
Parent - - By Uri Blass (*****) Date 2008-07-12 04:56
I did not say that the best was to use rybka 100% of the time but some comments
1)For the future rybka3 is relevant so the gap to zappa may be 150-200 elo.
2)It is possible to achieve lower branching factors by many ways so the analysis is important and not the branching factor.
lower branching factor may mean some unsound pruning so you do not find some good moves and it also may mean good pruning when you prune only illogical moves and reduce almost always illogical moves so I cannot say if smaller branching factor is good or bad.

Bigger branching factor can be achieved by good extensions when the extension help or by extending too much when the result is that you cannot see deep enough in the important lines.

Only knowing the branching factor simply tells me nothing.
You can say that rybka may sometimes miss good moves regardless of time that zappa finds and this may be a relevant comment.

Uri
Parent - - By Banned for Life (Gold) Date 2008-07-12 05:10
For the future rybka3 is relevant so the gap to zappa may be 150-200 elo.

I have been playing Zappa against the predominantly Rybka engines in the CB engine room, and have not had a problem maintaining a winning percentage well above 50% with average hardware. This is done by favoring openings that lead to positions that are closed and/or require king attacks where Zappa seems to outperform Rybka. Will this change when R3 comes out? I suspect not, but we shall see.

I would be very happy if people would use only Rybka to do their analysis. Every engine has weaknesses, and once these are identified, anyone who is overly reliant on this one engine will be susceptible to exploitation of those weaknesses.

Regards,
Alan
Parent - - By Uri Blass (*****) Date 2008-07-12 06:11
I do not see why you suspect not.
I may be wrong but I guess that things are going to be changed with rybka3.

100 elo improvement is not nothing but maybe larry can say if there is improvement in rybka in king attacks or in closed positions.

Uri
Parent - - By Banned for Life (Gold) Date 2008-07-12 07:20
I find Rybka to be very uneven in strength. Some positions it plays extremely well, some amazingly poorly. You yourself have highlighted many of these weaknesses in the late middle and endgame (my favorite is the one where an unstoppable pawn gets more weight than the eventual queen). I tend to focus on finding specific opening positions that Rybka plays badly.

Maybe R3 will cure all of its R2.3.2a's problems and be a more well rounded engine, but given the way that Vas and Larry test I tend to doubt this. Their tests allow very weak play in a minority of positions to be counterbalanced by stronger play in the majority of positions. This approach tends to allow weaknesses that occur in only a small percentage of positions. The trick is to be able to find these positions where they occur close to the starting position so that they can be exploited.

By the way, I do not think it would be impossible to come up with a set of openings which allow movei to be very competitive with Rybka. I have previously posted some games that reached these types of positions. Of course this would be easier if movei was 10X faster. :-)

Regards,
Alan
Parent - - By Uri Blass (*****) Date 2008-07-12 08:07
Thanks for posting these games.
I searched in your post and finding the link

http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?pid=69623;hl=

I did not work recently about movei and I will probably not do it in the near future and
it is encouraging to see that movei can beat rybka in the right type of positions.

Uri
Parent - By Banned for Life (Gold) Date 2008-07-12 17:14
I playing a few hundred 3-0 and 5-0 blitz games on the Playchess server with Movei, running on a 3.6 GHz core, against other engines, mainly R2.3.2 MP on average hardware (average was a 2.4 GHz quad). Obviously Movei was losing a lot of games by missing tactics due to being out searched.

As I think you've stated in the past, Movei would probably be very competitive if it were 10X faster. That's obviously much more easily said than done. Adding in MP capability and taking advantage of 64-bit processing would probably get you about half way there (in a geometric sense).

Alan
Parent - - By BB (****) Date 2008-07-14 03:58

>Their tests allow very weak play in a minority of positions to be counterbalanced by stronger play in the majority of positions. This approach tends to allow weaknesses that occur in only a small percentage of positions.


Yes, it is interesting that the VR/LK testing régime seems not to actually look at any positions (unless Iweta is still doing this). My own personal methodology has been loath to tend toward this extreme.
Parent - By Uri Blass (*****) Date 2008-07-14 06:59
How do you know what Vasik and Larry are doing?

I believe that Larry does not test random changes but test changes that he believes in them.
Changes that he believes in them are probably based on positions when rybka went wrong so 
I believe that he looked at positions before deciding what changes to test.

Uri
Parent - - By lkaufman (*****) Date 2008-07-12 17:59
Basically all aspects of Rybka play have been improved. This includes king attacks/defense and closed positions. I would not say that the improvement in these areas is more than, or even as much as, the overall improvement of !00+ Elo points, so I can't say whether it is enough for Rybka 3 to outplay Zappa in such closed positions. Rybka 3 will at least make more of an effort to avoid such positions or to open them when they occur than Rybka 2.3.2a did.
Parent - - By Banned for Life (Gold) Date 2008-07-12 20:58
Rybka 3 will at least make more of an effort to avoid such positions or to open them when they occur than Rybka 2.3.2a did.

This is a very interesting statement. As we all know, a lot of draws occur when an engine unwittingly locks up the position. Rybka was probably worse than most in this regard because for instance it would frequently lock up its attacking pawns in opposite castled positions, thereby killing off its own attack. Do you believe you have made good progress in preventing this from happening? I suspect that fixing this one problem might add a few Elo points all by itself, since it is seen so frequently.

Regards,
Alan
Parent - - By lkaufman (*****) Date 2008-07-12 21:36
This issue is tied in with the contempt setting. Without it, Rybka 3 only has slightly more incentive to avoid blocked positions than did Rybka 2.3.2a. However, we plan to make a non-zero value of contempt the default, on the grounds that we should have "contempt" for everyone and everything! We are still testing what value to set as the default. However, if this issue is of particular concern to any user, he can simply set a higher value for contempt and that should help, although of course there are many other consequences of doing this.
Parent - - By Kapaun (****) Date 2008-07-12 22:04
I hope this won't slow her down further, since I got the feeling that if you are using a contempt other than zero with 2.3.2a, she will need longer to reach a certain depth...
Parent - By lkaufman (*****) Date 2008-07-12 22:13
I think this must be something that just happened by chance when you watched, because if there is any slowdown from a small contempt value it should be less than 0.1%, nothing you would notice. I would not be getting good test results with it if even a 1% slowdown resulted from using it.
Parent - By Milton (***) Date 2008-07-12 22:29

>...we should have "contempt" for everyone and everything!


LOL.  This would make a great public relations quote on Rybkachess.com :-)
Parent - - By Banned for Life (Gold) Date 2008-07-13 00:00
OK, I guess you must have expanded the concept of contempt since R2.3.2a if it encompasses pawn structure evaluation. Doesn't R2.3.2a contempt only deal with draws from repetition (and maybe 50 move rule).

Alan
Parent - - By lkaufman (*****) Date 2008-07-13 00:58
     Yes, we talked about this at length during our many handicap matches with grandmasters, but perhaps you didn't follow them.
Parent - By Banned for Life (Gold) Date 2008-07-13 08:10
Thanks. I must have been in Europe at the time those discussions were going on. I'll go back and review them.

Regards,
Alan
Parent - By Carl Bicknell (*****) Date 2008-07-13 11:59
lol - yes I like this attitude!
Parent - - By Felix Kling (Gold) Date 2008-07-13 13:58
Won't this hurt the result in Rybka vs. Rybka play? I think for engine tests against weak engines this would of course make sense, but when you go on a server and play against other Rybkas...
Also does this make a difference in analysis (displaying not 0.00 for repetition)?
Parent - - By lkaufman (*****) Date 2008-07-13 17:42
Contempt will be defaulted to zero in analysis mode, I believe. If you play other Rybka 3's on a server, you should just set contempt back to zero unless you have unusually good hardware. My tests show that even against Rybka 2.3.2a, a contempt setting of 10 or so is helpful.
Parent - - By Kapaun (****) Date 2008-07-13 19:37
That sounds wierd. Why would you need unusually good hardware? And why is contempt being changed, if you need unusually good hardware for it? And haven't you said yesterday that contempt does NOT slow down the search? Now what?

(scratching head)
Parent - - By lkaufman (*****) Date 2008-07-13 20:21
I think you have some misunderstanding here. Contempt does not slow down the search. It's main purpose is to reduce draws, in many different ways, at the price of not always choosing the objectively best move. If you are playing another Rybka 3 on equal hardware, there is no particular reason to avoid draws in completely equal positions, and so any contempt setting will theoretically hurt your chances. If you are playing a weaker opponent, either a weaker program or the same program on weaker hardware, you want to try to avoid draws when you are just slightly worse. So if you are playing a weaker opponent (as in all the testing on the rating lists etc. or when playing any unaided human) some contempt will help, and so we are planning to set a non-zero value as the default. If your opponent is equal, as may be the case in engine room play or freestyle events, you may want to set contempt back to zero, although if we end up with a small default setting like 10, the likely gain in resetting it to zero against an equal opponent is tiny, maybe 2-3 Elo or so. I hope this explains it clearly enough.
Parent - - By Kapaun (****) Date 2008-07-13 20:38
Ah okay, now I got it. But then there comes up the question why you take a nonzero default at all.
Parent - - By lkaufman (*****) Date 2008-07-13 20:45
Because testers will test the program as it comes, in default mode. Also the number of people who use Rybka 3 to play against other Rybka 3s is presumably much smaller than the number who use it to play against either humans (including themselves) or engines other than Rybka 3. The default should be the way most people will use it. My opinion is that even for engine room play, if you don't want to change settings every game and you have good hardware you are probably best off leaving it in default mode (or perhaps with a smaller but non-zero contempt) because sometimes you will be playing older Rybkas, other engines, or inferior hardware.
Parent - By Felix Kling (Gold) Date 2008-07-14 07:40
OK, that sounds reasonable.
Parent - - By Banned for Life (Gold) Date 2008-07-11 05:30
wasnt this match more of a book vs book battle rather than engine vs. engine?

Jeroen has stated on multiple occasions that the Zappa team lost the book battle convincingly. This would imply that Rybka lost the over the board battle even more convincingly...

Regards,
Alan
Parent - By BB (****) Date 2008-07-11 05:39

>Jeroen has stated on multiple occasions that the Zappa team lost the book battle convincingly.


In fact, he said: Zappa won the match because it made fewer mistakes. If we would reverse the openings and replay the match, I fear it could be something like 7-3 for Zappa.

See also this thread about the book battle.
Parent - - By 8lrr8 (***) Date 2008-07-11 05:39
"Jeroen has stated on multiple occasions that the Zappa team lost the book battle convincingly."

wow, didnt know that.  kudos to jeroen.

"This would imply that Rybka lost the over the board battle even more convincingly..."

i dont think we can make that determination over a sample size of 10 games.
Parent - - By Banned for Life (Gold) Date 2008-07-11 05:46
I am not generalizing so my conclusions is valid. During the Mexican match, Zappa was better. How do I know that? Zappa won the match.

For the 10 game match, Zappa started in the hole on many of the games and still came out with a positive score. It was the better engine during this 10 game match.
Parent - - By 8lrr8 (***) Date 2008-07-11 05:58
"During the Mexican match, Zappa was better.  How do I know that? Zappa won the match."

maybe zappa was better given those match conditions, but not because it "won the match."  by that logic, if fritz were to win a 2 game match against rybka, one would say "how do i know that?  fritz won the match."  10 games simply isnt enough.
Parent - - By Banned for Life (Gold) Date 2008-07-11 06:05
When  a football team beats another football team on any given Sunday, I say it was the better team that day. I don't say it will always be the better team, or even usually be the better team, and I don't play the what if game either. Zappa won in Mexico fair and square. Its time to accept that it was the better engine during the Mexico match.
Parent - - By 8lrr8 (***) Date 2008-07-11 06:11
"Its time to accept that it was the better engine during the Mexico match."

ok, i'll concede that.  but only due to sample size.  :-)
Parent - By Banned for Life (Gold) Date 2008-07-11 06:28
I'll concede it too, but only due to the phase of the moon and its effect on Dagh's psyche.

Alan
Parent - - By Kapaun (****) Date 2008-07-12 21:04
We all know that 10-game matches don't tell you which one the strongest engine is, but only which one won and which one lost. But on the other hand exactly that is the thrilling aspect of these matches: anything can happen...
Parent - - By Banned for Life (Gold) Date 2008-07-12 21:13
Sure, but this is also true of any OTB competition and we don't usually spend a lot of time griping that the best player didn't win. In fact, as you point out, people generally root for the underdog and usually the spectators are happier when there is an upset.

Alan
Parent - By Kapaun (****) Date 2008-07-12 21:20
Yes, certainly!
Parent - By Wayne Lowrance (***) Date 2008-07-12 22:10
Sounds a little bit of passing the buck
Wayne
Parent - - By 8lrr8 (***) Date 2008-07-11 02:46
"Zappa would demolish Rybka in that case if you double its hardware and keep Rybka's the same."

what do u mean by crush?  can u give quantify this?  such as an elo or expected score rating (overall a large sample size, of course)?
Parent - - By turbojuice1122 (Gold) Date 2008-07-11 02:51
Let's see...Rybka 2.3.2a with Jeroen and Dagh at the book against Zappa Mexico with Erdo at the book?  Zappa should score at least 60% in that matchup.  From Mexico, I think you can cancel out the "things that went wrong" with the fact that the Rybka version was significantly improved to Rybka 2.3.2a.
Parent - - By 8lrr8 (***) Date 2008-07-11 02:55
i'm just the messenger.  maybe whoever said that was doing it to save face, i dont know.  it's certainly possible.

didnt someone say erdo has extensive knowledge of what positions rybka 2.3.2 plays poorly, and used that knowledge to construct an "anti-rybka" book for the mexico match?
Parent - - By turbojuice1122 (Gold) Date 2008-07-11 02:58
Erdo seems to have done this, though this was somewhat negated toward the end when they had Rybka play with more variety in the openings.  After that happened, Rybka almost came back in the match.
Parent - - By 8lrr8 (***) Date 2008-07-11 03:03
right.  which provides evidence the mexico matchup was far more of a "book vs book" duel than an "engine vs engine" one.  at least compared to what u typically see.
Parent - - By kstevens (**) Date 2008-07-11 04:15
I don't see what the issue is here. An opening book is part of an engines strength just like an opening book is part of Chess. Who cares if the Zappa team created an opening book made for use in the Rybka match? This is what you are SUPPOSED to do. I don't understand all the complaining about this. When a GM prepares for a high level match against another GM they know they will be playing against, you think the book is thrown aside? The GM prepares an opening they feel will favor themselves and not the opponent. This is called opening preparation and is a very real part of a Chess game. There was nothing wrong with the Zappa team preparing an opening book to be used in the match with Rybka. This is part of playing Chess.
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka 3 -- a hundred elo!
1 2 3 4 5 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill