Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka vs Ehlvest - my impressions
- - By Vasik Rajlich (Silver) [hu] Date 2007-03-08 22:53
Rybka has completed her match against Jaan Ehlvest at pawn odds. I'd like to thank IM Larry Kaufman for organizing the match and for being the driving force behind the anti-human Rybka version which played in this match. I'd also like to thank Grandmaster Ehlvest for being bold enough to participate in the experiment.

Our "information center" about this match can be found here:

http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=519

Larry and I will need to analyze a bit more closely what happened. Below are just my first reactions:

I) The format is interesting

A one-pawn advantage while having black in all games is rather mild. It seems that the results are competitive and the games themselves are played along normal chess themes and therefore interesting to watch. A too-big handicap could feature bizarre manoevers, but there was no sign of this in these games. The middlegames which were reached were normal chess middlegames which could have been reached from the normal starting position.

2) The story of two halves

Ehlvest started with 0/3 and, judging by those games themselves, he could hardly complain about the score at that point. Each of these three games was settled quickly and decisively, and not due to any special blunders. In fact, in game 3, Ehlvest found several forced moves in a tough position before succumbing.

In the rest of the match, Ehlvest scored 2.5/5 and more than held his ground. His game 6 win was rather convincing. In game 7, he again outplayed Rybka, only to allow the game to slip into drawn territory at the last moment as the clock wound down. In game 8, he used his knowledge of rook endgames to untangle himself. His only loss in this stretch came in game 5, in a long complex game which was balanced for many moves.

Did Ehlvest learn as the match went along? Are the games with queenside pawns missing somehow friendlier for a human? (There is no indication that this is so in engine vs engine play.) Or is this just normal statistical variance? We'll do our best to come to some sort of a conclusion.

3) From the human point of view

Aside from his dramatic improvement in performance as the match progressed, Ehlvest also deserves credit for avoiding major blunders. For some reason, man vs machine events tend to be marred by at least one such blunder from the human who participates. Thankfully, we did not have any of those here.

4) Takeaways for Rybka development

There were a number of Rybka behaviors which will need some analysis and possible correction. The most obvious were:

a) Early middlegame of game 6 - Rybka should understand that the solid pawn structure of that game allows black to untangle himself at his convenience. Black's advantage is probably already massive once the queenside is locked with a4-a5, since white's remaining central break with e3-e4 is harmless.

b) Endgame of game 6 - Rybka should not offer trades into simple lost king and pawn endings, no matter how bad the alternatives are.

c) Middlegame of game 7 - it's not clear what caused Rybka to play so indecisevely there, I haven't seen this before. Perhaps it was the strong incentive for avoiding trades. A position like that requires more forceful and consistent play.

d) Endgame of game 7 - needs more analysis. Rybka may have made some mistakes there.

e) Late middlegame of game 8 - Rybka needs to understand that this pawn-up rook endgame offers minimal chances. White has better prospects in the double-edged QRN vs QRB middlegame.

5) Future handicap matches

This will need some discussion. I think that some improvements to Rybka are needed before she is ready for a pawn-and-move handicap given to a player such as Ehlvest. A slightly longer time control should also be considered.

Vas
Parent - - By Carl Bicknell (*****) [gb] Date 2007-03-11 23:45
What the world REALLY wants to see is Rybka V Kramnik with no handicap at all. Then we'd see the world's strongest chess engine beat the w/c and quash the rumous that Fritz is best v humans etc.

And the other thing we need to see is Rybka V Hydra. I have emailed Chrilly to ask what's going on with Hydra (most people think it's been abandoned) - so far I have recieved no reply...

A year ago, however, Chrilly did a test suite for me which allowed me to compile 6 positions that hydra had to solve. these were typically deep tactical positions. The scores were good and a long way ahead of anything that p.c's have produced IMO. If anyone wants this test suite let me know...
Parent - - By grolich (***) Date 2007-03-12 09:08
Being better at certain tactical positions (i.e. compiled benchmarks) must be the thing that Hydra is good at.
Unfortunately, it doesn't say too much about the strength of the program in actual play.

I'd be very surprised if the results did not show better tactical performance than any PC.

Rybka may (or may not... it depends) fail on many of these tactical examples, but during a game, it knows where to look (at which positions to concentrate and calculate deeper, and which ones not to look at), which allows it in most cases to actually look further ahead on the important lines than other programs / monsters. It may be worse in certain positions, but in the majority of positions, it's just better, and its overall strength is higher.

Can Chrilly do the same with Hydra? Sure... But forward pruning is used by Rybka an order of magnitude more aggressively than in any other program to date. Chrilly has focused on other aspects.

Granted, the fish fight would be interesting... And is indeed something I would love to see:
The fish against the monster (Hey, doesn't that sound like "David vs. Goliath"?).

I would not even dare to predict the results of such a match (I believe Rybka may be a favorite, but cannot be sure).

As for Kramnik: Why Kramnik? The guy is out of form, and there are a few players who are playing better now.
He's getting back to his previous level, but he's not quite there yet.

While I'm at it, Larry, have you ever tested Rybka on an 8 CPU system?
If so, what is the difference in strength when compared to a quad?
If not, What's your estimate?
Parent - - By Carl Bicknell (*****) [gb] Date 2007-03-12 11:08
Weeeell lets just think about this.

In Summer 2005 (that was almost 2 years ago, i.e Hydra will be +100 elo since then) Hydra scored approx 3064elo V Adams
In Winter 2006 Fritz 10 scored 2904elo V Kramnik. So by those records alone we can assume Hydra is 160elo stronger than fritz and allowing for the improvement since 2005 maybe 260elo better.

Rybka is amazing. But it's not 260 elo above fritz.
Parent - - By grolich (***) Date 2007-03-12 11:34
First, I simply don't think Hydra has improved by any more than a couple of dozen points during that period.

Another thing is that performance rating of a single match cannot be used for rating evaluation (not even for a rough estimate, if that match was against a human).
Not only that, but Adama played (and still plays) a lot lower than his rating for a while, so I have my reasons to believe that even that performance rating is bloated.

Finally, ELO ratings of computers vs. computers are different than humans vs. computers.

I've been following everything that has been happening in the past few years and I have reasons to believe that Hydra and Rybka are very close to being evenly matched.

Rybka being the favorite is just my educated guess, though I won't bet my life on it.
In any case, I don't see Chrilly rushing to the challenge.

Come to think of it, I don't even know if Vas wants (or think the fish is ready for) such a challenge.
Actually, getting a reply from Vas about this would be grand.
Parent - - By Vasik Rajlich (Silver) [hu] Date 2007-03-14 13:59
Rybka doesn't get tired - we play everywhere. A match vs Hydra would be pretty cool, actually. The project is not that old - as recently as Dec 2005 they had entered Paderborn.

Vas
Parent - - By Ernst (***) [nl] Date 2007-03-14 18:49
Hi Vas,

I can't remember Hydra to be there in December 2005. I think the last time was February 2005. They withdraw in December, just when Rybka entered for the first time. ;)

Ernst.
Parent - By Vasik Rajlich (Silver) [hu] Date 2007-03-18 22:30
Yes, I remember. That was also the first time Anthony trotted out the big hardware.

Vas
Parent - By Fischerandom [se] Date 2007-03-29 13:13
If you were developing Hydra, wouldn't you use Rybka and some of the other strong engines to challenge Hydra in the lab?
Now, if Hydra is not the winner, or the clear winner, in such matches I suspect there is no point for the Hydra team to allow
a match between Hydra and Rybka. If Hydra (which is essentially Deep Blue III) would loose such a match against Rybka
running on much slower hardware, the purpose for Hydras existence would be questionable and thus the Hydra teams
existence would also be questionable because the hardware is so expensive and the time put into it must pay off.

But eventually a Hydra vs. Rybka match must happen, because Hydras title of "possessing ultimate dominance in the
computer chess world" will otherwise not mean anything with time. Rybka has nothing to loose on such a match since
Rybka is going to run on much slower hardware than Hydra. It would probably benefit the developement of Rybka and it
would be very interesting to watch! Regardless if Rybka wins or not, the approach of Rybka is the correct path,
bruteforce hardware is not, its a waste of energy.
Parent - - By lkaufman (*****) Date 2007-03-12 12:35
     Hydra played four more games against top human players in a computer vs. humans event after the Adams match, scoring two wins and two draws against about 2700 level opponents, which reduced its performance rating overall to just about 3000 for the ten games. Rybka would probably perform at at least that level on a quad (or maybe a octal, which should add around 40 points given equal processors, I would estimate), maybe higher. How much Hydra has been improved is just guesswork. While we don't know whether Hydra on 64 processors is stronger or weaker than Rybka on 8, it is pretty obvious that Rybka is stronger with an equal number of processors.  
    
    
    
Parent - - By Legendary (***) Date 2007-03-12 12:52
I don't think the Hydra team will challenge The Rybka team in a chess match. I think the reason why is because the Hydra team probably still can not figure out how to defeat Rybka 2.1.
Parent - - By Carl Bicknell (*****) [gb] Date 2007-03-12 13:21
hehe, I love these debates.

Anyone want to see the Hydra testsuite?
Parent - - By ernest (****) [fr] Date 2007-03-12 14:56

> Anyone want to see the Hydra testsuite?


I certainly would!

Did Chrilly also give the detailed analysis result Hydra achieved?
Parent - By Carl Bicknell (*****) [gb] Date 2007-03-12 15:58
yes he did although I found the printout hard to understand. If you look at the IBM printout of Deep Blue's analysis during the Kasparov match you'll see what I mean. In the end I just asked for the timings and he was good enough to edit the posts so I got these. Chrilly is very helpful.

One thing that did strike me as odd though is the way Hydra analyses. Usually with a multi-cpu machine all the cpus work at once to give more processing power. But with Hydra it is very different. It starts with only a few CPUs (FPGA cards to be exact) and these ask other CPU's to randomly join in to help. Apparently it's called random redundancy. So easy problems are solved just as quickly by P.C's but harder problems give time for all of Hydra's firepower to join in and it's staggering what it can do.

I'll create a separate post with the test suite in it.
Parent - - By Michael Waesch [de] Date 2007-03-12 16:15
There will be no such thing or do you have a million bucks to spend to sponsor such an event?

Mike
Parent - - By Carl Bicknell (*****) [gb] Date 2007-03-12 16:21
what are you talking about? Hydra v Rybka?

If so then you may have forgotten that the Hydra team sponsered an 8 game match with Shredder out of their own pocket. A million bucks is a drop in the ocean to the Sheik.
Parent - - By Michael Waesch [de] Date 2007-03-12 16:33
I don´t care about Hydra. The Playchess account of them is well known for being one of the biggest cherry-pickers, only playing extremely high rated opponents for not loosing too much of their oh-so-precious rating points. Hydra = Put it on the trash dump. Once and for all - or at least the people running it.

I was talking about the proposed match Rybka-Kramnik ...

Mike
Parent - - By Carl Bicknell (*****) [gb] Date 2007-03-12 16:37
oh. Yeah the Rybka-Kramnik match prob wont happen.

I didn't see anything wrong with the way chessbase portrayed Hydra though. I can't see how they benifit from it.
Parent - - By FWCC (***) [us] Date 2007-03-12 17:07
I don't care about Hydra.In my opinoin Rybka is the TOP DOG right now number1.As far as Hydra-put up or shut up.Where is Hydra right now?Where does it stand in CC world?This thread is about impressions of the Ehlvest match.My question is will Vasik implement what was learned in the Ehlvest match to FUTURE Rybka versions AFTER he does his homewrok and figure out the weak spots that occured in the match.Lets remeber the GM had a PAWN advantage and lost the match.This should tell everyone the strength of Rybka.I credit Ehlvest for remaing steady during most of the games and NOT blundering.(No "Blunder Of The Century") occured during this match,credit to Ehlvest who put up a strong fight and a good second half!
Parent - By Michael Waesch [de] Date 2007-03-12 17:27
Drawing valuable conclusions about this match could take a long, long, long - very long time.

Currently I am at analysing game 1 of the match. It already took me 3 days and I am not even near of any good analysis.

The only somewhat substantinated result of my efforts so far is that engines turn out to be next to useless when it comes to analysis, providing seemingly good variations and then, when you check them out in a painful, hours and hours lasting effort to verify the initial results you just end up losing your queen.

And this just led me to another conclusion: To really get an impression of what was going on and to draw conclusions for the further development of Rybka one would need:

* At least three top rated grandmasters
* At least five 8-way computers with the best engines running 24h/day
* 2 months for the game analysis - even under these favorable conditions.
* half a year of implementation into Rybka + plus deep testing

===> Don´t expect results before V3.XXX

Mike
Parent - - By Michael Waesch [de] Date 2007-03-12 17:19
I don´t see any sponsor popping up, so the Kramnik match will not happen. People should start to live with it.

I simply don´t like people who only play others where they can be sure not to severly damage their rating. Such rating-mongers don´t even deserve a rating.

And concerning Chessbase: Do you really know all their connections to the last bit, so that you can give a totally sure jusgement of whether they profit or do not profit from something. Well, I don´t. But it´s well known that their accounts did not get erased out of the top 100 when not played an appropriate amount of games in a 3 months period.

Mike
Parent - - By Legendary (***) Date 2007-03-14 19:58
You won't be seeing a Kramnik vs Rybka match any time soon. Unless Vas can generate $ 1,000,000 in the near future....
Parent - - By Banned for Life (Gold) Date 2007-03-14 20:19
Chessbase didn't pay Kramnik to play against DF10. They found a sponsor to put the money up in exchange for advertising their brand name in association with the match. Its just another form of advertising. Chessbase is reasonably good at arranging these kind of things and they have the ability to promote the events.

Regards,
Alan
Parent - - By Michael Waesch [de] Date 2007-03-14 20:24
Bad deal for the sponsor now associated with a lesser engine.

Mike
Parent - - By Banned for Life (Gold) Date 2007-03-14 20:28
The sponsor couldn't care less about the engine but if most people are like me and can't remember their name, they have wasted their advertising Euros. This isn't a reflection on Chessbase though.

Regards,
Alan
Parent - - By Michael Waesch [de] Date 2007-03-14 21:04
I don´t think it is that simple. Numerous people were arguing afterwards like this:

1) Why the hell did they choose Fritz and not Rybka?
2) 1.000.000 bucks prize money? Can this be real while people starve on this planet?
3) They must betray us with their treaties for providing energy when they have so much money to simply throw into the trash bin!

So I stick with it: Not a good deal for the sponsor.

Mike
Parent - - By Banned for Life (Gold) Date 2007-03-14 22:15 Edited 2007-03-14 22:18
Mike,

The chairman of RAG enjoys playing chess and has sponsored other chess events in the past. The event was promoted by Universal Event Promotion which has worked with Chessbase in the past. The sponsor's interest in this is to gain brand awareness and to get people to have a positive view of the brand. There's no reason for RAG to really care about what engine is used. The number of people that are aware of relative engine strength is rather insignificant anyway. The important thing was to have a well promoted match on a fairly even footing and in this regard, DF10 was probably a better choice than Rybka, since along with the match rules, it allowed for an expectation of a reasonably close outcome. So my response to your responses would be:

1) DF10 was chosen rather than Rybka, because it fits in much better with the primary goal of the match, promoting the sponsor. Rybka is the strongest engine, but that's not the point. Being a great engine developer doesn't make one either a credible sponser or a credible promoter. I'm always amazed to see that chess enthusiasts have such a hard time with this simple concept. Its clear that your personal bias is preventing you from acknowledging that Chessbase is the best promotional vehicle in the chess domain.
2) Most companies have an advertising budget, despite the fact that people are starving on this planet. Rest assured the $ 1M is not a significant portion of the RAG advertising budget. RAG also has a charity budget because this is also good for the brand. Rest assured that the charity budget is much, much smaller than the advertising budget. This is just the way of the world. Nothing new here.
3) As an American, I'm only vaguely aware of RAG's existance, so this is way over my head!

In any event, RAG will evaluate whether this was a good deal by evaluating the quantity and quality of the press coverage, using the same methodology as they would use for sponsoring a Formula 1 race or a football team. They don't really give a rat's ass about DF10 being weaker than Rybka. Trust me on this one. As a final thought, it would be very amusing to have you involved in a planning meeting for one of these events since you are totally out of sync with the goals and methods associated with this type of effort.

Regards,
Alan
Parent - - By Michael Waesch [de] Date 2007-03-14 22:29
And I thought you can´t even remember their name...

Mike
Parent - By Banned for Life (Gold) Date 2007-03-14 22:40
At my age, Google acounts for about 95% of my brain. :-)
Parent - By Michael Waesch [de] Date 2007-03-14 20:23
Fully agreed.

Mike
Parent - - By Ricardo_dk (*) [dk] Date 2007-04-08 20:45
Hi, thanks for your last message ( it was about i have to reinstall windows of my computer, i hope you can remenber that). I am sending now your last message so you can rembenber well:

http://www.rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?pid=6694#pid6694
Board: Rybka
Topic: Rybka 2.3.1 Beta
User: Vasik Rajlich

Of course, in this case, you can get a link from either me or Convekta.

Vas

Now i am using my computer a lot so i can not send my pc to computer "hospital". I think i will do it in summer ( Juli ), when i will make a travel and come back in one week.
I hope that it is not a problem to get (again) the rybka's 2,3 link from you or convekta,ok?
Thanks, i will wait for your answer
Parent - By Felix Kling (Gold) [de] Date 2007-04-08 21:12
Just send an email to info@chessok.com (don't forget your serial number if you have bought Rybka on CD!) and ask them to send you a new link.
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka vs Ehlvest - my impressions

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill