Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka vs Benjamin II - conclusions
1 2 3 Previous Next  
- - By Vasik Rajlich (Silver) Date 2008-01-08 14:11
Rybka has finished her match at draw odds vs GM Joel Benjamin and won by a score of 6-2. I'd like to thank Larry Kaufman for organizing the match, GM Benjamin for participating in what must have been a fatiguing experiment, and our sponsors for making it possible.

In many ways, the result of this match is no surprise. In computer vs computer play, draw odds is a lesser handicap than pawn odds, and Rybka had already won giving pawn odds against both Benjamin and Ehlvest. Going purely by Elo considerations, Benjamin would be expected to score just 1 or 2 points in this match.

Humans and Drawishness in Chess

The main reason for optimism for the human in this format is the theory that humans understand the drawing prospects in a chess position better than computers. There is some decent evidence for this belief, and Benjamin did slightly outperform his rating in this match. In fact, Benjamin performed in this match exactly as Hiarcs 10, Spike 1.2 and Fruit 2.2 did in CEGT 40/120 play against Rybka 2.3.2a, drawing and winning games at a 25% clip. These programs are already likely to be slightly stronger than the world's strongest human players.

This superiority of the human was demonstrated once in this match, in the fourth game, which was likely the most exciting and interesting from the human point of view. Benjamin made concessions to lock the position and then took the air out of it completely with a piece for two pawns sacrifice which sets up a rather complex fortress which no computer can even remotely understand.

Preparation and Preparation

Benjamin seems to have been reasonably well-prepared. He's been involved in computer chess for quite some time, understands the topics, and did appear to have a solid game plan.

Of course, you can always be better prepared, and some of Benjamin's experiments backfired. The best example is game 6, where he voluntarily made concessions only to watch the position explode immediately.

If Benjamin had been given a greater incentive to win, and had dedicated several months of preparation to the task, it's likely that a few of his errant experiments could have been avoided and perhaps the score would have been closer.

Human Fatigue

In previous matches between grandmasters and Rybka, the humans always performed better in the second half than in the first half. This match was the first exception. One possible explanation is that Rybka's extremely high draw-avoidance settings prevented simplifications, avoided draws by repetitions, and led to a general increase in the tension in each position and a related increase in the amount of energy required from the human to play each game. One thing is clear - there were no rest days, or even rest games, for Benjamin in this match.


The results very closely mirrored the outcomes of the openings. In three of the games (games 1, 2 & 4), Benjamin quickly obtained human-friendly positions. He scored two points in those three games and in fact nearly took the third point as well. In the remaining five games, for varying reasons, the opening positions got away from Benjamin, with at least moderate complications and somewhat open play. In none of these five games did Benjamin come anywhere near a draw.

What is Next?

As always, this is for Larry and potential sponsors to decide.
Parent - - By oudheusa (*****) Date 2008-01-08 17:24
To be quite honest I did not get Benjamin's match strategy, especially for such an experienced chess computer expert.

Especially in the second half of the match he went for open and dynamic positions that are a nightmare for humans to play against a computer.

I had predicted 5-3 for Rybka but think that a strong grandmaster with a very solid and conservative style can still beat Rybka in an event like this.
Parent - - By turbojuice1122 (Gold) Date 2008-01-08 19:23
Could he have avoided those positions?  That is why the Philidor was such a good opening choice against a human--it cedes an advantage to white, but at the cost of opening up the position into something where computers have a strong advantage.  However, I think that the downside is that if the match had continued for, say, 24 games, with Joel getting some more rest than what he had, Joel would eventually win a game or two against that opening--he would get "lucky" and not blunder.
Parent - - By lkaufman (*****) Date 2008-01-08 20:49
To be frank, I think that the Philidor would have been a dubious choice under normal scoring after a couple outings, because it is quite risky if White castles queenside and knows all the theory. But I think it is a perfectly acceptible opening if White castles kingside, as I expected Benjamin would do when a draw was satisfactory.
Parent - - By Laszlo (***) Date 2008-01-08 21:48
Bravo for the Philidor'choise! It was a decisive factor of the victory... It gives, like the Sicilian, a lot of inprevisibles pawn moves!
Parent - - By turbojuice1122 (Gold) Date 2008-01-08 22:17
Philidor's choice, or Philidor's curse?!  [*evil laughter*...]
Parent - - By Laszlo (***) Date 2008-01-08 23:13
sorry, i saw too late... "choice" :)
Parent - By turbojuice1122 (Gold) Date 2008-01-08 23:21
Okay, okay, but what about the curse? :-)
Parent - - By oudheusa (*****) Date 2008-01-09 08:42
Will Benjamin comment on the match as well? Or can Larry say something about it. You must have spoken, during and after the match. How did he feel it went; did he regret his opening strategy or did he feel he played against a superior Rybka?
Parent - - By lkaufman (*****) Date 2008-01-09 15:41
Joel did not indicate any regrets about his opening strategy. I don't think he has any idea of what he could do that would give him any better chances in a rematch. Of course he pointed out some oversights he made, and observed that fairly often he would realize that he had made an inferior move after he made it, but that is normal I think. He felt that for future matches we either must return to pawn odds, or allow the GM some takebacks (he felt that this would help considerably, but is rather contrary to the laws and spirit of chess), or allow the GM some computer assistance (like using Rybka on a sp machine for example). He felt that Rybka did not give static features enough weight by human standards, often making weakening moves that would be fatal between GMs, but due to Rybka's incredibly deep search she was able to find ways to complicate and make the weaknesses irrelevant. In short, Rybka is still below GM level in evaluation, but with 20+ ply searches this hardly matters against human GMs.
Parent - - By Uly (Gold) Date 2008-01-09 16:44

> he would realize that he had made an inferior move after he made it, but that is normal I think.

I didn't know that this still happens at the GM level.

> He felt that for future matches we [.] must [.] allow the GM some takebacks (he felt that this would help considerably, but is rather contrary to the laws and spirit of chess)

I find it as bad as Pawn odds, but we already had enough of those, so my "vote" goes for allowing take-backs (It'd be interesting to know how many take-backs a GM needs to beat Rybka.)
Parent - By turbojuice1122 (Gold) Date 2008-01-09 16:52
Actually, the proper term for this is the "retractor".
Parent - - By lkaufman (*****) Date 2008-01-09 16:53
I don't think that Joel believes that getting a takeback or two per game would give him much chance to "beat" Rybka, but it might give him good chances to win a match with the draw and White odds handicap. In short, takebacks would give a GM good drawing chances with White.
Parent - - By Uly (Gold) Date 2008-01-09 17:12
Sorry, I didn't understood. If it's "All white" + "Draw odds" + "Take-backs" I'm not interested.

How many take-backs would the GM need for a "fair" match against Rybka?
Parent - - By lkaufman (*****) Date 2008-01-10 02:36
My guess is that even allowing a 2600 GM to take back every move until making the next one would still leave him struggling for draws. Of course if you allow long sequences of moves to be taken back, there must be some number of takebacks that would even things up, but that number would probably be so large as to be ridiculous.
Parent - - By Uly (Gold) Date 2008-01-10 03:12
I see, so a take-back handicap is just not viable. So, what now? Are we doomed to have to use the same handicaps over and over? Or is still there some interesting handicap that hasn't been used yet?
Parent - By turbojuice1122 (Gold) Date 2008-01-10 04:01
Sure--make the computer play blindfold--it's not allowed to look at the board :-D.
Parent - - By lkaufman (*****) Date 2008-01-10 05:10
     There are several ideas. One that I suggested is castling odds, where the computer forfeits castling. Another is multi-move, where White gets some extra free moves (without crossing the midline) before the game starts. Material handicaps don't have to be limited to pawn or pawns; the Exchange (remove a1 rook and b8 knight) makes a good handicap too, though at the moment I think it's a bit too much for a GM at 40/2. Another option is two knights for a rook, which is probably slightly larger than the Exchange. Or knight for pawn (remove b1 knight from computer, f7 pawn from human), which according to my tests is about the same magnitude as just removing f7 from the computer.
     I have run some tests using the Rybka Randomizer on a recent private version on my quad at game/1'. I determined the Elo value of each handicap at this time limit. Naturally the handicap values should be larger at longer time controls, but perhaps the relative values of the handicaps would not change too much. Here are the results:
     Pawn and move (remove f7, I assumed that White would always open 1e4): 357 Elo.
     Exchange (remove a1 and b8): 303
     Three moves (White gets e4 and d4 free and plays first): 157 Elo.
     Two edge pawns (White removes a2 and h2): 203 Elo
     Knight for pawn (remove b1 and f7): 358 Elo

     If we make the educated guess that handicaps would be worth about 50% more at tournament time control, and if we assume that Rybka's rating against human opponents is 3050, then we should win against a 2600 GM at three moves, lose at f7 odds or knight for pawn, and perhaps split at Exchange odds, although I don't really believe we could do so. Humans can adapt to the handicaps more readily than the computer can, while engines always play the "best" moves, which is not optimum strategy when you start in a losing position. Maybe a goal of Exchange odds is a reasonable target for the next couple years. For now, pawn odds (other than f7), three moves, and castling odds are the most reasonable to me.
Parent - - By gala.martin (**) Date 2008-01-10 09:20
once again, I think time odds can be reasonable. For instance, you play a 10 games match with the following rules. Rybka plays at 5'+3'', while GM only limitation is to end up the match before 8 days have been elapsed. GM is allowed to use extra boards for analysis.

Of course, the numbers can me changed at need. There are several advantages
- good quality and exciting match
- strong motivation to improve rybka time management, and ponder time usage
- real chess is played, in which the starting position is the starting position, the rules are the rules, and a draw is a draw

- Live broadcasting can be a bit annoying (you can partially solve this, by asking that every day at 10:00 -or whenever- a new game has to start)
- If LK has to host the match, he really needs a lot of time.


Another possibility is to give severe material odds against unprepared players. For instance, during a chess tournament, you may offer money to the participant that defeats rybka at knight odds (I mean, the knight is removed from rybka's pieces). The rules are: one game at 5+3 computer time (the human can play on the PC at will), human must win to get the money. BTW, one would need an opening book explicitly prepared for the game, which can bu quite painful, as no knight odds databases exist.
Parent - By lkaufman (*****) Date 2008-01-10 16:37
Actually, I already have a small knight odds book for Rybka, which I could easily expand if there is a reason. I have played some knight odds games between Rybka and various visiting players, with or without time odds depending on their level, and so far Rybka has won nearly all the games. Perhaps I've been too stingy with the time odds.
Parent - - By turbojuice1122 (Gold) Date 2008-01-10 10:55
Wow, just looking at these, some give the same impression in my mind as if you had decided to introduce a special new piece on the board that has more powers for the GM than for the computer.  Time odds definitely seems better from the public standpoint (and it would seem more likely to get an opponent), or the (almost) equivalent of having Rybka play on a slower machine like a Pentium II or III or something.
Parent - - By lkaufman (*****) Date 2008-01-10 16:31
But time odds doesn't have much appeal to us, partly because everyone knows that computers calculate much faster than humans so a victory at time odds (even huge time odds) wouldn't be terribly impressive. Also, some people seem to believe that computers don't play much better at tournament levels than at blitz, and these matches are intended to show just how well Rybka can play with enough time.
Parent - - By Roland Rösler (****) Date 2008-01-10 18:19
Time odds matches are impolite! If you are searching the real task, make the next draw odds match with the sp-version of Rybka (best 32-bit). So you can show, that your software is best (and not your hardware).
Parent - By gala.martin (**) Date 2008-01-10 18:24
There is not much difference. The main point is that declaring "rybka won when given 1% of the time given to the human" is much more impressive than "rybka won even playing on an Intel iAPX 432" (very old cpu).
Parent - By lkaufman (*****) Date 2008-01-10 20:47
The hardware I used can now be purchased for under a thousand dollars, so it's hardly anything special now. Soon it will even be available on laptops. Running Rybka on obsolete hardware doesn't make much sense to me; it is just like giving time odds with ponder off but making the game take longer unnecessarily.
Parent - - By gala.martin (**) Date 2008-01-10 18:19
I understand your point. I assume that rybka is really strong, and I found exciting to see how a human performs against a strong adversary, when he is given a lot of time. However, for rybka marketing, it is more important to show how the engine performs. In other words
-to show human play features: engine plays standard chess (time etc), human is given long time (or other advantages).
-to show rybka play features: human plays standard, rybka is given some handicap.
That makes sense.

As for the knight odds games, the big shot would be challenging kramnik or anand just after their next match :) You could pop up to the loser, and ask: "ehi, consolation prize: if you win a knight odd match against my rybka running on my laptop, you get xxxx $. It takes just 5 minutes". Then you switch on a quadcore laptop (they will be out at reasonable price in a few months), and give him a chess lesson, thanks to your latest version of antihuman rybka and knight odds book :)

That's just joking, probably. If the next human to be challenged is not too strong, probably a knight odds game can be an option. In that case, a serious opening book is needed. I think you may ask some help about that to the forum community.
Parent - - By lkaufman (*****) Date 2008-01-10 20:56
It's hard to do a serious opening book for knight odds, when all moves are losing. All you can do is make sure that there is enough variety to avoid preparation or repeat openings. Actually I doubt that anyone in the forum has as much experience as I do giving piece handicaps, so in this particular instance I don't think I would have to ask for help.
Parent - - By gala.martin (**) Date 2008-01-10 22:41
Thanks for your reply. I have to say that your posts slowly converted me to the dark-giving pieces odds-side. Sometimes I play knight odds vs rybka, which I never did before. It would be nice to have a very basic pieces-odds (maybe pawn or knight) opening book merged with the new rybka3 book, just to get more variety in the games. A rude merging should work, since a standard opening book and a piece odds book do not share any position.
Parent - - By lkaufman (*****) Date 2008-01-10 22:52
There is no problem with this suggestion from my end, I could easily supply a small handicap book. The only problem might be that if the book is sold separately, the author might not want it to include this. Anyway if asked, I will supply such a book.
Parent - By gala.martin (**) Date 2008-01-11 10:56
great! i will see what happens. thx
Parent - - By Vasik Rajlich (Silver) Date 2008-01-10 23:23
That's very interesting.

Somehow, I'd think that a human GM would do better with knight for pawn than with f7 pawn and move. This might be related to the issue of Rybka overvaluing transient factors.

Parent - - By lkaufman (*****) Date 2008-01-11 03:49
I would think so too. It's possible that the result was a bit influenced by the choice of a large randomizer margin (25), since I started the tests with White already playing 1e4 as it's clearly the best move, whereas some of Black's replies within the randomizer margin may have been markedly inferior.
Parent - By Vasik Rajlich (Silver) Date 2008-01-12 10:40
Yes, that's possible. The current randomizer algorithm is really not very optimal - it's appropriate for handling the deeper part of that "Monte Carlo tree", but closer to the root, some sorts of adjustments should probably be made. We'll have to experiment with this later.

Parent - - By saxon (**) Date 2008-01-11 22:14 Edited 2008-01-11 22:22
Larry and Rybka team be brave!
Next match:Knight odds but Rybka always plays white:)
Can Rybka draw elo2500 under such circumstances?
Knight odds are really big advantage . It's a famous classic advantage with long history in the game
.It is the best known advantage strong players use to give  to much weaker opponents. 
If Rybka can do that that would certainly make headlines.
Parent - - By lkaufman (*****) Date 2008-01-11 23:10
Knight odds is always assumed to mean that the odds-giver gets White. If he has Black it would be called "knight and move". Yes, it would be truly impressive if we could do this, as it would be impressive if I killed a tiger with my bare hands or bench-pressed a hippopotamus. But none of these things will happen in the real world.
Parent - - By Vasik Rajlich (Silver) Date 2008-01-12 10:43
Actually, I'm pretty sure that I'd be favored vs a tiger in a fight to the death with no weapons, assuming some sort of 'neutral terrain'.

About the knight odds and bench-pressing an (adult) hippo, I agree :)

Parent - - By Banned for Life (Gold) Date 2008-01-12 14:52
Unless you can fly, I'd be betting on the tiger. :-)

The timing is a little off though. A month ago we could have flown you out to the San Fransisco Zoo to test this hypothesis! :-)

Parent - - By turbojuice1122 (Gold) Date 2008-01-12 16:08
From the sounds of it, I'm guessing that the tiger in question would have little interest in Vas unless he intentionally and directly harassed it.
Parent - - By Banned for Life (Gold) Date 2008-01-13 04:23
It's interesting that you don't think trying to kill it with his bare hands would qualify as harassment. You are a true Libertarian! :-)

Parent - - By turbojuice1122 (Gold) Date 2008-01-13 04:32
Well, since we're talking about an official "match" on "neutral terrain", then I assume that the tiger would have to officially accept the challenge from Vas.

Yes, I'm closer to Libertarian than I am to either of the two main political parties in this country. :-)
Parent - - By Vasik Rajlich (Silver) Date 2008-01-14 21:49

> I assume that the tiger would have to officially accept the challenge from Vas.

The underdog always has a chance :)

Parent - By turbojuice1122 (Gold) Date 2008-01-15 02:15
Well, my point is that the tiger can simply decline your challenge of a "match", especially since it has much more important things on its mind, such as using "The Force" on the idiots who were harassing and taunting it. :-)
Parent - - By saxon (**) Date 2008-01-12 18:13
I'm just telling you that GM I know (currently rated 2520) has quite a difficulty to beat Rybka in game with knight odds!
60 min time controls.I'm not kidding.Usually he can only draw it without much of the difficulties ,but to defeat the engine it's another story :).It's hard ,even for Elo2500.Think of it.
Parent - - By Felix Kling (Gold) Date 2008-01-12 18:21
This seems to be matching my experience. I had to play a lot of blitz games (3+0) to win at knight odds, I often missed a win in the end when I was close to winning or more often I was getting outplayed in the middlegame... Rybka seems to know how to cause problems :)
Parent - - By saxon (**) Date 2008-01-12 18:48 Edited 2008-01-12 18:50
Drawing issue isn't problem, of course.But in trying to win every game ,he even lost some :)
In some of these games he couldn't make any progress (in middlegame!) despite material advantage,and decided to sac a piece to continue the progress. Than Rybka defended ,equalized and won simetimes.Funny thing but looked like Rybka "knows" how to lock position. 
I think he plays Rybka on quad hardware.
Parent - - By SR (****) Date 2008-01-12 19:42 Edited 2008-01-12 19:48
I simply do not believe that. A grandmaster that cannot beat Rybka with an extra knight in a 60 min game. Either Rybka is much stronger on a quad or I have completely lost the notion of what a GM is capable of. Are you sure its 2520 elo and not some 2500 American soft rating? Is he a genuine grandmaster? 

I used to be an 2300 Elo player and not specially strong in blitz (I think I am currently something like 2450 in blitz rating on ICC but I did not play for some years), but I can speak the phone while beating Rybka leisurely with much shorter time controls.  With an extra knight on the board the machine only have one tooth (knight) that the human might decide to "pull out"  at the first instance. Without knights the machine is toothless and each exchange hurts the machine.  After a few more exchanges white is reduced to desperate fighter without any limbs.

Maybe grandmasters cannot play chess anymore?
Parent - - By saxon (**) Date 2008-01-12 19:55
Yes ,he's GM.It was strange to me when he told me that ,especially becouse it was longer time control.
He told he played knight odds ,but Rybka get full point if she survives ( makes a draw).
And he said he didn't won such match.I'll ask him to send me the copy of the games.
Parent - - By SR (****) Date 2008-01-12 20:17
Yes ,he's GM. It was strange to me when he told me that

Did you check this? I would not just take his word for this  (more likely he is a non-GM who is lying than he is a GM who cannot win playing with an extra knight)
Parent - - By lkaufman (*****) Date 2008-01-12 23:07
At first my attitude was the same as yours, this is impossible. But now I'm not so sure. My son, who is FIDE 2386 and just one norm away from IM title, just played the Benjamin match version two games on my quad at knight odds (with a tiny knight odds book), five minutes and five seconds increment, and lost both without making obvious blunders. In one game he reached the endgame with two extra pawns, but at the end he was down two pawns! He is not a bad blitz player, he scores around 25-30% or so against lower-range GMs. Of course he is not a 2520 GM and this is semi-blitz, but at least I can imagine that some GM with relatively poor technique (but perhaps extremely good opening play to compensate) might have trouble winning half the games outright with an hour each. Maybe if we pick the right GM and play rapid games Rybka could draw some games at knight odds. Perhaps I've underestimated the extent to which the many program improvements together with the book help at knight odds. As for your own experience, I can only say that playing on a 64 bit quad versus a 32 bit sp machine makes a huge difference.
Parent - By SR (****) Date 2008-01-13 00:02
Interesting! I am sure the playing on a 64 bit quad rather 32 bit sp makes Rybkas moves "sting" quite a lot more. But, maybe there is also another phenomena in play here.
Maybe there is a big difference in otherwise equally rated players, in playing computers. When I play a computer I do not "consciously" think about my opponent as a machine
but I think I do quite a lot of "higher level" thinking that would not work so well against humans. I always try to aim for middle game positions where the underlying "problem" needs a "logical" and principled treatment.  Or if this is not possible, try to reach balanced positions with good prospects for swapping pieces.

Later tonight I will try to play a few games at Night odds (I have done this before) and see how much effort is needed.
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka vs Benjamin II - conclusions
1 2 3 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill