Not logged inRybka Chess Community Forum
Forum RybkaChess Help Search Register Login
Up Topic Rybka Support & Discussion / Rybka Discussion / Hiarcs 12 worse than Rybka232a ..
1 2 3 4 Previous Next  
- - By sainzlei (*) [tw] Date 2008-03-30 05:40
According to CEGT test ..

CEGT Blitz 40/4  

1   Rybka 2.3.2a x64 4CPU  11½½½½0½101½1½½½0½½1½½1½1½0½½½1½10111½1½1110½½½111 32.0/50
2   HIARCS 12 4CPU             00½½½½1½010½0½½½1½½0½½0½0½1½½½0½01000½0½0001½½½000 18.0/50

http://husvankempen.de/nunn/phpBB2/viewtopic.php?t=976

Regards.,
sainzlei @ Republic of Taiwan
Parent - - By Jeroen (*****) [nl] Date 2008-03-30 06:02
Here some more results from CEGT (same link):

Rybka 2.3.2a 64-bit 4CPU - Hiarcs 12 4CPU:  32-18
Naum 3.0 64-bit 4CPU - Hiarcs 12 4CPU:   26-24
Zappa Mexico 64-bit 4CPU - Hiarcs 12 4CPU:   27,5-22,5
Hiarcs 12 4CPU - Deep Shredder 11 64-bit 4CPU:   32-18

Rybka 2.3.2a 64-bit 2CPU - Hiarcs 12 2CPU:  32,5-17,5
Naum 3.0 64-bit 2CPU - Hiarcs 12 2CPU:   27-23
Zappa Mexico 64-bit 2CPU - Hiarcs 12 2CPU:   26,5-23,5
Hiarcs 12 2CPU - Deep Shredder 11 64-bit 2CPU:   29-21
Parent - - By Permanent Brain (*****) Date 2008-03-30 08:45
So, any big expectations (if there were any?) are disappointed, at least in blitz. - Let's wait and see if it is much different, at medium or long time controls...

P.S. I was a Hiarcs fan long before I was a Rybka fan, and remain it. An engine doesn't need to be stronger than Rybka, to be interesting for me.

It looks like all competitors will be at least one 'version generation' behind after the Rybka 3 release, if they are even behind 232a. Amazing. Although, maybe I underestimate Deep Fritz 11. - From Junior, no signs and no news visible. So I think, DF11 will be the last big hero for a while, to try to challenge Rybka 2.3.2a. :-)
Parent - - By Werewolf (*****) [gb] Date 2008-03-30 10:20
I think the competeitors will be 2 versions behind.
The results clearly show that 2.3.2a is still top, and the NEXT generation of progs will be match it. therefore rybka 3 will be equal to the version after that, i.e shredder 14 / hiarcs 14 etc

Style matters a lot now
Parent - - By Not the real Kasparov (***) [au] Date 2008-03-30 11:40
After playing several blitz games against Hiarcs 12 on a quad against my e6600 dual core Rybka 2.3.2a at playchess, I am dissapointed in Hiarcs, Rybka basically crushed Hiarcs in most of the games, hopefully things are a little different in longer time controls. It's incredible that Rybka is still ahead even after 12 months worth of competitor programming against it. Well done Rybka team. Rybka 3 should be a monster. The gap between Rybka and other engines is ridiculous, kind of like Kasparov and the rest of the field.
Parent - - By Kreuzfahrtschiff (***) [de] Date 2008-03-30 11:44
but rybka 3 will be not much better than 232a, at least with new fast computers, only with old ones  :)
Parent - - By Werewolf (*****) [gb] Date 2008-03-30 11:56
that's baseless. There are excellent signs so far that R3 is well ahead of 2.3.2a
Parent - - By Kreuzfahrtschiff (***) [de] Date 2008-03-30 12:09
i have sources :)
Parent - - By Werewolf (*****) [gb] Date 2008-03-30 12:15
same.
Parent - - By Kreuzfahrtschiff (***) [de] Date 2008-04-14 16:33
as you could see in the last freestyle rybka 3 had NO advantage

i actually won some games only because zappa mexico 2 made some good moves
Parent - - By Kappatoo (****) [de] Date 2008-04-14 16:43
What do you mean with that? I heard of three teams which used the latest Rybka. Two of them qualified for the final, and the third one scored +2. What am I missing?
Parent - - By Kreuzfahrtschiff (***) [de] Date 2008-04-14 16:57
means 8 rybka 2.3.2a are in the final

and they did´nt play worse :)
Parent - By Kappatoo (****) [de] Date 2008-04-14 17:01
If you had to make a guess: How many Rybka 2.32a did not make it to the final?
Parent - - By lkaufman (*****) Date 2008-04-14 17:09
Besides Rybusia, which other team with new Rybka do you refer to as being in the final?
Parent - - By Kappatoo (****) [de] Date 2008-04-14 17:13
You probably know better, but I read (in the German section of this forum, inter alia) that Moro Gamgee was another 'official' Rybka team.
Parent - - By lkaufman (*****) Date 2008-04-14 17:28
In that case, there were at least 3 teams using new Rybka who made the finals.
Parent - - By Kappatoo (****) [de] Date 2008-04-14 18:21
:) Were you among them? And if not, is ist 3 out of 4 or 3/5?
Parent - - By lkaufman (*****) Date 2008-04-14 18:50
Yes, I qualified. I thought that Flyingfatman also had the new (or at least a recent) Rybka, but I could be mistaken on this. So either 3 or 4 qualifiers (at least) used a recent Rybka version.
Parent - By Fulcrum2000 (****) [nl] Date 2008-04-14 19:22 Edited 2008-04-14 20:14
Wow, this *is* news! Congratulations!!
Parent - - By Felix Kling (Gold) [de] Date 2008-04-14 20:06
Larry, did Dzindzi help you in freestyle? this would be cool :)
Parent - - By lkaufman (*****) Date 2008-04-14 22:52
After the final, I'll be glad to answer any questions about my participation. I'll just say for now that I had no prepared opening book at all, although of course I had access to all the standard ones during the games. I would also say that my reaching the final refutes the notion that freestyle success requires extensive experience. Not only have I never played freestyle, I've also never played correspondence chess. Of course I do have a lot of experience using engines in general and Rybka in particular for analysis, but I've never used a computer to play any of my own chess games.
Parent - By tomski1981 (****) [ca] Date 2008-04-15 01:13
maybe this experience will prove addicting ... (just so long as you keep it under control and don't ever cheat ;) )
Parent - - By turbojuice1122 (Gold) [us] Date 2008-04-15 04:08
In the finals on your first attempt--this is quite an achievement--congratulations!
Parent - - By lkaufman (*****) Date 2008-04-15 14:27
Thanks. Of course having the new Rybka and an octal machine helps quite a bit; four of the ten qualifiers used the new Rybka!
Parent - - By Wayne Lowrance (***) Date 2008-04-15 22:58
hi, what is the name of your team
Thanks
Wayne
Parent - By Roland Rösler (****) [de] Date 2008-04-16 00:12
Parent - By Werewolf (*****) [gb] Date 2008-04-14 17:44
This argument is flawed for the following reasons:

i) Freestyle is very different to engine play where the humans play a large role.
ii) the sample size of games is far too small to draw any conclusions. If you kept up with CEGT etc you'll see that  a 50 game sample only give an accuracy of plus/minus 100elo. For 9 games I bet it's plus/minus 200 elo - making any comparison meaningless.
Parent - By FWCC (***) [us] Date 2008-03-31 00:52
Rybka 3 will be unbelievable in strength.My first impression of H12 was that it is a slight dissapointment when I played a single version and I guess I was correct.
Parent - Date 2008-03-30 17:58
[topic branch moved]
Parent - By NATIONAL12 (Gold) [gb] Date 2008-03-31 21:03 Edited 2008-03-31 21:06
remember Hiarcs 12 is not a bit base engine and is only 32 bit.by the way i ordered today my FB-DIMM memoy through ORCA COMPS they say 5-8 days.this is the memory that Kullberg reccomended to me. god its expensive 4G £165.
Parent - - By Italian81 (****) [us] Date 2008-03-30 21:26
Fischer would have EASILY annihilated "Mr. OhMYGodIHopeICanDrawEveryMatch Kasparov."
Parent - By davidwhite (***) Date 2008-03-31 22:46
Odd placement for your Fischer/Kasparov comment but,I suppose,you could restrain yourself no longer.
Parent - - By Uri Blass (*****) [il] Date 2008-03-30 20:49
Hiarcs is more than one version behind Rybka2.3.2a
Even when we give hiarcs 2 cpu against single cpu of rybka1.x then hiarcs12 is clearly behind based on CEGT 40/20

17 Rybka 1.1 x64 2959 17 17 1796 82.7% 2687 20.7%
22 Rybka 1.2f x64 2949 16 16 1464 74.3% 2765 29.6%
37 Hiarcs 12 MP 2CPU 2910 33 33 287 40.8% 2974 35.5%
Parent - By Werewolf (*****) [gb] Date 2008-03-30 21:26
in my post above i claimed 2 versions. That applies to all the other progs too.
Parent - - By George Tsavdaris (****) Date 2008-03-30 11:47

>So, any big expectations (if there were any?) are disappointed, at least in blitz. - Let's wait and see if it is much different, at medium or long time >controls...


For 1 CPU and 32bit using own books, unlike CEGT and CCRL,  Hiarcs 12 performance seems very good and it beat Rybka 2.3.2a:
http://www.hiarcs.com/hiarcs_games.htm
Parent - - By Felix Kling (Gold) [de] Date 2008-03-30 11:51
this test was with 2 CPU, also the hiarcs book seems to be highly optimized against Rybka II ctg, so the test basically tells us nothing.
Parent - - By Ben Lau (*) [us] Date 2008-03-30 12:02 Edited 2008-03-30 12:17
The result of Blitz 40/4 says nothing at all. Just like the penalty kick. Such short a time control is just for funny play. To set the time to 40/120' you'll find something different.
Parent - - By turbojuice1122 (Gold) [us] Date 2008-03-30 12:59
I don't think you'll see anything different--Hiarcs is better compared with other engines at short time controls compared with long time controls.

However, I would bet a large amount of money that Hiarcs 12 would beat Naum 3 in true match conditions of 24 games.  I would not bet either way in a match between Hiarcs 12 and Rybka 2.3.2a, as I think that both sides have an equal chance of winning such a match.  The same goes between Hiarcs 12 and Zappa Mexico II.
Parent - By Ben Lau (*) [us] Date 2008-03-30 13:02 Edited 2008-03-30 13:06
Sounds 50 50 would be the better choice.
Parent - - By Juergen Faas (**) [de] Date 2008-03-30 13:23 Edited 2008-03-30 13:25
>>...match between Hiarcs 12 and Rybka 2.3.2a, as I think that both sides have an equal chance of winning such a match. <<

This 32 - 18 result doesn´t convince you? why not?
Parent - By George Tsavdaris (****) Date 2008-03-30 14:03

>This 32 - 18 result doesn´t convince you? why not?


I guess because:
He said: true match conditions of 24 games. So:

•This match was using blitz time controls, while true match conditions would probably mean long time controls of more than 2 minutes per move on top hardware.

•This match had 50 games, while turbojuice1122 speaks about 24 games, where a "surprise" can happen more frequently.

•This match was not using program's own books, while true match conditions would probably mean with engine's own books.
Parent - - By turbojuice1122 (Gold) [us] Date 2008-03-30 15:34
This 32-18 "result" is not a result of testing the programs at playing a game of chess.  It is a result of testing analysis.  You will continue seeing these ridiculous types of results at the same time that you continue to see Hiarcs beating Rybka in tournaments.  Again, Hiarcs loses at least 60 elo relative to other programs when one makes the change from own books, which is what you have in match conditions, to random "general" books.
Parent - - By Jeroen (*****) [nl] Date 2008-03-30 16:23
This is wrong. Hiarcs doens't 'lose' 60 points in matches with equal conditions.
In matches with equal conditions it is simply inferior to Rybka. Hiarcs' book can
make up for that by playing good lines for Hiarcs, so the book brings extra elo
points and not vice versa.

To call the 32-18 result 'not a result of playing chess' is very funny. Why is this
result ridiculous? Because it doesn't match with your expectations!? This is what
people often do when they are disappointed with result: they simply call them
ridiculous :-).

Therefore I call the result of the forthcoming european football championship
ridiculous when Holland doesn't win it. Then it simply is not a true tournament for
this title. Holland can beat any of these teams competing and therefore should win
the title.

(Sorry, couldn't resist this cynical joke!)
Parent - - By turbojuice1122 (Gold) [us] Date 2008-03-30 17:19
The reason that Hiarcs loses 60 points in matches with same general opening books is that when matches are played with own books for all engines, Hiarcs performs much better.  For example, Hiarcs 11 1 CPU with own book played against other engines with own books achieved 2880 on the CEGT rating list.  Interestingly, one can then go from there and use the recent test results by the Hiarcs team: while these aren't necessarily useful in getting the absolute strength of the engine (bullet games, after all), they ARE useful in getting its strength relative to Hiarcs 11.2.  In that case, we see a 70 elo improvement.  Thus, we're now basically at an estimate of 2950 CEGT for 1 CPU.  Now consider that Hiarcs 11.2 is about 20 elo stronger than Hiarcs 11, and all of a sudden, Hiarcs 12 1 CPU with own book, compared with other engines at own book (including Rybka), has the same rating as Rybka 2.3.2a 32-bit 1 CPU.

The reason the 32-18 result is ridiculous is because it uses different conditions than what exist in normal play to give a very different strength estimate than what is estimated based on normal play.
Parent - - By Jeroen (*****) [nl] Date 2008-03-30 17:44
So basically you say: the latest Hiarcs book is very good. And it has been well tested against RybkaII, Perfect 13 and so on.
Does that say something about the Hiarcs strength, or about the book strength? Will Hiarcs 12 be close to Rybka 2.3.2a if
Rybka plays with a more recent book? As RybkaII is about 16 months old now, it is not too difficult to make a very strong book
against it. So what are you saying? They killed RybkaII and now suddenly this is caused by 'Hiarcs being almost equal strong to
Rybka'?

Can you perhaps post the games of the Hiarcs-Rybka match played by Mark, so we can see how many Hiarcs 12 wins were
because of the book?

CEGT plays with a general book that every program has to use. This means equal conditions, no possibilities for book wins,
just measure engine strength. For all programs the same conditions. The 'extra elo due to book wins' disappears and that gives
a much better view on the engine strength than with books. At least, in my opinion :-).

So the problem is that you mix up engine strength with book strength. Given the proper conditions (i.e. best book for Hiarcs and
a worse book for the opponent) you are inclined to think that Hiarcs 12 is close to Rybka. But it isn't :-). Give Zappa Mexico the
latest Erdo book and Rybka my latest book and maybe the gap between Zappa/Rybka and Hiarcs could even be much bigger than
in CEGT.
Parent - - By Titu (*) Date 2008-03-30 18:30
Also if I may add, there are people out there who uses the engine to analyse chess positions and they don't to play lots of playchess engine vs engine tournaments. For them obviously the opening book does not matter so much. They want an universal engine which can analyse correctly in all positions, convert advantages and defend inferior positions. For me it does not matter if there are lines in Hiarcs 12 book which plays well against 16 months old Rybka book. I buy the engine to analyse. For me CEGT and CCRL tests makes more sense to me, where opening books have less impact.
Parent - By Jeroen (*****) [nl] Date 2008-03-30 18:41
I agree. In fact the product 'chess program for PC' has 3 components:

A) the engine
B) the book
C) the GUI

If playing strength is all that matters, you only search for the best A. Of course in that case you'd like to compare results
without book influence.

If you look for the best combination of engine and book (f.e. when you are a correspondence player), then you search for
the best A and B. It is very well possible that the best engine doesn't come with the best book, so a combination of engine A
with the book of engine B could be a reality here.

If you look for a nice GUI with lots of possibilities and strength and book doesn't matter, you can get yourself the GUI that
you like best and download a free engine with a free book.
Parent - - By George Tsavdaris (****) Date 2008-03-30 18:51

>CEGT plays with a general book that every program has to use. This means equal conditions, no possibilities for book wins,
>just measure engine strength. For all programs the same conditions. The 'extra elo due to book wins' disappears and that gives
>a much better view on the engine strength than with books. At least, in my opinion


I don't agree with this.  Actually i agree but let me describe what i mean:
•Yes with the way you describe we measure engine's strength on a wide range of positions.
•But what about if a programmer does not want his engine to play some kind of positions? That is the reason(among others) to create a specialized book for its engine. And perhaps with this book the engine may play extremely well not losing a single game! Let's have an example(random):

Hiarcs 12 on a wide range of positions with 1.f4, with 1.e4 e5 f4 etc, may be 100 ELO points worse than Rybka 2.3.2a playing against a big number of other engines, like it happens in CEGT CCRL.
1)But perhaps there is a book that when Hiarcs 12 plays against the same opponents(Rybka 2.3.2a too) and the opponents have their own best suitable books too, as also Rybka 2.3.2a to have its own best suitable book, then Hiarcs to be 20 ELO better than Rybka 2.3.2a.
2)Perhaps there is a book that makes Naum 3.0 stronger than everything else, and Rybka 2.3.2a, even if everyone else uses the best book suitable for them and Rybka using the latest Rybka III.ctg book.
Improbable? Maybe. The latest situation 2), but what about the first 1) ?

So in short we have 2 kinds of tests:

1) CEGT,CCRL type of tests where it tests engine strength in a wide range of positions. So it actually tests the analysis abilities/strength of the engines in a wide range of positions.
2) Own book type of tests(SSDF), where it tests engines at their highest strength they can have, if we assume own book is the most suitable for the engine.
Parent - - By Jeroen (*****) [nl] Date 2008-03-30 19:12
Of course it is good to have BOTH. The more data the better. In general I would be unhappy with an engine
that can play only a limited number of lines, but with the data of both CEGT and SSDF I could figure that out.

BTW, I find the SSDF list rather unsuitable for comparing engines with books, as there are a lot of good engines
missing and a few of them are not playing with their best book. I think it would be rather helpful if they take
a CEGT type of approach, i.e. a single version list and a MP version list and do faster time controls. Now you
see results like Zappa Quad - Fritz 9 A1200 with something like 36-4, all 40/120 games, not very interesting IMO.
I would love it when they make a special Quad list with all the latest versions of the programs available, with their
own books.

To find out which engine plays best with which book is interesting, but probably this is impossible to find out
within a reasonable time. You will have an enormous amount of possible combinations engine-book :-).

But to give you a counterexample which might be confusing for you as a customer:

Suppose you find information on several sites that engine A with book A beats Engine B with book B. You confidently
assume that engine A is the better one, so you purchase it. Alas, when you start playing your correspondence games
you are outplayed most of the times..... And your opponents are all using Engine B (they tell you after you resigned).
Now wouldn't you be a bit pissed? What I want to tell you is that relative engine strength DOES matter :-). Another
counterexample is when your engine plays only a few lines well and (too bad) it appears in your correspondence
games it doesn't understand anything of your gambit lines and King's Indian's. Now wouldn't you be better off with
an engine that understands a wider variety of lines!?

Have a nice evening!

Jeroen
Parent - - By Nelson Hernandez (Silver) [us] Date 2008-03-30 21:00
Delighted to see you agree with my points (from the other thread) so comprehensively!  You'll notice I was the first to object to the muddling of engines and books.  It seemed transparently obvious that you can't beat up on a 16-month-old opening book and boast that it proves anything.
Up Topic Rybka Support & Discussion / Rybka Discussion / Hiarcs 12 worse than Rybka232a ..
1 2 3 4 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill