Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / importance of search tree pruning
- - By Vasik Rajlich (Silver) [hu] Date 2007-01-09 14:43
Masomusic wrote:

--------------------

Hi Folks and Vas,

Ok this will shock some (again), but I found a way to really give Rybka 2.2n2 single a hard time by descreasing the forward pruning severely of Hiarcs 11. Hiarcs 11 is an engine with far too optimistic pruning and then plays total unsound attacks and really annoyed me. Decreasing the pruning from factor 10 to 2 it played very steady chess suddenly. Basically I turned a drunk Kasparov into a sane Karpov by messing with this parameter Smile

It wouldn't be such a bad idea of maybe add a pruning option in Rybka as well. I never imagined the importance of it this much. So Vas, what about a slider for treepruning and tactics/positional play? Smile
Wouldnt surprise me if Rybka prunes too much as well and maybe less would improve and oversee less optional lines that are more important that they look... Wink

Note that Hiarcs 11 with default pruning setting gets killed on almost any game by Rybka 2.2!

Settings for tournament:
Hiarcs 11 (forward pruning 2, position learning off, rest default) vs Rybka 2.2n2 (default settings, disabled "display current move" to avoid issues)
P4 3.0 ghz, 256 MB hash, ponder off, 5-men TB's (with 16 MB cache), Arena 1.99b3 GUI, timecontrol: 40/20, 40/20, rest in 30min
Well chosen preset openings.

Here's the first game I tried with this setting, more to follow of course. Funny thing is that the author of Hiarcs has no clue, he and his testteam presume pruning set to 11 is best...

----------------------

Actually, this looks like an interesting thread. If you want to see "maso" under fire, have a look here :)

http://chessobserver.com/rybkaforum/viewtopic.php?t=1143

Just a few general comments:

1) Welcome to our forum, Harvey. Can I go ahead and send you a "Rybka Rules" T-shirt? :)

BTW - I enjoyed your report in Selective Search.

2) Pruning is not really the right word for what an engine is doing. Every move at the top of the tree is looked at and given enough time, everything will be seen. It's just a question of shaping the search tree.

3) You definitely don't want a uniformly-shaped search tree. Unless in the starting position, you'd like Rybka to spend as much time on 1. e4 e5 2. Nf3 Nc6 as on 1. e3 Nf6 2. Qg4 a6.

4) The amount of selectivity which an engine can afford to apply is proportional to how good her pruning heuristics are. If you prefer avant-garde terminology, it's proportional to how much chess knowledge the engine has.

5) It's unlikely that different selectivity algorithms will work against different engines.

6) It's also unlikely that you'll take a mature chess engine and improve it by 50 Elo by changing a setting which that programmer has made available for you. This becomes even more unlikely when if becomes clear that the official testers of that engine are running tests involving multiple days of CPU time :)

7) In general, engine parameters for mature engines have one of two properties:

a) They don't really change anything important
b) Changing the default value will significantly weaken the engine

There are some exceptions to this. In Rybka, "outlook" falls in this category. For whatever reason, this parameter for Rybka has a very broad peak - there is a broad range of values where the pros and and cons just about balance out.

Vas
Parent - - By Harvey Williamson (*****) Date 2007-01-09 19:48 Edited 2007-01-09 19:51
Hi Vas,

Thanks for your warm welcome to the forum! I am gald you enjoyed the Selective Search article.  I am told it is being translated into Dutch and will appear in the CSVN Magazine, if any of  your forum readers from Holland are interested.

Best Wishes,

Harvey

ps I look forward to receiving the t-shirt
Parent - - By George Tsavdaris (****) Date 2007-01-09 20:58
For what Selective Search article are you talking about? Any link....?
Also where(in what topic) did you post this article in the old Rybka forum?

I can live without answering me the second question but not the first :-)

Thanks.....

BTW: Preview of this new forum sucks! :-)
Parent - - By Harvey Williamson (*****) Date 2007-01-09 21:18
Hi George,

The article was written for a British Computer Chess magazine called Selective Search http://www.elhchess.demon.co.uk/sscontnt.htm

So its not free I am afraid but you can subscribe or order a single issue.

Best Wishes,

Harvey
Parent - - By Eelco de Groot (***) Date 2007-01-14 12:38 Edited 2007-01-14 12:54
I am sure that Vasik does not want me to turn this into The HIARCS Chess Community in my very first post, but I had one small positive Hiarcsy result to add to Masomusic's thread. I like to be able to change settings in a program, especially if this does not hurt the performance.  Chessmaster is a great example of this but tuning Pro Deo or Glaurung, Gambit Fruit (!) also has still a lot of uncharted territory. It can really be a great pastime, although it's pretty addictive as well... I think it could be a welcome addition for the Rybka fans if Vasik will re-introduce this kind of "Personalisation" possiblities again in version 2.3 or so like he has written here he will try.

About tuning Hiarcs, I wanted to say that instead of decreasing Hiarcs' selectivity it is also possible to increase it and so, like "that Hiarcs guy" has done, even further "ruin a good program".

I had some incidental success with my Hypermodern IIc settings in Blitz yesterday when for the very first time on my computer Rybka Beta 1.0  was beaten in a short Blitz match! 

The previous Hypermodern IIb settings seemed to perform slightly worse than Hypermodern IIc, although these are of course only a very small matches. But beating the Rybka Beta in Blitz like Hypermodern IIc did, on equal hardware, is still a very rare occurrence!


Engines                 Win      Score   Wht win Blk win Draws
Rybka 1.0 Beta 32-bit   55 %     11,0    5       4       4
HIARCS 11v2 Hyper IIb   45 %      9,0    4       3       4

TimeControl "300" (game in 5 minutes)
ERT 0.9.0.7 Autoflag=On Pondering=Off OperatorTime=300 mS Start positions=Corus 2006 Wijk aan Zee Nunn-type openings.pgn


But this was a first!


Engines                 Win      Score   Wht win Blk win Draws
Rybka 1.0 Beta 32-bit   42 %      8,5    3       3       5
HIARCS 11v2 Hyper IIc   57 %     11,5    6       3       5

TimeControl "300"
ERT 0.9.0.7 Autoflag=On Pondering=Off OperatorTime=300 mS Start positions=Corus 2006 Wijk aan Zee Nunn-type openings.pgn


This was I believe the shortest victory from Hiarcs in the match

[Event "Test Hiarcs 11 Hypermodern IIc [7,14,6 + Combinations]"]
[Site "Groningen"]
[Date "2007.01.14"]
[Round "2.2"]
[White "HIARCS 11 UCI"]
[Black "Rybka 1.0 Beta 32-bit"]
[Result "1-0"]

1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. Qe2 b5 6. Bb3
Be7 7. a4 Bb7 8. c3 O-O 9. O-O d5 10. d4 Na5 11. Bc2 dxe4
12. Nxe5 c5 13. dxc5 Bxc5 14. Bg5 Qd5 15. Bxf6 gxf6 16. Ng4
Be7 17. Ne3 Qe5 18. Nd2 h5 19. f4 Qe6 20. Qxh5 Rfc8 21. Kh1
Rc5 22. f5 Qe5 23. Ng4 Qd6 24. Nxe4 Bxe4 25. Nh6+ Kg7 1-0

Hiarcs Hypermodern IIc Settings:


Selectivity 7 [Default, I would try 8 if that were possible]
Forward Pruning 14 [Default = 10, 12 at first try did not seem to work quite as well as in X54 Hypermodern]
Threat Depth 6 [Default = only 3, probably 6 can introduce some Horizon effects]
Hypermodern On [Default Off]
Combinations On [Default Off]
Book Learning Off [Default On, turned off only for testing purposes]
Position Learning Off [Default On, turned off only for testing purposes]


Eelco
Parent - - By masomusic (**) [nl] Date 2007-01-16 19:05 Edited 2007-01-16 19:08
Hiarcs 11 (FP5) vs Rybka 2.1o (default) +12=17-23 (= 39% / 61%)

Hiarcs 11 (default) vs Rybka 2.1o (default) +9=19-24 (= 36% / 64%)

Special note:
I especially chose for the "obsolete" Rybka 2.1o as against Rybka 2.2n2 any current Hiarcs version
gets killed in a very humiliating way :-)

masomusic
Attachment: Hiarcs11vsRybka2.1o.zip (346k)
Parent - - By Eelco de Groot (***) Date 2007-01-17 12:04
Interesting that Forward Pruning 5 works so well, Masomusic, and seems better here than FP 10?! Why not, as a further experiment, repeat the match against Rybka 2.2? Is Rybka 2.2n2 much better against Hiarcs 11?

I played a small Blitz Return match with my Hypermodern IIc, this time the match was even, new openings were used made from the first Rounds of Corus 2007. Rybka Beta lost on time in one game, otherwise that game would have been a draw. I am sure Rybka 1.2 or 2.2 would have fared better.



Engines                 Win      Score   Wht win Blk win Draws
Rybka 1.0 Beta 32-bit   50 %     10,0    3       4       6
HIARCS 11v2 Hyper IIc   50 %     10,0    3       4       6

TimeControl "300" (game in 5 minutes)
ERT 0.9.0.7 Autoflag=On Pondering=Off OperatorTime=300 mS Start positions=Corus 2007 Wijk aan Zee Nunn-type openings.pgn


Regards,
Eelco
Parent - - By masomusic (**) [nl] Date 2007-01-17 12:37
Hoi Eelco,

Indeed, seems that against Rybka decreasing the pruning of Hiarcs 11 helps a bit. Not as much as I expected though in the end. Against other engines I don't know if this is true, according to the author it isn't. It's no sense to play against Rybka 2.2 as no-engine at moment is capable of competing more than 35%.
There's a uge strength difference between 1.0b and 1.2f/2.2n2 in rapid/tournamentplay. To me it's amazing that Vas' first "public" brainchild is as strong as Hiarcs 11 in some timecontrols.
Parent - By Vasik Rajlich (Silver) [hu] Date 2007-01-18 11:54
There's one other basic statistical thing which you should take into account.

If you start with twenty versions, and play them all against each other, the best one is going to look much better than it really is. The difference might well look "statistically significant".

Same thing happens if you're just playing around with an engine until you "notice a pattern", and then run a bigger test.

The guys who analyze the stock market even have a very cool word for this: "survivor bias".

Vas
Up Topic Rybka Support & Discussion / Rybka Discussion / importance of search tree pruning

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill