Not logged inRybka Chess Community Forum
Up Topic The Rybka Lounge / Computer Chess / 12 Engine, 2 round, round-robin 171019
- - By Peter Grayson (****) Date 2017-10-22 12:18
Being completely dismayed to see two move openings being employed in the TCEC season 10, my immediate thought was "what is the point?"

The inclusion of a two move Bird's opening seemed to me to make the whole Stage 1 event farcical. Surely the engines are designed to have strength going into the middle game? As good as today's engines are, it is difficult to envisage engines being able to rewrite opening theory in already well defined and established lines. From my perspective, I think it would have been much more of a spectacle to incorporate opening lines that are recognisable as being from contemporary human chess games.

I recall the first chess computer I bought circa 1982 had two and maybe some 3 move openings ... that was the Sargon 2.5. So here we are in 2017 with immeasurably more sophisticated engines when most would probably see off all of today's GMs and yet they are being given openings that may be taught in Kindergarten chess classes.

There is also the question of the inclusion of some of the engines that seem to be just making up the numbers that again does little to inspire enthusiasm for the first stage. Perhaps for the lesser established engines a qualifying stage may be warranted. The only surprise from Stage 1 will be if Houdini, Komodo and Stockfish do not form the top 3 engines going through to stage 2.

However, this caused me to consider the question "Why not use opening books for such a competition given the nature of it?" Inspired by that thought I decided to run a 12 engine, 2 round, round-robin tournament where each engine had their own opening book of the same construction and for this event with learning turned off. I may run with learning on in a subsequent tournament for comparison.

As far as I could see, there were no real shocks in the results and there was no significant performance distortion? As a spectacle, the games looked more attractive, appropriate for the sophistication of the engines and in most cases where I could identify with the thematic of the lines.

Machine dual Xeon E5-2675w + 64Gb RAM
Engines' resources: 6 cores, 4 Gb hash except Deep HIARCS 14 being 32 bit maximum 2 Gb
Time control: 40 moves in 5 minutes repeating to a finish, ponder = on.
Criteria for opening book: Human games in the last 12 months to 16/10/2017 with reducing criteria of removal of games where -
1. Both players<2400 Rlo
2. One player <2000
3. Games <20 moves
4.Draws <30 moves.
This gave just under 60,000 games to form the basis of the ctg book that was used with tournament conditions, minimum 3 games and learning off.

The three Stockfish engines took the top three positions with the other engines performing as expected. asmFish was undefeated and sad to see the once acclaimed Rybka engine at the lower end.

Attachment: - The Games (200k)
Parent - - By Labyrinth (*****) Date 2017-10-22 21:38
I wonder why Brainfish did so badly. 74 Elo less than generic Stockfish 111017.
Parent - By Peter Grayson (****) Date 2017-10-23 11:55

> I wonder why Brainfish did so badly. 74 Elo less than generic Stockfish 111017

Yes, that surprised me a little too but only relatively badly with 3 Stockfish code representatives in the tournament and of course with opening books there is a higher element of luck than when using my normal mode of testing that is fixed line sets. Brainfish still finished above all of the non Stockfish code engines. In a two round, round robin tournament I would not read too much into the tournament performance Elo's because the ordering of the engines in the final result looked about right.

Brainfish's speed would be down around 6 to 8% because I had disabled LP hash and NUMA awareness for this tournament, even so it still benefitted from the more efficient compile 4.5% speed gain over the development site version and suggests that it makes little difference.

I used the standard Houdini engine rather than the Pro version because after completing testing of the impact of NUMA awareness in ponder = on matches it was clearly evident that with just 6 cores of the 16, the engines that were NUMA aware showed no benefit and indeed actually interfered with the other enignes because with NUMA defaulting to CPU 0, the first move outof book meant the engine would take resources from its opponent and slow it down for that first move. With both asmFish and Houdini Pro being NUMA aware, Houdini Pro suffered worse with its kN/s varying between 3,500 to 10,500 kN/s when playing asmFish so the solution was to run with the NUMA aware parameter disabled.

Therefore here, the engines were very much on an equal footing with no benefit from LP hash or NUMA. The fact that asmFish did so well without LP hash and NUMA awareness confirms the pure code speed up looks to be significantly more beneficial than that achieved through a more efficient compile.

If I remove asmFish and Stockfish leaving just one Stockfish code representative, Brainfish's performance looks OK confirming the distortion caused by the inclusion with more than one engine from the same source code base.

Up Topic The Rybka Lounge / Computer Chess / 12 Engine, 2 round, round-robin 171019

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill