Not logged inRybka Chess Community Forum
Up Topic The Rybka Lounge / Computer Chess / So you think you know how your engine works?
- - By Peter Grayson (****) [gb] Date 2017-07-19 11:59
In the days of the single core "old school engines" hash impact on position solving was very predictable with changes making small differences to solving times but usually a particular hash value gave a marginally best score. All of that changed with Rybka and the trend continues when a change to hash value can have a significant impact on an engine's ability to identify key moves sometimes making the engine characteristic look completely different when compared with other hash values. The trend continues with today's engines.

  Too hot here to run matches and even the tests I carried out yesterday suggested some heat related CPU throttling with just one core active for the tests. Despite that, tests to look at hash impact on position solving gave interesting results especially the repeat test of the best performing hash value that highlighted the Komodo 11.2 engine is certainly not deterministic when run in single core mode.

  The suspicion of thermal related speed throttling of the CPU was indicated by the two other engines giving a predictable result albeit slightly slower on the second best hash setting run. I am confident that confirms Stockfish and the Stockfish based SugaR XPrO engine are deterministic but raises the question as to what influence the noise of single core variability within the Komodo engine has during the course of a game, particularly when additional cores and thus MP variability is also introduced? Could its performance actually be improved by making the engine deterministic? It seems to me that it makes it more difficult and would require much more extensive testing to prove Komodo changes are effective and having the desired impact. The second test run at 256Mb hash looks almost unrecognisable as the same engine.

  That digression aside, the main purpose of the test was to look at hash impact. Despite being deterministic, the Stockfish and SugaR Xpro engines confirm changing hash can change the engine's characteristics almost to the point of being unrecognisable. The question also arises then as to what impact does this have on game results when certain hash settings may find key moves missed with other hash settings. There is also the question then of how to set the optimised hash setting for an engine's hardware allocation and time control. Is the generally accepted method of no more than 50% hash fill rate for three times the average move time actually the best way to optimise hash? The only way to obtain some indication of this is to run a series of comparative games but until the weather cools somewhat, matches are off the agenda!

  A fixed set of 14 positions were used with a 60 second time allocation. There were two anomalies thrown up by the GUI when both Komodo and Stockfish were allowed to slightly overrun the permitted time in the yellow highlighted recordings. If they were caused by the engine, it could indicate a potential to miss a time control and thus lose on time under match conditions.

  Position 10 threw up something very interesting. I had always assumed Stockfish missed the correct Kg3 move because of strong King Protection direction that made it reluctant to bring the King into the battle while there were still major pieces on the board. Surely hash cannot influence that so it would seem it was a case of it just missing the move.
Position 11 also indicates a rapid cut-off when major piece sacrifices are required.

  The engine and GUI were unloaded after each test run to ensure no hash retention. Because SugaR XPrO used LP hash, after rebooting the machine, the engine was tested with reducing hash to save having to keep rebooting for the higher hash values avoiding RAM fragmentation.

PeterG

Engines: Komodo 11.2 64-bit, Stockfish 150717 64 bit, SugaR XPrO 1.0e 64 bit.
Time 60 seconds
Hardware/Software: CPU=Q9550 O/C's 3.53GHz with 8Gb RAM. Windows 7 64 Pro, Fritz GUI 13. SugaR XPr0 1.0e used LP hash.
Position scores shown as time in seconds/ply depth t(s)/pd
Attachment: Komodo11.2Tets.pgn.7z - The positions with scores (3k)
Parent - - By shrapnel (***) [in] Date 2017-07-19 16:06
The only way to obtain some indication of this is to run a series of comparative games but until the weather cools somewhat, matches are off the agenda!

No ACs in England ??
Parent - - By Peter Grayson (****) [gb] Date 2017-07-19 16:14 Upvotes 1

> No ACs in England ??


It isn't worth it for 2 to 3 months of the year. Of course I could run the matches in my car!:lol:
Parent - By shrapnel (***) [in] Date 2017-07-20 03:45 Edited 2017-07-20 03:50
:lol: right !
You're lucky ; here in India in the North, its the other way round. We barely get 2 months of decent Winter.
So no question of running a PC without AC.
   In fact, I think we're quite spoilt now and can't think of Life without AC !
AC at Home, in the Car and in the Office !
Parent - - By Geomusic (*****) Date 2017-07-20 04:37 Edited 2017-07-20 04:42
Funny my S7 cellphone can outcalculate my old laptop from 2009 which was about 1/2 as fast as my top of the line Q6600 from 2007. I bought a new laptop about 6 months ago thats about 5.2x faster than the S7 chess engine wise. Sorry this has nothing to do with the topic except that we're both talking about computer chess. LOL
Parent - - By Peter Grayson (****) [gb] Date 2017-07-20 10:21

> Funny my S7 cellphone can outcalculate my old laptop from 2009 which was about 1/2 as fast as my top of the line Q6600 from 2007. I bought a new laptop about 6 months ago thats about 5.2x faster than the S7 chess engine wise.


Such are the advances in technology.

The brief test posted here indicates that changing hash values can change the engine's ability to select key moves when it seems the more sophisticated positions cause more difficulty. Certainly relevant for position solving and perhaps analysis when giving more hash may only be beneficial for certain values of hash. Similar tests I carried out in the Rybka era revealed the characteristics of hash size related to position solving stayed with the engine rather than the PC hardware. It seems the characteristic remains with Stockfish but unclear what is happening with Komodo. It would be interesting to know if the Stockfish characteristics for single thread was maintained on the cellphone.

GUI's can change the way an engine performs when for example a GUI supporting sampled search can produce a different characteristic than one that does not for engines that can provide sampled search.

It comes down to; is the user set up getting the best out of the engine?

PeterG
Parent - - By Akbarfan (***) Date 2017-09-25 11:00 Edited 2017-09-25 12:04
Löse: ...\K11.2.cbh

Houdini 6 x64-pext-512MB/ i76700/ 4cores/ 3.4GHz
Maximale Lösungszeit = 60s.

1. #1,  RebTest   1.45s/17; Gelöst: 1
2. #2,  RebTest   > 60s.
3. #3,  RebTest   0.61s/15;   solved: 2
4. #4,  RebTest   0.14s/12;   solved: 3
5. #5,  RebTest   9.94s/19;   solved: 4
6. #6,  RebTest   0.66s/15;   solved: 5
7. #7,  RebTest   0.11s/11;   solved: 6
8. #8,  Richter   21.28s/24;  solved: 7
9. #9,  Richter     2.78s/18;  solved: 8
10. ,  BS2830    > 60s.
11. ,  BT2630    > 60s.
12. arasan18.44   14.21s/19;  solved: 9
13. arasan18.51     6.93s/21;  solved: 10
14. arasan18.62     > 60s.

Ergebnis: 10 aus 14 = 71.4%. Durchschnittszeit = 5.81s / 17.10
Parent - By Peter Grayson (****) [gb] Date 2017-09-25 14:00
For comparison, just ran my overclocked 9 year old quad machine through the testsuite with Houdini 6 Pro LP Hash enabled in Windows 7 Pro ...

Solving: C:\Users\Peter\Documents\ChessBase\Bases\Komodo 11.2.2 test.cbh
Maximum time = 60s.

Houdini 6 Pro 64, 512 Mb LP hash
Q9550 at 3.53GHz, 4 cores, 8Gb RAM
5 man Syzygy bases

1. ,  RebTest   Solved in 0.31s/14; Solved: 1
2. ,  RebTest   Solved in 41.84s/30; Solved: 2
3. ,  RebTest   Solved in 0.31s/14; Solved: 3
4. ,  RebTest   Solved in 0.22s/12; Solved: 4
5. ,  RebTest   Solved in 12.08s/19; Solved: 5
6. ,  RebTest   Solved in 0.98s/15; Solved: 6
7. ,  RebTest   > 60s.
8. ,  Richter 2017   > 60s.
9. ,  Richter   Solved in 3.39s/18; Solved: 7
10. ,  bs2830   Solved in 4.23s/20; Solved: 8
11. ,  BT2630   Solved in 33.57s/26; Solved: 9
12. arasan18.44,  Arasan 18   Solved in 1.25s/13; Solved: 10
13. arasan18.51,  Arasan 18   Solved in 3.23s/17; Solved: 11
14. arasan18.62,  Arasan 18   Solved in 29.17s/23; Solved: 12

Result: 12 out of 14 = 85.7%. Average time = 10.88s / 18.41

PeterG
Parent - By Peter Grayson (****) [gb] Date 2017-09-27 21:56
An important fact to bear in mind is because of multi-threaded variability then each run will likely give different timings and possibly different solved positions. For cosistency, best to test with freshly loaded engine, cleared hash and single thread,

PeterG
Up Topic The Rybka Lounge / Computer Chess / So you think you know how your engine works?

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill