Not logged inRybka Chess Community Forum
Up Topic The Rybka Lounge / Computer Chess / GUI evaluation toolkit
- - By h.g.muller (***) [nl] Date 2012-02-19 22:09
I developed some tools for evaluating GUI performance during engine-engine matches on Windows. In particular, the toolkit consists of two engines, (one WB, one UCI) printing precise timing information, and a command-line tool to extract these printed times out of a GUI log ('debug file'), and print a statistical summary of those.

So if you are curious as to how your favorite GUI is doing, feel free to use them. They can be downloaded from:

http://hgm.nubati.net/timingkit.zip
Parent - - By Labyrinth (****) [us] Date 2012-02-20 02:43
Can you give some sample benchmarks? Was there a significant difference in performance between GUIs?
Parent - - By h.g.muller (***) [nl] Date 2012-02-20 13:19
I only tried WinBoard and XBoard, to see how expnsive the display updates are (i.e. comparing -noGUI mode with normal operation), and to measure the impact of running UCI engines through an adapter (by comparing the timing of a WB engine with that of an UCI engine running through Polyglot or UCI2WB). I also compared that with a dedicated command-line tool, cutechess.cli. E.g.

Display updates, with WB engines:

WinBoard, no GUI: 1.3ms / 4ms
WinBoard, small board only: 13ms / 7ms
WinBoard, small board, arrow highlighting: 23ms / 6ms
WinBoard, big board, arrow, SAN engine-output: 20ms / 30ms

XBoard, no GUI: 1ms / 4ms
XBoard, no GUI, posting: 1ms / 6 ms
XBoard, small board only: 1ms / 60 ms
XBoard, big board, SAN output: 1ms / 90ms

No-GUI comparisons

cutechess-cli + WB (Linux): 1.0ms / 6ms
cutechess-cli + UCI (Linux): 1.25ms / 7ms
cutechess-cli + WB (Windows): 0.95ms / 3.1ms
cutechess-cli + UCI (Windows): 1.45ms / 3.1ms
XBoard + WB: 0.9ms / 7ms
XBoard + UCI2WB + UCI: 1.25ms / 7ms
Xboard + Polyglot + UCI: 2.0ms / 9ms
WinBoard + WB: 1.5ms / 4ms
WinBoard + UCI2WB + UCI: 2.1ms / 4.1ms
WinBoard + Polyglot + UCI: 3.2ms / 3.6ms

All on a 1.3GHz Celeron-M (running Ubuntu 10.04 or WinXP), for 40 moves / 10 sec games.
Parent - - By Labyrinth (****) [us] Date 2012-02-21 07:00
So in the first two paragraphs of results, the first number is an engine running through the winboard format and the second is running through UCI? These are based on -noGUI at 0 ms right?
Parent - By h.g.muller (***) [nl] Date 2012-02-21 09:15 Edited 2012-02-21 10:43
No, the first two paragraphs are all for running an engine using WB protocol (so Fairy-Max). The two numbers behind them are delay (wall-clock time elapsed between one engine sending its move, and the opponent receiving it), and the second number the average system load on the CPU while the engine was searching (presumably the load caused by the GUI, as I wasn't running anything else during the test).

The 'no GUI' cases do not imply 0 ms delay; there still is a client program that connects to the two engines, and is responsible for passing the moves between them, and perform all other functions usually performed by a GUI. Except that it doesn't display anything of what is going on on the screen. From the data you can see that the display does eat a fair amount of CPU time; the difference between Windows and Linux (i.e. WinBoard and XBoard) is that in Windows this displaying seem to have priority over running the engine, so that it increases the delay. In Linux the engine seems to be always immediately started (i.e. after 1 ms), and the displaying then competes with the engine search for CPU. For the same board size, displaying in Windows seems to take less total time than in Linux, however.
Up Topic The Rybka Lounge / Computer Chess / GUI evaluation toolkit

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill