I developed some tools for evaluating GUI performance during engine-engine matches on Windows. In particular, the toolkit consists of two engines, (one WB, one UCI) printing precise timing information, and a command-line tool to extract these printed times out of a GUI log ('debug file'), and print a statistical summary of those.
So if you are curious as to how your favorite GUI is doing, feel free to use them. They can be downloaded from:
http://hgm.nubati.net/timingkit.zip
So if you are curious as to how your favorite GUI is doing, feel free to use them. They can be downloaded from:
http://hgm.nubati.net/timingkit.zip
Can you give some sample benchmarks? Was there a significant difference in performance between GUIs?
I only tried WinBoard and XBoard, to see how expnsive the display updates are (i.e. comparing -noGUI mode with normal operation), and to measure the impact of running UCI engines through an adapter (by comparing the timing of a WB engine with that of an UCI engine running through Polyglot or UCI2WB). I also compared that with a dedicated command-line tool, cutechess.cli. E.g.
Display updates, with WB engines:
WinBoard, no GUI: 1.3ms / 4ms
WinBoard, small board only: 13ms / 7ms
WinBoard, small board, arrow highlighting: 23ms / 6ms
WinBoard, big board, arrow, SAN engine-output: 20ms / 30ms
XBoard, no GUI: 1ms / 4ms
XBoard, no GUI, posting: 1ms / 6 ms
XBoard, small board only: 1ms / 60 ms
XBoard, big board, SAN output: 1ms / 90ms
No-GUI comparisons
cutechess-cli + WB (Linux): 1.0ms / 6ms
cutechess-cli + UCI (Linux): 1.25ms / 7ms
cutechess-cli + WB (Windows): 0.95ms / 3.1ms
cutechess-cli + UCI (Windows): 1.45ms / 3.1ms
XBoard + WB: 0.9ms / 7ms
XBoard + UCI2WB + UCI: 1.25ms / 7ms
Xboard + Polyglot + UCI: 2.0ms / 9ms
WinBoard + WB: 1.5ms / 4ms
WinBoard + UCI2WB + UCI: 2.1ms / 4.1ms
WinBoard + Polyglot + UCI: 3.2ms / 3.6ms
All on a 1.3GHz Celeron-M (running Ubuntu 10.04 or WinXP), for 40 moves / 10 sec games.
Display updates, with WB engines:
WinBoard, no GUI: 1.3ms / 4ms
WinBoard, small board only: 13ms / 7ms
WinBoard, small board, arrow highlighting: 23ms / 6ms
WinBoard, big board, arrow, SAN engine-output: 20ms / 30ms
XBoard, no GUI: 1ms / 4ms
XBoard, no GUI, posting: 1ms / 6 ms
XBoard, small board only: 1ms / 60 ms
XBoard, big board, SAN output: 1ms / 90ms
No-GUI comparisons
cutechess-cli + WB (Linux): 1.0ms / 6ms
cutechess-cli + UCI (Linux): 1.25ms / 7ms
cutechess-cli + WB (Windows): 0.95ms / 3.1ms
cutechess-cli + UCI (Windows): 1.45ms / 3.1ms
XBoard + WB: 0.9ms / 7ms
XBoard + UCI2WB + UCI: 1.25ms / 7ms
Xboard + Polyglot + UCI: 2.0ms / 9ms
WinBoard + WB: 1.5ms / 4ms
WinBoard + UCI2WB + UCI: 2.1ms / 4.1ms
WinBoard + Polyglot + UCI: 3.2ms / 3.6ms
All on a 1.3GHz Celeron-M (running Ubuntu 10.04 or WinXP), for 40 moves / 10 sec games.
So in the first two paragraphs of results, the first number is an engine running through the winboard format and the second is running through UCI? These are based on -noGUI at 0 ms right?
No, the first two paragraphs are all for running an engine using WB protocol (so Fairy-Max). The two numbers behind them are delay (wall-clock time elapsed between one engine sending its move, and the opponent receiving it), and the second number the average system load on the CPU while the engine was searching (presumably the load caused by the GUI, as I wasn't running anything else during the test).
The 'no GUI' cases do not imply 0 ms delay; there still is a client program that connects to the two engines, and is responsible for passing the moves between them, and perform all other functions usually performed by a GUI. Except that it doesn't display anything of what is going on on the screen. From the data you can see that the display does eat a fair amount of CPU time; the difference between Windows and Linux (i.e. WinBoard and XBoard) is that in Windows this displaying seem to have priority over running the engine, so that it increases the delay. In Linux the engine seems to be always immediately started (i.e. after 1 ms), and the displaying then competes with the engine search for CPU. For the same board size, displaying in Windows seems to take less total time than in Linux, however.
The 'no GUI' cases do not imply 0 ms delay; there still is a client program that connects to the two engines, and is responsible for passing the moves between them, and perform all other functions usually performed by a GUI. Except that it doesn't display anything of what is going on on the screen. From the data you can see that the display does eat a fair amount of CPU time; the difference between Windows and Linux (i.e. WinBoard and XBoard) is that in Windows this displaying seem to have priority over running the engine, so that it increases the delay. In Linux the engine seems to be always immediately started (i.e. after 1 ms), and the displaying then competes with the engine search for CPU. For the same board size, displaying in Windows seems to take less total time than in Linux, however.
Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill
