Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka 3 in 10 test suites
- - By Jouni (**) Date 2008-08-10 19:03
Here's comparison in 10 test suites (total time used):

             Rybka 2.3.2a        Rybka 3
    
tactical 1        27               17
           2        38               22
           3       103               34(!)
           4       110               46

endgame 1        34               35
               2        48               49
               3        42               32
               4       112              100
        
positional 1      31               28
              2     150              117

Great job Vasik!

Jouni
          
                   
Parent - - By Permanent Brain (*****) Date 2008-08-10 19:19
I guess the first (left) column is Rybka 3?
Parent - - By George Tsavdaris (****) Date 2008-08-10 19:20
No it's total time used and not how many positions it solved....
So smaller number is better.

But i would be interested in how many position each engine solved....
Parent - - By Permanent Brain (*****) Date 2008-08-10 19:34

> No it's total time used and not how many positions it solved....


That makes no sense, as a performance comparison. Number of solutions has priority.
Parent - - By Jouni (**) Date 2008-08-10 19:52
No! Imagine 30 positions suite and 10 minute limit. Engine A solves all positions in 9m59s, but engine B
29 positions in 1s but one is unsolved. Which one is better :-)

Jouni
Parent - By George Tsavdaris (****) Date 2008-08-10 19:58
Yes but i don't think that extreme thing happened here.
You should give the detailed results. Both positions that have been solved or not and the times for it.
Parent - By Permanent Brain (*****) Date 2008-08-10 21:01 Edited 2008-08-10 21:03
Solving time is important, but under reasonable conditions, the number of solutions has priority in the test result information. The solutions are the primary achievements, and to find them a fast as possible is the secondary aspect.

If time is secondary it does not mean it is totally useless, but it seems very strange to me, to give total times only as a test suite result, without any other information. You gave neither total number of positions, nor max. time per position, (nor hardware etc.), in your initial posting.

I guess you include the time for unsolved positions too, into your total times? It would be good but still not convincing. If you need to compress the result info, better would be: number of solutions, and average time for the solutions. Just two numbers instead of only one. Total time alone is just too intransparent IMO.

Having said that, I need to mention that Jouni is known in computer chess as someone who has found, collected, compiled and presented many very good test positions throughout the recent years. :-)
Parent - By Pia (****) Date 2008-08-10 22:13

>No! Imagine 30 positions suite and 10 minute limit. Engine A solves all positions in 9m59s, but engine B
>29 positions in 1s but one is unsolved. Which one is better :)


Engine A solved 29 positions in 1s and 30th position in 9m58s. Engine B in 10m the 30th position didn't solved. Engine A is better.
Parent - - By TedSummers (***) Date 2008-08-10 21:23
Here were the results of Marathin Test Suite. Not a very good test suite to judge by, as there are alot of alternate solutions for some positions.

Example Marathon #157
5rk1/p1p1br2/1p1p2qp/4p3/2PPn1pP/3Q2P1/PPN1P1K1/2RR2B1 b - - 0 1


The solution that is given is Bxh4!. However Rybka 3 gives Rf3!! a much higher eval. Anyway here were the results.

Solving: C:\Users\Ted Summers\Documents\ChessBase\Testsets\Marathon.cbh
Maximum time = 600s.

1. Tactics Marathon,  946 ECM   Solved in 0.69s/2; Solved: 1
2. Tactics Marathon,  949 ECM   Solved in 1.55s/7; Solved: 2
3. Tactics Marathon,  952 ECM   Solved in 2.87s/11; Solved: 3
4. Tactics Marathon,  961 ECM   Solved in 1.26s/5; Solved: 4
5. Tactics Marathon,  973 ECM   Solved in 2.32s/9; Solved: 5
6. Tactics Marathon,  977 ECM   Solved in 1.51s/6; Solved: 6
7. Tactics Marathon,  978 ECM   Solved in 1.44s/7; Solved: 7
8. Tactics Marathon,  986 ECM   Solved in 1.69s/8; Solved: 8
9. Tactics Marathon,  987 ECM   Solved in 2.14s/9; Solved: 9
10. Tactics Marathon,  989 ECM   > 600s.
11. Tactics Marathon,  993 ECM   Solved in 1.50s/7; Solved: 10
12. Tactics Marathon,  1011 ECM   Solved in 1.19s/5; Solved: 11
13. Tactics Marathon,  1448 ECM   Solved in 2.11s/10; Solved: 12
14. Tactics Marathon,  1007 ECM   Solved in 1.31s/6; Solved: 13
15. Tactics Marathon,  1009 ECM   > 600s.
16. Tactics Marathon,  1011 ECM   Solved in 1.26s/4; Solved: 14
17. Tactics Marathon,  1016 ECM   Solved in 1.58s/8; Solved: 15
18. Tactics Marathon,  1018 ECM   Solved in 1.22s/5; Solved: 16
19. Tactics Marathon,  1020 ECM   Solved in 1.86s/8; Solved: 17
20. Tactics Marathon,  1022 ECM   Solved in 1.62s/7; Solved: 18
21. Tactics Marathon,  1025 ECM   Solved in 1.56s/7; Solved: 19
22. Tactics Marathon,  1028 ECM   Solved in 1.25s/6; Solved: 20
23. Tactics Marathon,  1038 ECM   Solved in 1.09s/3; Solved: 21
24. Tactics Marathon,  1041 ECM   Solved in 1.08s/4; Solved: 22
25. Tactics Marathon,  1045 ECM   Solved in 1.37s/5; Solved: 23
26. Tactics Marathon,  1053 ECM   Solved in 1.50s/7; Solved: 24
27. Tactics Marathon,  1057 ECM   Solved in 1.73s/9; Solved: 25
28. Tactics Marathon,  1065 ECM   Solved in 1.65s/7; Solved: 26
29. Tactics Marathon,  1066 ECM   Solved in 1.42s/7; Solved: 27
30. Tactics Marathon,  1068 ECM   Solved in 7.10s/13; Solved: 28
31. Tactics Marathon,  1073 ECM   Solved in 3s/10; Solved: 29
32. Tactics Marathon,  1076 ECM   Solved in 1.22s/4; Solved: 30
33. Tactics Marathon,  1080 ECM   Solved in 3.53s/11; Solved: 31
34. Tactics Marathon,  1082 ECM   Solved in 1.33s/6; Solved: 32
35. Tactics Marathon,  1084 ECM   Solved in 12.06s/12; Solved: 33
36. Tactics Marathon,  1085 ECM   Solved in 1.44s/7; Solved: 34
37. Tactics Marathon,  1093 ECM   Solved in 1.84s/9; Solved: 35
38. Tactics Marathon,  1096 ECM   Solved in 2.45s/10; Solved: 36
39. Tactics Marathon,  1098 ECM   Solved in 1.59s/7; Solved: 37
40. Tactics Marathon,  1100 ECM   > 600s.
41. Tactics Marathon,  1114 ECM   Solved in 1.11s/3; Solved: 38
42. Tactics Marathon,  1122 ECM   Solved in 1.67s/6; Solved: 39
43. Tactics Marathon,  1124 ECM   Solved in 2.29s/9; Solved: 40
44. Tactics Marathon,  1125 ECM   Solved in 4.54s/12; Solved: 41
45. Tactics Marathon,  1127 ECM   Solved in 1.75s/8; Solved: 42
46. Tactics Marathon,  1128 ECM   Solved in 0.92s/2; Solved: 43
47. Tactics Marathon,  1131 ECM   Solved in 2.96s/11; Solved: 44
48. Tactics Marathon,  1132 ECM   Solved in 1.62s/7; Solved: 45
49. Tactics Marathon,  1133 ECM   Solved in 1.23s/6; Solved: 46
50. Tactics Marathon,  1134 ECM   Solved in 1.33s/6; Solved: 47
51. Tactics Marathon,  1147 ECM   Solved in 2.68s/10; Solved: 48
52. Tactics Marathon,  1148 ECM   Solved in 1.67s/7; Solved: 49
53. Tactics Marathon,  1151 ECM   Solved in 16.04s/13; Solved: 50
54. Tactics Marathon,  1152 ECM   Solved in 1.05s/3; Solved: 51
55. Tactics Marathon,  1160 ECM   Solved in 2.20s/9; Solved: 52
56. Tactics Marathon,  1161 ECM   Solved in 1.01s/3; Solved: 53
57. Tactics Marathon,  1169 ECM   > 600s.
58. Tactics Marathon,  1176 ECM   Solved in 1.06s/7; Solved: 54
59. Tactics Marathon,  1177 ECM   Solved in 1.53s/7; Solved: 55
60. Tactics Marathon,  1178 ECM   Solved in 1.44s/7; Solved: 56
61. Tactics Marathon,  1185 ECM   Solved in 1.51s/8; Solved: 57
62. Tactics Marathon,  1189 ECM   Solved in 1.45s/7; Solved: 58
63. Tactics Marathon,  1193 ECM   Solved in 1.53s/7; Solved: 59
64. Tactics Marathon,  1199 ECM   Solved in 1.37s/6; Solved: 60
65. Tactics Marathon,  1200 ECM   Solved in 1.59s/8; Solved: 61
66. Tactics Marathon,  1202 ECM   Solved in 1.51s/7; Solved: 62
67. Tactics Marathon,  1204 ECM   Solved in 1.48s/7; Solved: 63
68. Tactics Marathon,  1208 ECM   Solved in 1.34s/7; Solved: 64
69. Tactics Marathon,  1214 ECM   Solved in 1.48s/6; Solved: 65
70. Tactics Marathon,  1217 ECM   Solved in 1.65s/7; Solved: 66
71. Tactics Marathon,  1218 ECM   Solved in 1.89s/9; Solved: 67
72. Tactics Marathon,  1223 ECM   Solved in 1.47s/7; Solved: 68
73. Tactics Marathon,  1224 ECM   Solved in 1.90s/8; Solved: 69
74. Tactics Marathon,  1227 ECM   Solved in 49.39s/16; Solved: 70
75. Tactics Marathon,  1229 ECM   Solved in 4.32s/11; Solved: 71
76. Tactics Marathon,  1232 ECM   Solved in 1.06s/3; Solved: 72
77. Tactics Marathon,  1234 ECM   Solved in 1.72s/8; Solved: 73
78. Tactics Marathon,  1249 ECM   Solved in 1.22s/5; Solved: 74
79. Tactics Marathon,  1254 ECM   Solved in 2.25s/9; Solved: 75
80. Tactics Marathon,  1257 ECM   Solved in 1.67s/7; Solved: 76
81. Tactics Marathon,  1259 ECM   > 600s.
82. Tactics Marathon,  1262 ECM   Solved in 1.36s/6; Solved: 77
83. Tactics Marathon,  1265 ECM   Solved in 6.62s/11; Solved: 78
84. Tactics Marathon,  1267 ECM   Solved in 1.62s/6; Solved: 79
85. Tactics Marathon,  1269 ECM   Solved in 1.14s/4; Solved: 80
86. Tactics Marathon,  1270 ECM   Solved in 2.15s/10; Solved: 81
87. Tactics Marathon,  1271 ECM   Solved in 1.48s/7; Solved: 82
88. Tactics Marathon,  1273 ECM   Solved in 12.46s/17; Solved: 83
89. Tactics Marathon,  1274 ECM   Solved in 1.92s/9; Solved: 84
90. Tactics Marathon,  1277 ECM   Solved in 1.87s/7; Solved: 85
91. Tactics Marathon,  1278 ECM   Solved in 1.22s/4; Solved: 86
92. Tactics Marathon,  1280 ECM   Solved in 0.92s/2; Solved: 87
93. Tactics Marathon,  1281 ECM   Solved in 7.22s/13; Solved: 88
94. Tactics Marathon,  1285 ECM   Solved in 0.92s/2; Solved: 89
95. Tactics Marathon,  1293 ECM   Solved in 1.14s/3; Solved: 90
96. Tactics Marathon,  1295 ECM   Solved in 1.34s/6; Solved: 91
97. Tactics Marathon,  1306 ECM   Solved in 2.04s/10; Solved: 92
98. Tactics Marathon,  1308 ECM   Solved in 2.09s/9; Solved: 93
99. Tactics Marathon,  1314 ECM   Solved in 1.25s/6; Solved: 94
100. Tactics Marathon,  1317 ECM   Solved in 35.49s/17; Solved: 95
101. Tactics Marathon,  1318 ECM   Solved in 1.90s/9; Solved: 96
102. Tactics Marathon,  1321 ECM   Solved in 0.95s/2; Solved: 97
103. Tactics Marathon,  1325 ECM   Solved in 3.17s/11; Solved: 98
104. Tactics Marathon,  1327 ECM   Solved in 1.44s/7; Solved: 99
105. Tactics Marathon,  1329 ECM   Solved in 1.12s/4; Solved: 100
106. Tactics Marathon,  1332 ECM   Solved in 1.73s/7; Solved: 101
107. Tactics Marathon,  1335 ECM   Solved in 1.76s/8; Solved: 102
108. Tactics Marathon,  1336 ECM   Solved in 1.64s/8; Solved: 103
109. Tactics Marathon,  1339 ECM   Solved in 1.39s/6; Solved: 104
110. Tactics Marathon,  1340 ECM   Solved in 1.47s/7; Solved: 105
111. Tactics Marathon,  1341 ECM   Solved in 1.90s/8; Solved: 106
112. Tactics Marathon,  1344 ECM   Solved in 2.95s/11; Solved: 107
113. Tactics Marathon,  1346 ECM   Solved in 1.51s/7; Solved: 108
114. Tactics Marathon,  1347 ECM   Solved in 1.12s/3; Solved: 109
115. Tactics Marathon,  1348 ECM   Solved in 9.20s/9; Solved: 110
116. Tactics Marathon,  1350 ECM   Solved in 2.95s/9; Solved: 111
117. Tactics Marathon,  1352 ECM   Solved in 1.12s/4; Solved: 112
118. Tactics Marathon,  1354 ECM   Solved in 1.98s/8; Solved: 113
119. Tactics Marathon,  1355 ECM   Solved in 1.26s/6; Solved: 114
120. Tactics Marathon,  1357 ECM   Solved in 1.61s/7; Solved: 115
121. Tactics Marathon,  1358 ECM   > 600s.
122. Tactics Marathon,  1360 ECM   Solved in 4.31s/12; Solved: 116
123. Tactics Marathon,  1371 ECM   Solved in 1.53s/7; Solved: 117
124. Tactics Marathon,  1376 ECM   Solved in 1.58s/8; Solved: 118
125. Tactics Marathon,  1377 ECM   Solved in 4.87s/12; Solved: 119
126. Tactics Marathon,  1380 ECM   Solved in 2.42s/11; Solved: 120
127. Tactics Marathon,  1381 ECM   Solved in 2.68s/11; Solved: 121
128. Tactics Marathon,  1383 ECM   Solved in 1.64s/7; Solved: 122
129. Tactics Marathon,  1387 ECM   Solved in 1.23s/5; Solved: 123
130. Tactics Marathon,  1394 ECM   Solved in 1.55s/7; Solved: 124
131. Tactics Marathon,  1397 ECM   Solved in 1.40s/6; Solved: 125
132. Tactics Marathon,  1398 ECM   Solved in 1.50s/6; Solved: 126
133. Tactics Marathon,  1403 ECM   Solved in 4.14s/11; Solved: 127
134. Tactics Marathon,  1405 ECM   Solved in 1.33s/6; Solved: 128
135. Tactics Marathon,  1408 ECM   Solved in 2.43s/9; Solved: 129
136. Tactics Marathon,  1409 ECM   Solved in 1.08s/3; Solved: 130
137. Tactics Marathon,  1410 ECM   Solved in 1.34s/6; Solved: 131
138. Tactics Marathon,  1412 ECM   Solved in 1.51s/7; Solved: 132
139. Tactics Marathon,  1413 ECM   Solved in 1.61s/8; Solved: 133
140. Tactics Marathon,  1420 ECM   Solved in 1.28s/5; Solved: 134
141. Tactics Marathon,  1421 ECM   Solved in 1.14s/4; Solved: 135
142. Tactics Marathon,  1423 ECM   Solved in 3.11s/11; Solved: 136
143. Tactics Marathon,  1426 ECM   Solved in 1.48s/7; Solved: 137
144. Tactics Marathon,  1432 ECM   Solved in 36.69s/17; Solved: 138
145. Tactics Marathon,  1433 ECM   Solved in 1.83s/7; Solved: 139
146. Tactics Marathon,  1434 ECM   Solved in 0.87s/2; Solved: 140
147. Tactics Marathon,  1439 ECM   Solved in 1.76s/7; Solved: 141
148. Tactics Marathon,  1448 ECM   Solved in 2.09s/9; Solved: 142
149. Tactics Marathon,  1450 ECM   Solved in 1.15s/4; Solved: 143
150. Tactics Marathon,  1452 ECM   Solved in 1.53s/8; Solved: 144
151. Tactics Marathon,  1454 ECM   Solved in 1.62s/7; Solved: 145
152. Tactics Marathon,  1457 ECM   Solved in 1.62s/7; Solved: 146
153. Tactics Marathon,  1458 ECM   Solved in 1.26s/6; Solved: 147
154. Tactics Marathon,  1478 ECM   Solved in 1.20s/5; Solved: 148
155. Tactics Marathon,  1482 ECM   Solved in 1.40s/9; Solved: 149
156. Tactics Marathon,  1488 ECM   Solved in 0.95s/8; Solved: 150
157. Tactics Marathon,  1495 ECM   > 600s.
158. Tactics Marathon,  1503 ECM   Solved in 1.36s/6; Solved: 151
159. Tactics Marathon,  1504 ECM   Solved in 1.47s/7; Solved: 152
160. Tactics Marathon,  1507 ECM   Solved in 1.73s/11; Solved: 153
161. Tactics Marathon,  1516 ECM   Solved in 7.04s/13; Solved: 154
162. Tactics Marathon,  1518 ECM   > 600s.
163. Tactics Marathon,  1525 ECM   Solved in 1.67s/8; Solved: 155
164. Tactics Marathon,  1527 ECM   > 600s.
165. Tactics Marathon,  1529 ECM   Solved in 1.08s/3; Solved: 156
166. Tactics Marathon,  1533 ECM   Solved in 2.78s/11; Solved: 157
167. Tactics Marathon,  1536 ECM   Solved in 1.08s/7; Solved: 158
168. Tactics Marathon,  1539 ECM   > 600s.
169. Tactics Marathon,  1543 ECM   Solved in 132.13s/24; Solved: 159
170. Tactics Marathon,  1545 ECM   > 600s.
171. Tactics Marathon,  1546 ECM   Solved in 2.23s/12; Solved: 160
172. Tactics Marathon,  1547 ECM   Solved in 1.78s/9; Solved: 161
173. Tactics Marathon,  1548 ECM   > 600s.
174. Tactics Marathon,  1551 ECM   Solved in 2.98s/11; Solved: 162
175. Tactics Marathon,  1553 ECM   Solved in 1.83s/9; Solved: 163
176. Tactics Marathon,  1554 ECM   Solved in 2.42s/10; Solved: 164
177. Tactics Marathon,  1561 ECM   Solved in 2.15s/10; Solved: 165
178. Tactics Marathon,  1563 ECM   Solved in 1.58s/7; Solved: 166
179. Tactics Marathon,  1564 ECM   Solved in 1.55s/7; Solved: 167
180. Tactics Marathon,  1567 ECM   Solved in 1.51s/13; Solved: 168
181. Tactics Marathon,  1573 ECM   Solved in 1.69s/9; Solved: 169
182. Tactics Marathon,  1574 ECM   Solved in 1.95s/10; Solved: 170
183. Tactics Marathon,  1578 ECM   Solved in 5.52s/15; Solved: 171
184. Tactics Marathon,  1579 ECM   Solved in 1.39s/6; Solved: 172
185. Tactics Marathon,  1581 ECM   Solved in 0.98s/8; Solved: 173
186. Tactics Marathon,  1582 ECM   Solved in 2.14s/10; Solved: 174
187. Tactics Marathon,  1584 ECM   Solved in 1.23s/7; Solved: 175
188. Tactics Marathon,  1590 ECM   Solved in 1.54s/7; Solved: 176
189. Tactics Marathon,  1592 ECM   Solved in 3.03s/11; Solved: 177
190. Tactics Marathon,  1594 ECM   Solved in 1.20s/3; Solved: 178
191. Tactics Marathon,  1596 ECM   Solved in 0.95s/2; Solved: 179
192. Tactics Marathon,  1597 ECM   Solved in 1.61s/8; Solved: 180
193. Tactics Marathon,  1600 ECM   Solved in 1.28s/6; Solved: 181
194. Tactics Marathon,  1601 ECM   Solved in 1.93s/9; Solved: 182
195. Tactics Marathon,  1602 ECM   Solved in 1.16s/4; Solved: 183
196. Tactics Marathon,  1604 ECM   Solved in 1.53s/7; Solved: 184
197. Tactics Marathon,  1605 ECM   Solved in 1.45s/7; Solved: 185
198. Tactics Marathon,  1606 ECM   Solved in 1.67s/7; Solved: 186
199. Tactics Marathon,  1607 ECM   Solved in 1.26s/6; Solved: 187
200. Tactics Marathon,  1610 ECM   Solved in 1.26s/5; Solved: 188
201. Tactics Marathon,  1612 ECM   Solved in 29.48s/14; Solved: 189
202. Tactics Marathon,  1613 ECM   Solved in 13.64s/13; Solved: 190
203. Tactics Marathon,  1614 ECM   Solved in 6.62s/12; Solved: 191
204. Tactics Marathon,  1617 ECM   Solved in 20.83s/14; Solved: 192
205. Tactics Marathon,  1618 ECM   Solved in 1.70s/9; Solved: 193
206. Tactics Marathon,  1620 ECM   Solved in 1.06s/3; Solved: 194
207. Tactics Marathon,  1621 ECM   Solved in 18.41s/14; Solved: 195
208. Tactics Marathon,  1622 ECM   Solved in 68.36s/18; Solved: 196
209. Tactics Marathon,  1623 ECM   Solved in 1.72s/8; Solved: 197
210. Tactics Marathon,  Shirov   Solved in 1.81s/9; Solved: 198

Result: 198 out of 210 = 94.2%. Average time = 4.03s / 7.83
Parent - By Pia (****) Date 2008-08-10 22:20

>there are alot of alternate solutions for some positions.


It can be fixed. You can define alternatives as variants.
Up Topic Rybka Support & Discussion / Rybka Discussion / Rybka 3 in 10 test suites

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill