Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Chessultant - UCI chess-cluster-engine
- - By SchachProfi (****) Date 2010-04-16 00:23 Edited 2010-04-16 23:42
Dear all,

i have seen a few people in the "Rybka 4 Update"-Thread (may)being interested
into my personal chess-cluster-software which i have developed over the last
few years for my own purposes. It didn't even had a name till today (used to
be just chess[.exe]), so i decided to name it "Chessultant" starting today.

In this thread i want to show you the current capabilities and limitations of
Chesssultent, want to evaluate whether there is a bigger interest in a public
release of a freely available version of it (which would justify the further
work needed) and also want to collect any ideas & suggestions you might have.

Chesssultent is a chess-cluster-software which is using currently & publicly
available single-PC x86-chess-engines to evaluate a chess position on a (in
best case symmetric) hardware-cluster. Chesssultent is capable of using any
UCI-conform engine for its analysis and is itself also a UCI-confirm engine
coming with its own little GUI to control the calculation and providing in-
depth analysis & logging of the current postion. It's written in plain C/C++
(the GUI being based on the Qt-library), being carefully handoptimized and
can be configured with an options-file and/or parameters and can run under
the Micro$oft Windows Family or any Linux-distribution using a KDE-frontend.
It's currently based on & optimized to my hardware-cluster (see below for
detailed specifications) consisting of a single master (M) and 16 slaves (S).

Chesssultent can provide several modes of calculation:

- Width: Two phases:
o First Phase: (M) runs a master-engine with Multi-PV-analysis,
the best 16 moves are moved for further Multi-PV-analysis on the (S).
If a new entry reaches Top16 on (M) and holds there for a threshold
(adjustable depth/time), the (S) analysing the weakest move is stopped
and the new Top16-entry is moved to this (S). The (M) can also run another
engine in parallel on limited cores to solve tactical/win-finding problems.
o Second Phase: out of this +16x16-move-matrix following possibilities arise:
a) the overall best 16 moves are choosen for further analysis on (S)
b) the 4 best B-answers to any of the 4 best A-moves are choosen for
    for further analysis on (S)
c) the best 8 B-answers to the best A-move, the best 4 B-answers to
    the second best A-move and the 2 best answers to the third and forth
    best (or one forth/fifth) A-moves are choosen for further analysis on (S)
d) there is already a little intelligence implemented which does categorize
    the position tactical/positional, open/closed and winning/drawish/losing,
    considering available time and the number of moves played, and can choose
    out of a)-c) by itself and maybe also build a more intelligent custom
    search-pattern in future (but this topic is a whole Ph.D. by itself :-)).
The master does analyze the non-Top16 of the first Phase meanwhile and
does move upmovers to a (S) for further analysis. It can also analyze the
current best move off all (S) or continue its calculation of the first Phase.
Finally (M) takes the best move which reached a reasonable depth in analysis,
calculates on while waiting for black's move and then starts the next iteration.

- Depth: Again (M) runs a Multi-PV-analysis. Optionally adjustable,
(M) runs further four engines to analyze the best 4 A-moves out of
this analysis and moves the best 4x4 AxB-moves to the (S) for further analysis.
It's also possible that 4 (S) run the B-move-analysis instead of (M),
then the best 4x3 moves are moved to the (S) or any customized number.
Also the same intelligence as described under "-Width" above can be used.
Finally (M) takes the best move which reached a reasonable depth in analysis.

- Quick: Again (M) runs a Multi-PV-analysis, 8 (S) calculate the best B-moves to
this A-moves, the best 4 (or a custom pattern) are analyzed in parallel on 4 (S),
and these best C-moves are analyzed on 2 (S) and the best D-moves on another 2 (S).
Finally (M) takes the best move which reached a reasonable depth in analysis.
- Aggressive: Same as Quick, takes two moves at once and therefore doubles depth
for sake of width.

- Engine-Group: Again (M) runs a Multi-PV-analysis.
o First Phase: the 8/4/2 best moves are moved to a (S)-group consisting of 2/4/8
strong engines (which have nearly equally strength but a different character and
reach a similiar analysis-depth a time in best case) for further analysis.
o Second Phase: There are two different possibilites now:
a) The leading engine calculates the analysis of the created 16-moves-matrix
out of the first Phase or an adjustable custom pattern on the (S).
b) The groups analyze the best 4 moves out of the 16-moves-matrix each.
Finally (M) takes the best move which reached a reasonable depth in analysis,
in case of b) it may choose to most favored move or a custom pattern.

- Nalimov: In a Nalimov(currently: 6)-position only (M) runs.

Chesssultent is capable of saving&using persistent hash-files, providing the (S)
with the best-suited one out of the 16x16x16-matrix it has on file, which
consums pretty exactly a terabyte for each 256 MB hash-size on harddisk.
In case available time is not infinite as in UCI analysis-mode, my time-per-
move algorithms are based on a degressive schedule involing the balance of
the current position, it spends most time in early moves and drawn positions.

Chesssultent as of today has the following known limitations:
- It is a optimized for a symmetrical architecture, and while it would be
pretty easy to scale it upwards (i'm not doing this soon, my electricy-
bill has already reached 10k ;-)), downwards will be a more difficult task.
I would even say that the number of (S) should be a magnitude of 2 to
make things easier, another number and/or unsymmetrical architectures
would need far more balancing and intelligence in the analysis of results.
- It is best suited to analyzing. It can play games with long time-frames,
it must be further optimized for blitz/rapid and there isn't to much sense
of using it for (kiddy-)bullet, my interest is in correct chess anyways.
Anyways there is room for other (experimental) time-per-move algorithms.
- The intelligence of creating (S)-patterns depending on the current
position for sure has a Ph.-D.-room to improve.
- It needs a huge network-bandwith, especially for the persistant hash-files.
Maybe i will implement some kind of compression there.
- When to stop a (S) for another calculation is also a science for itself.
- The GUI suites my needs, it may not suite yours. But you can use it with
your favorite GUI which can show the best (PV-)lines, everything else is
in the logs and you may write scripts to handle that for further analysis.
- Experience with the modes and all the parameters is of course limited,
there may be better ones and a lot room for further improvement.
- Thousands other limitations that i can't think of at the moment.

Anyways, it plays pretty good chess and beats any single engine mostly with ease.
It already has analyzed several openings to win against itself, but as you know,
this often only holds until it is refuted by a bigger approach - that is nature.

My cluster currently consists of the following components:
- 1x Master: (Brandnew) 2x Intel Xeon DP X5680 (12 Cores @ 4,0GHz) running on a
(brandnew&lovely) EVGA Classified SR-2 with Intel Server NIC and 24GB Ram, the
most beautiful piece of (affordable) x86-workstation/server-hardware ever seen.
- 16x Slaves: (Mature) Intel Xeon UP W3520 (4 Cores @ 4,0GHz) running on a
Asus P6T Deluxe V2 with Intel NIC and 6GB Ram
- 1x Switch: (Young) Cisco Catalyst 2960G-24TC

Regarding my person:
I'm a self-employed participant of financial markets interested in chess-analysis,
especially analysing interesting positions and openings, developing opening-systems
while refuting others and finding the "perfect game". I'm currently providing my
analysis to my favorite two GM's and one IM, and myself having only rarely interest
and time for over_the_board-play and even less for point-hunting on chess-servers.

Please share what you think about Chesssultent with me, your ideas & suggestions
to further improve it and bring it to a useable public release, and also what kind
of cluster-specifications you have at your hands and what i may can do for you.

Thanks for all your time,

Alex
Parent - By BrandonSi (***) Date 2010-04-16 01:07
Wow! Very impressive Alex, I look forward to checking it out!!
Parent - - By Androcles (**) Date 2010-04-16 01:58 Edited 2010-04-16 02:07
Impressive. As you are seemingly a successful quant, besides chess,
do you use this cluster for financial data acquisition and consolidation
(or options valuation)?
I mean, now with the bandwidth available to ordinary people connected
to the Internet there is a lot of possibilities for creative individuals
not bound to the formal environment of a bank's trading floor.

When the investing litterature is all too often still based on lame OHLCV
daily data as input of technical analysis indicators, it happens that even
an amateur trader operating from home can have tick data with bid/ask size,
individual transaction timestamp and volume.
Besides there are a myriad of other potential data types to be aggregated
e.g. from text mining from financial sources.

If you are willing to share some ideas on how to leverage on all those
new data sources suddenly made available even to average Joe
and how to fuse them for the creation of new TA indicators, please send me a PM.

Cheers
Parent - - By M ANSARI (*****) Date 2010-04-20 06:28
Where were you all this time.  If you had posted this about 2 years ago I would have offered you good cash for the effort.  Even before Vas started his cluster project I had ideas to get something going but could not get Vas to spend time on it until of course he went his own way with his cluster project.  Today I find I spend far more time on work and other hobbies that I would not think it is worth the effort.  You can probably search this forum and find some good information regarding clustering on older posts where Vas also gave some inputs.

From what I learned during the brief period I worked on clustering is this ... if you split the nodes on the Master it is best to split to about 4 or 5 moves.  More than that and you reach a point of diminishing returns.  Another thing is that all computers should be running the same engine (at least until a decent algo can sparse different evaluation results from different engines).  One more thing is that all hardware should be identical, otherwise you get one hardware that is looking deeper and thus evaluating differently.  At least this is the setup you should go for to start, and of course you can become more sophisticated later and add stuff ... but it is always better to start simple.

You can also improve things by designating one slave computer as the EGTB probing computer.  This way you can remove all EGTB probing code on all computers (except the designated EG computer) which will give you better performance while still having coverage of all EGTB's.  The idea is to keep the Master running totally passively and to only have an override when a certain evaluation trigger difference is reached.  So let us say the Master sends 5 moves to look at, then the 5 slaves will run single PV on these 5 nodes.  These 5 nodes will be obviously much deeper than the Master nodes and then before the Master issues a move generation signal it checks the slaves and compares evaluation, and if a trigger is reached it will change its move choice.

This system is especially good with Rybka 3 because R3 has an incredibly good nose for the correct move.  In MPV mode where 5 moves are to be chosen, it has by far the best nose for choosing the relevant moves.  This dramatically focuses search processing power in an extremely effective manner.  I expect this to be further improved with R4, and the good news is that it seems that R4 will have the ability to tailor your search.  If you can tailor your search for aggressive pruning while still keeping the same evaluation, that also dramatically improves the flexibility of how you configure your slaves and I would guess would make the clustering even more effective.

Good luck and wish you luck in your project!
Parent - By SchachProfi (****) Date 2010-04-21 00:01
Hello M ANSARI,

i'm already aware about the real big clusters in the (oil$-)middle east, wonder why they most often play against themself and not in international competitions, but i guess the little clusters in the rest of the world can't be regarded as competition ;-).

- Yes, i like symmetry (having identical 16 (S)s) and know of its importance
- Yes, i implicitely split to 4 or 5 main-moves in the width-mode, the 16 slave-moves include different answers to this up to ~5 main-moves
- Yes, the main modes run the same engine, the "engine-mode" is just at a experimentell stage, but i had pretty good results already when combining Rybka3 with its Dynamic&Human-peers.
- Yes, its also my impression that Rybka3 has a good nose, but often the best move is not not the best move in long-term, i'm also looking for Rybka4
- Thanks for your valuable input about (M) & EGTB, will dig further into your thinkings
- Will search your old posts, thanks for pointing me out
- Still interested in improving this cluster-project? ;-)

Alex
Parent - - By mikal (**) Date 2010-04-16 05:49
Proof or download link please or it didn't happen.

And by the way, for analysis Aquarium "idea" can utilize multiple computers or nodes using multiple engines if you can call that cluster that is so clustered analysis isn't really that new. If you plan a playing version you have to break some elo barriers, a 3000 elo 16 node cluster simply won't be that revolutionary.
Parent - By Androcles (**) Date 2010-04-16 13:30
Why to be so negative? What he is trying to do is to dynamically reschedule the search
priorities given results made known so far among other engines.
This is sort of a metaengine, very useful IMHO for deep analysis.

Of course we could imagine other ways to achieve this with a larger
number of (weaker) nodes and share the results from a peer to peer
overarching application through the Internet.
Has such an approach already been attempted?
Parent - - By Geomusic (*****) Date 2010-04-16 14:29
16 node, 3000 elo cluster? What hardware are you using LMAO. I can achieve this with a single core budget laptop.
Parent - By BrandonSi (***) Date 2010-04-16 16:28
Ignore the detractors Alex. Us who understand the implications of this are still interested in your utility. :)
Parent - - By wajit (**) Date 2010-04-16 23:10
I look forward to seeing how this thing really runs. I'll keep an eye out for your link to the software.  Thanks for the info.
Parent - By dragon49 (****) Date 2010-04-17 03:32
You stated ..." my electricy- bill has already reached 10k " 

Over what period of time?  How many billing periods?  What are your rates for kilowatts per hour?
- - By ppipper (*****) Date 2010-04-17 06:51 Edited 2010-04-17 07:10
go ahead alex.

I totally support this kind of tool.

1 - it is supposed that you may use single engine versions (not clustere ones). Are you able to to use different engines and manage each hash file (per engine) along the net?

2 - how are the results of your analysis presented?

Thx very much
Parent - - By SchachProfi (****) Date 2010-04-18 03:08 Edited 2010-04-18 03:45
Dear all,

yesterday i improved upon "Brain", the hopefully-increasing intelligent mode-
dispatcher of Chessultant, which can -if optionally enabled- choose different
modes of further calculation based on the current move, the current evaluations
including the difference and history of difference between the evaluations and
of course -if not infinite=analysis- the time left in the game. It currently
works before the next move or phase, but i will work on a realtime-logic soon.

Today i implemented another mode, Linear, which is simply a queue of all (S).
(S)1 takes the best move of (M), S(2) the resulting best move of S(1) and so on.
As there most likely will be several changes throughout the whole queue while
calculation (each (S)n+1 has to be updated on (S)n or (S)0=(M) prefering another
move), this may be a massive test for the network-bandwith with the persistant
hash migration-option being enabled. In best case it enables a look to depth+16+
over the depth on (M) while sacrificing the width-accurancy and may be useful to
check and differ between winning and drawing (and also losing) lines in a pretty
speculative manner. As you may have experienced already during your analysis or
game-play, the best lines prefered by a lot engines may only lead to draws in
some situations while one of the nextbest may be able to lead to a won position.

In the next days i will implement the possibility to use multiple instances of
an engine of the same type per (S) and on the (M) (one engine per core seems to
be the lowest sensefull granularity here, most engines just work in multiple
instances on the same system by copying them and appending a new number to the
exe-filename; in other cases a virtual machine may help) to prepare a release
which makes sense for smaller clusters (two dual-cores being the lowest useful
entity in my opinion, well you could even run the cluster on a single quad-core
then to test it out). I will also create a simple documentation and improve a
little bit on the public-ergonomy of the GUI. Yes, a lot of work has to be done! :-)

@Brandon: Thanks for your kind regards. I will take 16 slaves eh beta-testers and
you will be on of them, hope you will like it when finally playing with it. Btw.,
looking at your profile, i lived three years in the windy city learning to trade,
do you know where the Merc (CME) and the Bot (CBOT, now part of CME group) are?

@Androcles: Yes, i indeed built it with optimizing my trading strategies in mind,
but my intuition regarding the markets and trading is still the best algorithm i
can offer till now. I trade mostly futures and only seldomly options (on futures),
so there is actually no reason to calculate BlackScholes&Hull up and down for me. ;-)
You know the most important point about indicators is that you know what they do
and how they are calculated, and its important not to optimize indicators and
automated systems building on them matching curves of the past but to imply your
knowledge of markets and trading into it. I don't trust literature to much (most
writing books about that topic do so because they simply can't earn money with
trading itself or are lazy and retired now) and i don't even ever considered all
that fancy indicators or systems i do not know how they work... instead i prefer
to do my own work and developed my own (semi-)automated systems... but i still hit
the button myself. Yes, there is of course more granularity in tick-data then in
the good old candles (well, you can build OHLCV's out of several ticks too) and
nowadays its pretty easy to use it, but you know, firstly trading is all about
risk management... thats were the magic is in, not tick-data ;-). And my opinion
only very fast systems (those sitting in the Merc or Eurex-Building) can make
money with scalping-systems, i'm concentrating on classic day- and swing-trading.
Text mining is indeed a very interesting topic, i build my own one on DJ Newswire
and Reuters, but again, i just use it to give me a quick traffic-light "green"
(you may have seen how stormy markets with US job market data or similar can be)
but i still hit the button myself even it would be able to trade alone, i have
never let it to do so... control can be everything and i love & prefer having it.
May i ask what you trade and what trading platform you use? And yes, of course
i'm willing to share knowledge, just drop a PM how i can help you or you me...

@mikal: Yes of course clustered analysis is neither new nor did i invent rocket-
science, there are several research-projects already out in the web regarding it.
I just offered informations about Chessultant because people asked me about it in
the "Rybka 4 Update"-thread and to see if there is an interest in realising it
to the public. If you read my lines you would have noticed that i'm not too much
interested in elo's or leading playchess.com-lists (thats kiddy stuff who should
be better of with playing WoW), i'm interested in chess-research and nothing else.
Btw., Rybka does 3000 Elo on a single computer aged a few years...

@dragon49: Well, it "eats" ~4kw an hour, i refered to the period of a year,
and of course was just a little bit ironic...

@ppipper: Yes, of course you may use single engine versions, that was the main
idea beyond creating Chessultant, because Vasik and Lukas won't share their
cluster version of Rybka in recent time and besides that there are other interesting
chess-engines out there too, and of course you can use different engines (see
above "engine-group"). The results are presented in a way every UCI-engine does
in multi-PV-analysis into your GUI, my own GUI shows additional informations which
are not covered by UCI-standard and therefore can't shown there about all the
deeper calculations done by Chessultant which were collected through the log-file.

I will get back to you and meanwhile await your further ideas and suggestions,
Best wishes,

Alex
Parent - By BrandonSi (***) Date 2010-04-18 05:31
Alex,

Thanks for sharing your progress! I'd be happy to help with the project in anyway I can.

Regarding CBOT, I work very close by. My office is near Dearborn and Monroe, 4-5 blocks from CBOT/CME. I've had a number of interviews with trading firms, as you can imagine they place a high importance on database development and administration (my technical area of expertise). I've never ended up working for a trading firm (I'm in the legal industry) but from my limited experience, the trading firms have some very impressive systems.
Parent - By peterhiarcs (**) Date 2010-04-18 06:05 Edited 2010-04-18 06:55
Dear Schachprofi, lieber Alex!
If you are still looking for one or the other slave eh beta- tester, I'd really like to be one of them but just don't know if my time, hardware and knowledge would allow me to.
You might already know me from CSS- forum, I just started a thread of my own there about your project, not looking at your nick and your profile closely first.
http://forum.computerschach.de/cgi-bin/mwf/topic_show.pl?pid=23619#pid23619
Ich verspreche auch weder auf Deutsch noch auf Englisch soviel unnötige Worte darüber zu verlieren, wie ich das aus reiner Schreiblust im CSS- forum manchmal gemacht habe. Ich bin nur ein Troll, wenn's der Sache dient.
:)
Parent - - By keoki010 (Silver) Date 2010-04-18 14:07
I would be interested in helping you beta test.
Parent - By dragon49 (****) Date 2010-04-18 17:53
Can your system can work with remote clients over tcp-ip.  This presents a lot of interesting possibilities.  At the next WCC, you can represent one of the other top engines, and "gang up" on Rybka with a 400 core beast!

In all seriousness, it is well known that the rate of elo gain decreases with each additional core.  Have you worked out the diminishing returns?  Do you know at what point, where adding cores will have no value.

I'd like to work with you on this as well.
Parent - - By SchachProfi (****) Date 2010-04-18 23:39
Dear all,

today i have created the possibility to use multiple instances of an
engine of the same type per (S) and on the (M) and will further test
it throughout the week. As an engine(-instance) per core makes sense,
16*4=64 sounds pretty good to me, but i will have to look what new
possibilities it provides to the above and yet-to-be-created modes.

Dear applying beta-testers, please write me about your clusters and
preferences you have in mind so i can try to cover them all with some
adjusted and optimized modes and custom search-patterns. As i already
said above, i would think of two dual- or better quad-cores as a good
system to start with. I designed the modes highly adjustable by the
GUI, config-file and/or parameters, so you can yourself optimize and
tweak them to your style and preference of calculation you want then.

Is here anyone who could host a little site + host for the beta then?

@Brandon: Hey cool, the world can be so small... i guess you could
help me in database questions then? What dbs do you mainly work with?
Oracle/IBM/Microsoft? Have a experience with a few smaller open ones?

@Peter: Thanks for your kind regards and welcome on our ship, i will
have a look at your thread! I have never posted in any chess-forum
till the last days (didn't know there were so many interested people),
so seems you may referencing to other people sharing my nick or name.
Are you involved in Hiarcs or is it your favorite engines? Well i may
have good news for you, i have written a small adapter to use the
non-UCI-engines of a bigger engine-developer and -distributer, i can't
officially support it when releasing but you may click&try and little
wonders may happen... ;-)

@keoki: Noted, thanks for your interest.

@dragon49: Yes it can, but i don't have to much experience with it,
and of course uploading a 2GB persistant-hash wouldn't make to much
sense there :-). I'm not familiar with what is allowed in WCCs or
what is not (aren't there limitations using only one computer and/
or something like that), if that would be possible, i would wonder
why Vasik & Lukas and all the others haven't gone this way already.
Of course the rate of elo gain (and the same goes for the efficiency
overall) decreases with each additional core, but as long as we're
speaking about sinking "gains" there is no worry... there of course
is a point where adding additional cores makes absolutely no sense
anymore because the overhead of parallelisation of the calculation is
bigger than the gain, but this is not the case on a 16+ chip-cluster.

Best wishes,

Alex
Parent - - By peterhiarcs (**) Date 2010-04-19 03:45 Edited 2010-04-19 03:48
Hi Alex!
Sorry for confounding your nick obviously. The thread in CSS was mainly meant for somebody else I thought I'd know already. But you could send me private message there or here with an email adress so we could share further details thereby.
Hiarcs is just one of my favorite engines and the hardware I could use is at least two dual and one quad. By the way, as you probably will have noticed, my mothertongue is German, just in case yours is the same yet.
:)
Parent - By SchachProfi (****) Date 2010-04-19 23:07
Peter, you can reach me in English/German or French under my mail:

Dr(point)Alexander.Schmidt(atTheDomain)hotmail.com

Thanks for any input,

Alex
Parent - - By BrandonSi (***) Date 2010-04-20 00:37
Alex,

I mainly work with MS SQL Server, but have experience with MySQL and PostgreSQL as well. Happy to help answer any DB questions. Not sure what you're looking for as far as "clusters", but I have an i7 laptop (820QM), and i7 desktop (860 @4Ghz), as well as an older C2D laptop.
Parent - - By SchachProfi (****) Date 2010-04-20 00:49
Good, i'm currently thinking about my own database-approach to store the positions and calculations for Chessultant with having (now Sun's) MySQL or maybe PostgreSQL in mind, will contact you when having questions. For calculations on-the-fly i use my own datastructures which are fine & speedy, but for storing all the positions & calculations long-term i will better go with a database instead of the currently saved logfiles-only.
Parent - - By BrandonSi (***) Date 2010-04-20 02:24
I completely agree.. Relational databases would be advantageous there.. Keep in mind stored procedures could provide increased efficiency versus regular SQL statements, as stored procedures can be precompiled (at least in MySQL and MS SQL Server).
Parent - - By SchachProfi (****) Date 2010-04-20 02:45
Brandon, thanks for your kind answer. I for myself might experiment with Oracle or Micro$oft here, but want to keep things open and universell so the others aren't required to use these but can use MySQL or PostgreSQL, which limits the usage of stored procedures to standard ones. :) The database-scheme is simple anyway, as i just want to save the main evalutions and the log-files to a position... do you think the position's-FEN (for example: "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1") is a good unique key? Alex
Parent - - By BrandonSi (***) Date 2010-04-20 03:24
That's a really interesting question, I'll have to do some thinking on that.. My initial thought is that FEN should obviously be unique for a position, but that position can be reached from a variety of methods, so it may not serve the right purposes if you're storing evaluation.. What data would you be relating to the key (or FEN)? Whatever you do, you'll obviously have to use a case-sensitive collation if you're going to store FEN.
Parent - - By SchachProfi (****) Date 2010-04-20 03:46
http://en.wikipedia.org/wiki/Forsyth%E2%80%93Edwards_Notation
Yes, the FEN-position part1to4 gives a unique position including castling-possibilites and en passant-information and also what color to move, part5&6 just provides additional information (halfmoves since the last pawn advance or capture and number of the full move) which i will cut. In my philosophy it doesn't matter how a position may be reached, because the (forward-)evaluation of the engines stays the same. My concern is more whether there are better alternatives out of a performance point-of-view, because before every position check the following has to be processed: a) FEN needs to be calculated/formed and b) a complicate string/VARCHAR-pattern-search on the database has to be performed. But still to me the FEN seems a nice enough solution to my question, but i'm open for better ones. Alex
Parent - - By Banned for Life (Gold) Date 2010-04-20 04:49
The FEN for the current position is not sufficient because it does not point out positions which are good or bad because reaching them allows a draw by repetition. To get around this, you need a list of FEN that goes back to the last irreversible move (i.e. a pawn move or a capture).
Parent - By SchachProfi (****) Date 2010-04-20 23:32
Banned, thanks for your valuable input. Well, i just have a for-now-simple database in mind to look up calculations&informations which were already collected in that position, therefore the FEN seems sufficient for my aim. The information to avoid a drawish position will be stored in the corresponding parent-position(s), but not in the position which is already drawish which may be never reached if not wanted, here "good or bad" will be also represented by the corresponding evaluations to the possible moves. A (double-)linked-list may be interesting with more calculations going through the positions-tree in mind, yes, i will have to further think about a good datastructure&-representation here. Alex
Parent - - By oudheusa (*****) Date 2010-04-20 05:38
One problem is capturing three move repetition. This also popped up in Aquarium's IDea. I think they found a solution now. Perhaps one of the Aquarium experts can offer their insights.
Parent - - By BrandonSi (***) Date 2010-04-20 12:40
Once a position that can result in 3 move repetition has been identified, setting a flag would be easy, but I agree, I think the difficulty here lies in determining if a given position (or its preceding position) creates a 3 move repetition situation.
Parent - By SchachProfi (****) Date 2010-04-20 23:35
Yes, i had also a flag in mind (for the corresponding parent-position), of course a lot of 3folds also allow you a little sacrifice and still play on, but the ones without a major disadvantage may be seldom... true. But at the moment i can't make all the information being part of the calculations of the child-engines, i just can make it part of the desicions which directs those child-engines... thats why a simple lookup with stored calculations/informations + draw-warning and a few other thinks is sufficient in my thinking. Alex
Parent - - By BrandonSi (***) Date 2010-04-20 12:38
b) a complicate string/VARCHAR-pattern-search on the database has to be performed.

This would be were an index (perhaps clustered with depth value / engine ) would be useful.

I personally have never looked at the code to generate FEN, but I would imagine it's not too time consuming. I'd also been thinking some sort of hash table might be useful for this, but the overhead for the hash function could be an issue.
Parent - - By SchachProfi (****) Date 2010-04-20 23:41
Brandon, i have thought about other possibilites, but FEN seems to be a pretty standard-way and i want to support standards. Of course i could encrypt it into an own format... chessboard has 64 fields, need 4bits to store what piece is on or not and a few other bits about the rochade- and en passant-possibilites, would be a little more speedy but not standard... hash would be another possibility yes, but we just have to find a good-suited-FEN-one to not get a degenerated list-like-one. Alex
Parent - - By Sesse (****) Date 2010-04-21 23:26
You can store a chess position (including en passant and castling) in 21-22 bytes or so if you really want to, by using a simple form of Huffman encoding -- no need to use 4 bits per field. Look at the CTG spec in http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=2319, for instance.

I don't really see the point, though. What are you trying to do here -- just store the position, or actually query against it? If the latter, what kind of queries do you want to run?

/* Steinar */
Parent - - By SchachProfi (****) Date 2010-04-22 00:11
Well, i don't care about a few bits more or less, building the information straight-forward into 4 bits to get a easy big number to compare against suits my needs, thats seems to be the most speedy way as the MMX/SSE-extensions of x86 provide nice instructions to do so.
The kind of queries depend whether it will stay the UCI-kind it is or further develop into a native approach... for the UCI-kind i just need a few queries per move to doublecheck (nothing performance-conserning at all, but i prefer to build solid solutions from start on), for the native one there maybe several billions per move needed to look for all possible positions... i will further think about it, thanks for your input. Alex
Parent - By Sesse (****) Date 2010-04-22 11:42
You can't really compare four and four bits in MMX/SSE, unless you mask off only the parts you're interested. You can, of course, compare the entire position, but if so, you could just as well use a Huffman encoding and get less to compare.

You still haven't really said what you want to query by, FWIW. The entire position? If you need billions of lookups in a very short time, an SQL database might not be for you, though.

/* Steinar */
Parent - - By Sesse (****) Date 2010-04-21 23:21
That's easy, as almost any database in the world except MySQL will be case-sensitive by default. :-)

/* Steinar */
Parent - By BrandonSi (***) Date 2010-04-22 00:09
Actually, MS SQL Server is case-insensitive by default, as well.
Parent - - By Sesse (****) Date 2010-04-21 23:28
What do you mean by “MySQL or PostgreSQL [...] limits the usage of stored procedures to standard ones”? PostgreSQL has a wide array of stored procedure languages available, actually none of which are ANSI SQL.

/* Steinar */
Parent - By SchachProfi (****) Date 2010-04-22 00:14
Yes, because i want to keep it open&universell to any kind of database one might prefer thats why i can't use Oracle-propertiary stored procedures and so on, for my own version i will go with my prefered one anyways. Alex
- - By ppipper (*****) Date 2010-04-20 16:44
Alex,

when you deliver tasks to the different computers, are they sharing hash files (I mean tasks on the same computer will benefit from previously analyzed tasks)?
Parent - By SchachProfi (****) Date 2010-04-20 23:37
Yes, read above ;-), i have implemented an option to get the parent-hash or the best-suited available.
Of course this option limits the possibilites of inet-clusters, because uploading 256MB or 2GB may take its time... and be useless in some cases. Alex
- - By ppipper (*****) Date 2010-05-01 08:50
Alex, your Chessultant is optimized for 16x4 (phisical) cores, isn't it?

Is it possible to optimize it  for different configurations (to be defined by user)

Thanks
Parent - By SchachProfi (****) Date 2010-05-01 23:30
Hi ppipper,

thanks for your kind interest in Chessultant.
I hope to get it out in the month of May, as you know the whole
(computer-)chess-world is waiting for Rybka 4 now, including me.

With the release in mind i (re-)designed the modes to be highly
adjustable via custom search-patterns by the GUI, config-file and/
or parameters, so you can yourself optimize and tweak them to your
style and preference of calculation you want then. I will provide
a few search-patterns for clusters (beta-)users asked for or i
thought of being practical useful already with it, if you have
a special cluster in mind please just drop a line here. I improved
Chessultant for working with unsymmetrical architectures including
more balancing and intelligence in the analysis of results, but of
course a symmetrical architecture -the more cores the better- will
always be advantageous, like with most other kinds of parallelisation.

Wishing you all a good start in the month of May,

Alex
- - By ppipper (*****) Date 2010-05-02 15:22
well atm I just have a single i7 75% chess, but i think in a short future I could add two more i7 at nights
Parent - By SchachProfi (****) Date 2010-05-02 17:15
@ppipper: You can simulate Chessultant on this single i7 (one core for the (M), the other three for the (S) (or even seven (S) with hyperthreading enabled)), but of course the magic will just begin with the additional i7's... here i have already implemented a custom search-pattern for you: (M) runs on one i7, the other two i7's can host 8 one-per-core (S)s.

Please drop me a mail to my email-address: Dr(point)Alexander.Schmidt(atTheDomain)hotmail.com

Have a great time,

Alex
Up Topic Rybka Support & Discussion / Rybka Discussion / Chessultant - UCI chess-cluster-engine

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill