Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Aquarium / Clear Tree to analyze with another engine
- - By ChiefPushesWood (**) Date 2019-06-05 19:08
Hey folks,
Can someone describe the process to 'clear' a tree so that I can re-analyze all of it's positions using a different engine? I've found a couple of posts about this but they are pretty old.

Thanks for helping,

Parent - - By Ghengis-Kann (***) Date 2019-06-05 22:11

You can use the tree utilities that are hidden under the fish icon in the upper left.

Export the tree as an EPD set, then import that into an IDEA project and analyze the positions with the new engine.
Parent - - By ChiefPushesWood (**) Date 2019-06-05 22:29
Thanks Ghengis...
So the evals don't stay with an EPD? I'm not familiar with them.

Parent - - By Ghengis-Kann (***) Date 2019-06-06 02:43
It's just the positions.

Dickie answered this question with a bit more explanation in a thread towards the bottom of the page entitled  "Overwrite Evaluations in IDeA Tree".
Parent - - By ChiefPushesWood (**) Date 2019-06-06 04:03
Ok... Figured out how to do this...
Another question though. Is exporting to an .epd file actually necessary? For example, what is in the .epd file for a given project in the file location \Data\IDeA\Results

Parent - - By pawnslinger (****) Date 2019-06-06 07:46
Hi Chief:

If you have a large tree, re-evaluating each position is, of course, very time consuming.  I have always just updated the tree in-place.  The method I use has the advantage of time, but that is also a disadvantage (as it is unclear when the tree is completely ready for serious use).

On my BIG trees, 4m+ nodes, several hundred 1,000 evaluations may not be enough, and it is hard to tell when one has done enough.  But even if I had to do 1 million evaluations that would still be less than the 4m+ nodes in the tree.  And yet, there is still some uncertainty about the reliability of the tree.

Instead of exporting the entire tree to epd, it would be nice if there was a way to just export all the leaf nodes.  Then those could be evaluated and re-imported with new evaluations to the tree (minimax would do the rest).  However, this is just wishful dreaming.  An old program that I used to use during the 90s (long ago), actually had such a feature... an old program that no longer runs on modern systems and modern OSs.
Parent - - By dickie (**) Date 2019-06-06 10:37

The method you describe has been available for many years with Moz's script However it does not work because the assumtion is flawed, that it is sufficient to re-evaluate all the leaf nodes in order to re-evaluate a tree. Try it, and you will see. The problem is that many leaves, often the majority of leaves in a tree, do not exist at leaf nodes but rather as childless alternatives to positions scattered throughout the tree. All the leaves have to be re-evaluated to give a valid re-evaluated tree. If memory serves, when I looked at this a few years ago, in a typical tree something like 50-80% of all tree positions contain leaves. This is one of the reasons why, when using a new or upgraded engine, starting a new tree can be expected give better results than continuing to grow an old large tree.
Parent - By pawnslinger (****) Date 2019-06-06 12:19 Edited 2019-06-06 12:22
Hi dickie:

I never tried this script, although I have heard of it.  I do not think that the problem is with the script or the method, but with the way that Aquarium looks at leaf nodes.  If you actually look carefully at what Aquarium is doing, perhaps we will find that the tree is actually a collection of moves, not positions.  In this case, it would not make logical sense to have an empty leaf stored... because it would only come into existence when a move was played from it.  But often times we do see that the node can exist in the other associated trees, such as the IA tree.  Where leaf nodes can exist without moves as a result of analysis.

Funny, I think the program I mentioned from the 90s did not have such a concept as IA or these other "columns".  It did not store moves as such, to it a game was simply a linked list of positions (moves being the thing that formed the links).  So if you looked into its database all you would see would be a collection of FEN.  The moves or links were stored in the index file (and there was no leaf node in the index file (think about it, it will bend your brain).

A long time ago, I opined that Aquarium seemed to assign evaluations to moves, not positions, and that I thought this was strange ... I was used to the other program, which assigned evaluations to positions, moves being the links between, did not have evaluations!!  Two different ways of looking at things, but it would explain the discrepancy that made the script you mention fail.
Parent - - By dickie (**) Date 2019-06-06 10:01 Edited 2019-06-06 10:38

The results epd files hold the result of every completed task in each project. If you open the file in a text editor you will see the position fen, eval (ce), depth (acd), time taken (acs) and the pv, line by line for every completed task for a project. The tree can be recreated from the results file, but will include the evaluation which you want to strip out. G-K has given a working solution. A description and instruction for each epd and tree utility is at They are easy to use.
Parent - - By Ghengis-Kann (***) Date 2019-06-06 17:04
Hi Pawnslinger.

I'm not sure how you concluded that a position is only added to the tree when there is at least one move from played from it.
My experience has been that most of the "bad" leaf node evaluations are dead end alternatives that were added to the tree by Aquarium.

Minimax should insure that different moves leading to the same position get the same evaluation.
Parent - - By pawnslinger (****) Date 2019-06-06 19:29
Logically, in Aquarium a position in a tree that has no further position after it, is an end node (or leaf node).  I have never seen Aquarium store an evaluation for such a position.  Aquarium only adds evaluation when there are candidate moves to assign the evaluation to.

By definition, in my opinion, a node that contained a candidate move, could not be a leaf node.  The leaf node would be the position, if contained in the tree, that was pointed to by that candidate move.  If a node contained let's say 3 candidate moves, then the nodes pointed to by those candidates might be leaf nodes (we would not know for sure until we access those positions). 

I mean by definition of Chess, the final position (or leaf node) is the position which contains no further moves... in pgn notation, it would be the node that contained the final evaluation (result).  I have never seen such in an Aquarium tree.  So it is entirely likely that the script described by dickie would fail.  After thinking about it for awhile, how could it not?
Parent - - By Ghengis-Kann (***) Date 2019-06-06 20:21
I get it now.

But all of the moves that result in the same position would have to have the same evaluation, so the position is being evaluated, but the information is displayed as associated with a move in the tree.

There is that weird thing where it will report thousands of moves downstream from a position when really it is 3 moves that go in a circle.
That would not happen if it was only evaluating the position once, so maybe you are right about storing the evaluation with the move.

That also creates an issue related to the current thread, because none of the 3 positions are leaves.
Parent - - By pawnslinger (****) Date 2019-06-06 21:17
The really odd thing is that Aquarium does store empty nodes!!  But they are NOT leaf nodes.  This happens in the course of IDeA Auto Prolongation.  Aquarium will create downstream nodes, sometimes many moves deep, that contain no real moves, just entries in the IA column.... so kind of precursor moves, if we can call them that.  If you click thru one of these precursor moves and get to a true leaf node, you can hit "Alternative" and a funny thing happens... Aquarium suddenly realizes that you are adding a move into a node that is not really truly connected to the tree, so it automatically adds the evaluation steps for the intermediate positions.  If it did not do that, then the minimax wouldn't work... there would just be a leaf node hanging out at the end of a sequence, but no intermediate positions to connect it to the tree.

At least that is my interpretation of what happens... I could be wrong.  Just my observation.
Parent - - By dickie (**) Date 2019-06-07 09:41 Upvotes 1
A tree consists of nodes and the links that connect the nodes. According to the old Aquarium scripter documentation a node in Aquarium stores information about a position including the fen of the position, an array of moves from the position, the score of the node (evaluation of the best move from the position), an array of backward (parent) positions etc, etc. The link (move from the position) stores information about the move including its evaluation.

A new node is only created when a new position is evaluated. The evaluation has a “move to” but a new node is not created at the “move to” position. If the “move to” node already exists then the link is established, otherwise the link is left hanging. It is these hanging links that I refer to as leaves, though that may not be the correct terminology – I am not a programmer or mathematician. The leaves, as I call them, obviously exist at leaf nodes but also throughout the body of the tree as move alternatives that have never been prolongated. It is likely that they were not originally prolongated because their score may have been much weaker than their node score. However, over time and much minimaxing, some of these leaf scores may become significant, but the leaves are not developed because the Aquarium roots have moved further into the tree.

Moz’s script works as it claims “It generates a copy of an existing IDeA tree  - minus the evaluations - and then exports the leaf nodes to EPD so we can add them as tasks later.  The net result is that the structure of the new tree is identical to that of the original but the evaluations are all replaced with evals from a new engine”, but with one exception. The evaluations of the leaves in the body of the tree are not replaced. In practice I found their absence had rendered the new replacement tree quite unsatisfactory. I tried to amend the script to export all leaves to epd to solve the problem, but soon discovered that these leaves exist in such vast numbers that it made more sense to create a new tree rather than re-evaluate an old tree.
Parent - - By pawnslinger (****) Date 2019-06-07 11:53 Edited 2019-06-09 05:23
Parent - - By dickie (**) Date 2019-06-08 09:22

> As my son would say, "More information than I needed, Dad!"

A little patronising?

I was hoping to fill in some of the gaps in your knowledge, apparent from your earlier speculations.
Parent - - By pawnslinger (****) Date 2019-06-09 05:23
Okay, I apologize.  I take it all back.
Parent - - By dickie (**) Date 2019-06-09 06:21
It's okay. Not a problem. I know you are a great enthusiast.
Parent - By pawnslinger (****) Date 2019-06-09 07:34
I do not appreciate these terms: "patronising" or "great enthusiast".

I repeat my request to the Admin... please remove all my messages from this thread.  Thank you.

Maybe the Admin is not reading this thread.

I feel that my messages in this thread were off-topic and unhelpful.  Please remove them.  Thanks.
Parent - By pawnslinger (****) Date 2019-06-09 05:27
Sorry, it would only allow me to take back the last message.  Imagine the others are gone too.  If an admin is reading this, please remove all my messages from this thread.  Thanks.
Up Topic Rybka Support & Discussion / Aquarium / Clear Tree to analyze with another engine

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill