Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Support / Loading 6-men TB's issue takes 45 friggin' mins!!
- - By yorkman (**) Date 2011-12-20 01:06 Edited 2011-12-20 01:12
I have Win'7 x64, 16 GB of ram and 3930K cpu.  Nalimov cache size doesn't matter for this issue as the result is always the same.

One problem I have, and I've always had this for many many months is, when loading table bases (6-men) R4 always has to load it 3 times!! I can see this in Task Manager. 

After "Loading" R4.1x64 (I press Load to load the engine), Aquarium is then frozen for about 15 mins while it's loading the tb's into memory.  I can see the Memory column in the Processes tab for each R4 core and it'll (slowly) rise to a size of about 1.5 GB per core before my Aquarium can respond to anything.  Once it reaches the 1.5 GB size it automatically starts all over again, repeating the same cycle a 2nd time and costing me another 15 mins!  Once 1.5 GB per process is reached for a 2nd time, it then stays that way until I press Analyze and while analyzing I load the hash file for that game.  Then it resets the memory used for each process for a 3rd and final time and once each process reaches 1.5 GB in size it is then (finally) finished!  So that's 45 mins. wasted each time I want to run R4. 

This time could've been better spent analyzing the position at hand (from where I saved the hash).  Instead, I have to wait for it to finish jerking itself off before it's ready to work.  Makes me wonder if I'm really gaining anything from using TB's at all...it already makes R4 run slower with it enabled so why bother?!  And I'd save myself about 1.2 TB of space on the hdd too if I got rid of it.  But I do believe it finds mate quicker with it on when you're trying to finish your opponent off so I'd prefer to keep it if this can be fixed.

To make matters worse, after loading my saved hash file, Task Manager will show all processes are showing 0% cpu usage (except for one) until I stop analyzing and then start it again.  That's actually when R4 is working perfectly.  Very frustrating to use TB's! Anybody else have this problem, or know why I'm having this issue?!

I had the same behavior on my Q6600 pc with 8 GB ram (though it was a bit faster since it only has 4 cores to fill).  Is there any way to fix this? Or at least a workaround?

I'm using 256 MB hash file size but I tried increasing that to other sizes and it doesn't make a difference.  I also noticed that no matter what size I set the hash size to, it doesn't seem to reflect that when I look at all the R4 processes in Task Manager.  They always look like this with TB enabled & 256 MB hash size (on my 3930K system):



On my Q6600, with TB disabled it looks like this (and still 256 MB hash):



On the 3930K, it'll be almost the same as the one for Q6600 above except ALL the processes will be about 40K in size.  So it doesn't seem to reflect the size of hash file I set it to. Only if I hover over the R4 engine on the bottom left in Aquarium will I see what hash file R4 is running at...and when saving a game's hash it's also 256 MB.  But shouldn't Task Manager also show that each process is 256 MB? Or is 40K per process correct which equals to about 256 MB?
Parent - - By Lukas Cimiotti (Bronze) Date 2011-12-24 10:26
It's not a good idea to have all 6 man tablebases as it wastes too much memory and time. This is my recommendation for a good set of tablebases: http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?pid=258084#pid258084
If you want to have a real benefit of tablebases, you should put them on an SSD.

>But shouldn't Task Manager also show that each process is 256 MB?


No, only the parent process does have hash, the child processes don't - but they can access that memory.
Parent - By yorkman (**) Date 2012-01-03 19:15
Thanks for the reply Lukas.

I hear ya on the "wastes too much memory and time" and I'd add "space" too.  I downloaded ALL the 6-man tablebases from sesse's ftp site a long time ago...1.2 TB's worth.  It contains:

3+3
3+3+p
3+4+5
4+2
4+2+p

Out of all these, and for most efficient and strength, which should I keep? They're on a 2 TB raid 0 array using 7200 rpm hdd's (until I one day upgrade to ssd's).

I checked your link and the guy that asks that question doesn't really get the answer (I read all 3 pages). It mostly talks about hardware anyway (ssd's, hdd's, etc).

I think if I were to delete some of those it would cut my loading time significantly.

But I still don't understand why does R4 have to reload the TB's into memory so many times as I mentioned before? Just loading R4 (without analyzing yet) takes about 15 mins (all 1.2 TB)....I can see in Task Manager how the memory usage is increased.  It then seems to reset itself and load a 2nd time when you start analyzing.  Then a 3rd time after you load the saved hash for a game.

Also, with 16 GB of ram, and after removing whatever TB's you suggest, what do you suggest I set the engine hash to for R4.1 SSE42? I also assume I should set it to Rarely until I upgrade to ssd's?

And what pagefile, if any (I currently have that disabled completely), should I set it to?
Parent - - By RFK (Gold) Date 2012-01-04 03:27
What is your assessment of having 6-man TB 4+2 Pawnless 110 Gigs.
Parent - - By Lukas Cimiotti (Bronze) Date 2012-01-04 07:47
My set only has 6 man TBs with pawns. 6 man positions without pawns occur very rarely, so these TBs are (almost) worthless.
Parent - - By RFK (Gold) Date 2012-01-04 07:56
Thanks for letting me know. Without trashing the entire set of 6 man  can I supplement them by download just the necessary pawns or do certain tablebases have to be discarded.
Parent - - By Mark Mason (***) Date 2012-01-04 08:10
Hi,

Just to add to what Lukas has said about not needing all the 6 man tables. These are not my words but I saw this on aother forum and thought it useful to share:

Best advice is to only collect the most important ie. the most frequently occurring combinations. There has been some exhaustive analysis on this and there are 365 possible 6-man combinations. If you collected just the top 20 you will have over 56% of the 6-man situations that usually happen in engine vs. engine games. If you collected the top 50, you will have over 79% of situations that occur in engine/engine matches. Also some tables are for positions that are very unlikely to ever happen so no point having them eg. any 5+1 combinations because if your engine can't mate with four pieces to a king without a TB file you need a new engine eg. KNNNNK as when would you ever have a King + 4 Knights ?? Also KRRKBB (King + 2 Rooks vs K+ 2 Bishops) happens less than once every 100,000 engine games so no need to have it.

There is a list of top 100 most frequently occurring combinations, ranked in order. If you have all of these it would cover about 92% of all 6-man combinations that actually happen in engine games - this is far more than you need . So my advice is download them in the order of most frequently occurring, from the top of this list downwards. The list is given in the first post of this thread :

http://kirill-kryukov.com/chess/discussion-board/viewtopic.php?t=59
Parent - By RFK (Gold) Date 2012-01-04 08:17
Mark,

Thank you for being gracious enough to take the time in sharing  this information with me. I'll do exactly as you indicated. Thanks again, and thanks  for the link.

Best Wishes,
Robert
Parent - By Banned for Life (Gold) Date 2012-01-04 09:03
That's a good list even though it was put together by a shady character that goes by the name of Cato the Younger. :lol:
Parent - By Nelson Hernandez (Gold) Date 2012-01-04 13:02
Thanks for referencing my article, Mark.  I am 'Cato the Younger'.
Parent - By Lukas Cimiotti (Bronze) Date 2012-01-04 16:17
Actually I picked those TBs which occur most frequently - but I kicked out all stuff that's an easy win - like krppkp.
Up Topic Rybka Support & Discussion / Rybka Support / Loading 6-men TB's issue takes 45 friggin' mins!!

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill