Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Aquarium / multicore vs. single core local engines best for IDeA?
- - By Loboestepario (****) [us] Date 2010-02-19 04:21
I bought a new i7-920 computer and started experimenting using single core vs. multicore local versions for IDeA.

With single core local rybkas, I can't get more than 50% of the total CPU involved, each process takes only 13% CPU (which totals 4x13~50%).

If I use multicore local instances of Rybka, the task manager displays 16 rybka processes (as opposed to only 4 with the single core version), some of them idle, and the CPU usage for each process varies from 0 to 13%, totalling 100% CPU usage.
Both cases involve the same number of tasks being analyzed (4).

I suspect it's better to use single core instances but it also feels like a waste not to utilize 50% of the available CPU.

Which one is best?
Parent - - By Dhanish (***) [in] Date 2010-02-19 06:00
There is some problem in your engine / system setup. In either case you should get 100% CPU usage.

Is hyperthreading on? In single core, did you set the number of CPU cores to 1? Is RAM sufficient? How much RAM did you allocate for each core?

Last week, I also purchased an AMD Athlon 2 X4 and I get 100% in both cases.
Parent - By Loboestepario (****) [us] Date 2010-02-19 06:43
Yes hyperthreading is apparently on, according to the person that built the computer for me. RAM is default. These are local engines being used with IDeA so I entered "4" not 1. I don't understand why hyperthreading will "affect" the single core engines and not the multicore ones...
Parent - - By buffos (Silver) [gr] Date 2010-02-19 06:06
i am not familiar with i7's but could there be something involved with hyper threading? Could you turn hyperthreads off and try again? ( i am almost sure this is why its happening. Its also known hyperthreads do not work well with rybka..)
Parent - - By Loboestepario (****) [us] Date 2010-02-19 06:45
Ok, I'll check with the seller and see how to turn hyperthreading off. Wish me luck because I have no idea how to even access the bios
Parent - - By Loboestepario (****) [us] Date 2010-02-19 07:42
You guys were right, disabling HT makes the single core rybkas utilize 100% CPU.
Thanks!

Gino
Parent - By buffos (Silver) [gr] Date 2010-02-19 08:24
it seems logical , based on your comment "the task manager displays 16 rybka processes" :)
Parent - - By ppipper (*****) [es] Date 2010-02-19 10:08
I have an i7 too.

If HT is on, then you should use 8 engine's intances, thus you would process 8 tasks at a time.
Parent - - By buffos (Silver) [gr] Date 2010-02-19 12:35
and how do you make sure that an engine is on an actual core and not running on a virtual core?
Parent - - By ppipper (*****) [es] Date 2010-02-19 14:36
I did several tests some time ago. In principle, when you are running 8 instances, you can actually see 8 green boxes and 8 process in admin tasks. I think you can't know if the task is run by a virtual core or by a physical one.

Nevertheless, most of time is better to run just 4 cores. I have to check the opposite when you are running more than 3 projects at a time.
Parent - By Loboestepario (****) [us] Date 2010-02-19 16:55
I think an interesting test would be to compare 4 single core rybka processes with hyperthreading off versus 8 single core processes with hyperthreading on. If I understood you correctly the latter allows 8 tasks being analyzed simultaneously?.
I'll try to do this once I get some free time this weekend. We'll see if I get the memory issues you experienced.
Parent - - By Vempele (Silver) [fi] Date 2010-02-19 15:38
With HT on, all 8 cores are virtual.
Parent - By buffos (Silver) [gr] Date 2010-02-19 15:40
yes ofcourse.. mea culpa
Parent - - By Chaz (**) [us] Date 2010-02-19 15:24
I have an i7 920. My observations working with IDEA and HT on in all cases are:

1) With one multi-cpu rybka engine installed my performance monitor shows 100% utilization for all 8 processors.
2) With one 1-cpu rybka engine installed configured for 1 instance I'm seeing 12-13% cpu utilization. This makes sense since the resourse monitor is showing 8 cpu's, 1/8 = 12.5%. Similarly 4 instances of the 1-cpu engines results in 50% utilization.
3) If I run 8 instances of the 1-cpu engine I become memory constrained (I have 6GB ram installed), so any cpu utilization score is meaningless (it was bad).

I'll defer to the experts on this, but with IDEA doing short time/depth evaluations I suspect that HT is less of an issue compared to optimizing processor utilization.

There's some other factors going on in IDEA worth looking at as well. I ran a couple of projects with one Rybka multi-cpu and again with four Rybka 1-cpu engines. When I aproximate clock time with a stopwatch I get nearly identical move evaluations and move order, but the 4 instances of 1-cpu crank out a lot more positions to look at. (I really don't understand how to use cpu time to compare true processing time).

Anyway, that's my lesson's learned. I thought that the 1-cpu configuration was someting of a free lunch with all of the extra tree positions.
Parent - - By buffos (Silver) [gr] Date 2010-02-19 15:29

> When I aproximate clock time with a stopwatch I get nearly identical move evaluations and move order, but the 4 instances of 1-cpu crank out a lot more positions to look at.


There is not evaluation , because IDEA did not find anything to radically change the evaluation (very common)

> I really don't understand how to use cpu time to compare true processing time


Cpu time = time the engine works X number of cpu's

So if it says 4 hours and you have a quad-pc , then you have been working for 1 hour
Parent - By Chaz (**) [us] Date 2010-02-19 21:02
Thanks buffos, your posts and videos have been an enormous help for me.

I understand what you are saying about the evaluations, so I had created new projects and trees and cleared hash for each case to make sure I was doing an apples to apples comparison.

From a practical perspective an hour is an hour, so if I have an hour to evaluate a position I'm interested in configuring IDEA to best utilize that time. My hour includes engine cpu time + idle time (actually my little test was for 20 minutes).
Parent - - By Loboestepario (****) [us] Date 2010-02-19 19:48
Thanks for your feedback. I started IDeA with 8 single core instances of rybka, HT is on, and CPU is 100%, so far 1 hour and no obvious problems yet.
My IDeA method is 300 sec OR d=23 and so far it looks like HT gives me twice as many cores. I'm running 2 IDeA projects simultaneously. I'll keep watching over the weekend but I am very impressed :)
Parent - - By Chaz (**) [us] Date 2010-02-19 21:46
Your quality setting are much more stringent that anything I looked at. I'll be curious to know how your project works out.

I did get 1-cpu 8 engines going at 100% with 5.1GB memory utilization, but I'm reluctant to run continuously at that level since I never know when windows will spin up some other tasks. You must have more memory or courage than I do.....
Parent - - By Loboestepario (****) [us] Date 2010-02-20 00:38
It's about 5 hours now. Things look pretty good, each project is getting 6 cores almost all the time (8 local and 4 remote), CPU 100%, memory 1.96-2.0 GB all the time (my computer has 6 GB RAM, i7-920 factory overclocked to 4 GHz). So I am not seeing any memory problems.

Each project produced about 360 tasks/5 hours with time 300 sec OR d=23.

This looks like a cheap option of getting 8 cores. The question would be if the Aquarium team would be able to network 2 or more of these computers with virtual cores. I don't think the current remoteIDeA server would support virtual cores...
Someone posted that cloud computing can provide 24 cores (2.4 GHz) 24hr/day for 1000 USD a month. I could buy 3 computers like mine for under 4000 USD and get the same number of cores forever :)
Parent - By Loboestepario (****) [us] Date 2010-02-20 20:51
I got a blue screen after 24 hours. Restarted the computer and the tree seems ok. I am now trying 7 local engines plus 4 remote ones.
Parent - - By keoki010 (Silver) [us] Date 2010-02-20 17:42
You could run 7 instead of 8. I run 7 on my main computer and some more on another machine I have.
Parent - - By ppipper (*****) [es] Date 2010-02-20 21:53
just my humble opinion:
- analyzing till d=23 with IDeA is a total waste of IDeA power. You can easily see that in most of positions you will get similar moves and in the same evaluation order than analyzing with d=15. Of course, some positions mayu require deeper settings.

- analyzing with 8 virtual cores may be little bit better (less than 1.3x) when you are running 2-3 projects at a time. However the gain is no so big since CPU speed with HT on, cannot be as fast as with HT off (i assume we are overclocking)
Parent - - By Chaz (**) [us] Date 2010-02-20 23:32

> - analyzing till d=23 with IDeA is a total waste of IDeA power.


That is an interesting point! Since IDEA is going to minimax the tree what do we get by going to the extreme depth vs quickly moving down the tree with more moves and shorter move evaluations?

I wonder if you would get to a d=23 in 300sec in many positions anyway. When I ran a recent project with 225sec AND d=20 too many tasks ran to my max time setting of 600sec.
Parent - By Loboestepario (****) [us] Date 2010-02-21 02:00 Edited 2010-02-21 02:17
I don't know which way is better but I like to move the depth up close to the time I want to spend in each task. In order to do this I divide the number of positions(# of tasks analyzed in project status) by the CPU time in hours. I believe this estimates the % of short tasks (duration< time I set, in this case 300 sec).

For a time of 300 sec this should be a bit higher than 12. If it's lower than 12 then in no case the depth of 23 was reached. I am looking for something between 12 and 13 (for time=300 sec) which is equivalent to a range of 1/12 % of short tasks (0-8%).

The depth for me is dependent on the "% of short tasks" for the time I arbitrarily want to allow for each task.
In other words, for the positions I am analyzing right now, I am seeing 0-8% of "short tasks" with a time of 300 secs.
Up Topic Rybka Support & Discussion / Aquarium / multicore vs. single core local engines best for IDeA?

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill