Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Aquarium / Supercomputer 100 cores & Aquarium
1 2 Previous Next  
- - By Merkury (*) [ru] Date 2016-10-01 06:53
The supercomputer will be presented at the Russian Championship Superfinal in Novosibirsk from 15 to 28 October.
We have updated the Stockfish engine for IA analysis now uses faster PedantFish engine - an exact copy Stockfish assembler.

Updated block diagram supercomputer.
https://yadi.sk/i/3i_VNK1qvyTKw

The new record - 84 engines (a very good and stable operation of the Aquarium2016)
https://yadi.sk/i/1S2B-EYUvyZRs

http://superchess.wixsite.com/chesscomp/--2-c8gk
Parent - - By yorkman (**) Date 2016-11-06 16:06 Edited 2016-11-06 16:44
Has to be a scam. They're using Stockfish running on multiple computers and the only way I know how to do combine multiple pc's' to get increased speed using IA is if you cluster them. Since there's no cluster version of Stockfish then this is not possible.

In the video I saw he's just using multiple copies of Stockfish from several remote computers. Sure, you can always do that in IDEA but not in IA. Not to get a combined speed. His video even shows his Xeon e5-2670 in IA and he's only getting about 26,000 kN/s. If it was truly a cluster with a clustered version of Stockfish running on several remote computers than that speed would be MUCH higher.

Hell, I can split my 72 core machine into 72 engines running on one core and I'd also be able to say I can run 72 engines in idea...and yes, you can do this in IA as well but they'd be "separate" engines. It wouldn't be running as ONE engine at 80,000 kN/s or whatever.
Parent - - By Merkury (*) [ru] Date 2016-11-06 17:51
Our project is not cluster in the classic sense. Many computers together in IDeA for Aquarium.
In this video there is the launch of 80 engines (each engine about 1,000 k/Ns in the starting position)
and start the server Dual Intel Xeon E5-2670 (27,000 k/Ns in the starting position).
https://www.youtube.com/watch?v=xqRWhq7YQHk&t=2s

We are not scams. You can see this if you are interested in a large number of engines in IDeA for Aquarium.
It is also possible to rent server for IA analysis.
Connect to the server, you can programs: Aquarium, Chessbase or Fritz.

P/S
We do not claim that all of our 80 engines operate as a single engine.
11 computers used to launch 80 engines, including 2 server Dual Intel Xeon E5-2670.
You're right, our project is not a cluster. We just use the word for convenience.
http://superchess.wixsite.com/chesscomp/--2-c8gk
http://superchess.wixsite.com/chesscomp/--3
Parent - By yorkman (**) Date 2016-11-08 02:12
Ok, I understand but I never did find the benefit of using so many one-core engines all at once since I only use IA.
- By Merkury (*) [ru] Date 2016-11-10 13:02
Our supercomputer continues work.
We replaced the engine in the IDeA on Stockfish 8, and to use the accelerated IA 100% clone Stockfish 8 on assembler - Pedantfish. On our server Dual Xeon E5-2670 speed of 27400 k / Ns in the starting position.
Now we continue to work on improving the engines for IDeA. Now all engines received the same maximum hash - 512mb, and the engines are aligned for speed, which makes the job more balanced supercomputer.
Updated block diagram of supercomputer.
https://yadi.sk/i/wUqG6jGEyJeSr

http://superchess.wixsite.com/chesscomp/--2-c8gk
- By Merkury (*) [ru] Date 2016-11-20 14:07
A significant portion of our computers was busy and we provide 50 engines for the IDeA, now you can see exactly what kind of computers and engines will be received in rent. Two servers will be purchased by the spring, and we will expand our offer.
Updated block diagram supercomputer.
https://yadi.sk/i/SP3Pw3aAz4qmZ
- By Merkury (*) [ru] Date 2017-03-07 05:10
Our supercomputer has become more compact and more powerful engines IDeA number has remained the same - 50 pieces, but significantly increased the speed of the engines, now each engine uses 1 physical core. Now - 50 cores (82 threads)!
Minimum speed engines from 1200 to 1400 k / Ns in the starting position, the maximum speed of from 1450 to 1700 k / Ns in the starting position.

For IA (infinite analysis) available servers Dual Xeon E5-2670 3,0 GHz 16 cores, 32 threads in the starting position of 30000 kN / s. Hash engines 4096Mb. Connected to the engines 6-piece Syzygy bases.
​Attention! We have registered domain and now the address of our site http://www.superchess.org
English section http://www.superchess.org/--2-c8gk
Some time the site will be available at the old address.

Updated block diagram supercomputer.
https://yadi.sk/i/qDQasATL3F2Qj7
- By Merkury (*) [ru] Date 2017-03-11 13:09
There is no limit to perfection!
And we continue to improve our project.
Today, good news for those of our customers who use servers for IA (infinite analysis). Now they can reboot the engine independently with the help of the remote access program - Supremo. This provides 24/7 and 100% access to the engine without contacting technical support.
- By Merkury (*) [ru] Date 2017-03-24 03:37 Edited 2017-03-24 03:41
Our project continues to develop and improve.Attention! Users of 50 IDeA engines got the opportunity reboot engines. This provides 24-hour and full control over the engines.
https://yadi.sk/i/ZOrOajZq3GJ4tj
- - By Merkury (*) [ru] Date 2017-04-13 09:09
We continue to improve our project and now we provide a new service for those users who want to quickly get the finished result.

We offer a 24-hour automatic position analysis by a 16-core server, a hash engine is installed in 16 gigabytes, and 6-piece syzygy bases are connected.
You will get full results in text files, in pgn files, as well as screenshots of analysis. The results of the analysis will be recorded after approximately 1 hour, 3 hours, 12 hours and after 24 hours.

Tell me an interesting position and I'll post the results of the 24-hour analysis, so that you can visually evaluate the quality of the new service.

Our website is http://www.superchess.org/--2-c8gk
Parent - - By CumnorChessClub (***) [gb] Date 2017-04-13 16:54
How about the Kasparov game against Deep Blue where he resigned in what maybe was a draw? https://www.chess.com/forum/view/game-analysis/did-kasparov-resign-too-early-vs-deep-blue
Parent - - By Merkury (*) [ru] Date 2017-04-13 17:29
I have already analyzed this position on my supercomputer. The analysis showed that the position is objectively a draw. Here is a small pgn-file with analysis.

https://yadi.sk/d/4zUPZDE03Gwd85
Parent - - By CumnorChessClub (***) [gb] Date 2017-04-14 08:30
How about this one by Aronian in the opening 9.Be1!

[Event "Wijk aan Zee"]
[Site "?"]
[Date "2017.04.14"]
[Round "10"]
[White "Levon, Aronian"]
[Black "Richard, Rapport"]
[Result "*"]
[ECO "E18"]
[Annotator "user"]
[PlyCount "17"]

1. d4 Nf6 2. c4 e6 3. Nf3 b6 4. g3 Bb7 5. Bg2 Be7 6. O-O O-O 7. Nc3 Ne4 8. Bd2
Bf6 9. Be1 *

Parent - By user923005 (****) [us] Date 2017-09-08 18:22
From my database:
r1bq1rk1/pp1n1ppp/2p1p3/2Pp4/2nP4/P3BNP1/P3PPBP/2RQ1RK1 w - - acd 50; acs 14400; bm Bf4; ce 0; pm Bf4;
pv Bf4 h6 Qb3 Qa5 e4 Qxa3 Rfe1 b5 exd5 exd5 Bf1 Qa5 Bd6 Rd8 Re7 Rf8 Ree1;
r4r2/4nppk/1n1p3p/1p2p3/4P1P1/1PP3N1/3N1PP1/R3R1K1 w - - acd 54; acs 14400; bm Nf5; ce 1; pm Nf5;
pv Nf5 Nxf5 exf5 Rfd8 f3 f6 Ne4 d5 Rxa8 Nxa8 Nf2 Nc7 Nd3 Rd6 Kf2 Rc6 Rc1 Ra6 Nb4 Ra3 Rb1 Kg8 Rd1 Rxb3 Nxd5 Nxd5 Rxd5 Kf7 Rd7+ Kg8 Rc7 Rb2+ Kg3 b4 cxb4 Rxb4 Kh4 Rb2 Kh3 Kh7 Kg3 Rd2 Rc5 Rd3 Rb5 Rd4 Rb1 Rc4 Rd1 Rc3 Kh4 Rb3 Rd6 Rb2 Kg3 Re2 Re6 Rb2 Re8;
rn1q1rk1/pbpp1ppp/1p2pb2/8/2PPn3/2N2NP1/PP2PPBP/R2QBRK1 b - - acd 48; acs 14400; bm Nxc3; cce -102; ce -17;
pm d5 {8} Na6 {2} Re8 {2} Nd6 {1} Nxc3 {1} c5 {1}; pv Nxc3 Bxc3 Be4 Ne5 Bxg2 Kxg2 c5 Rc1 d6 Nf3 Nd7 e4 Qc7 b3 Rfe8 Re1 Qb7 dxc5 dxc5 Bxf6 Nxf6 e5 Rad8 Qe2 Ng4 Red1 Nh6 Kg1 Qe7 Rc3 Rxd1+ Qxd1 Rd8 Rd3 Rxd3 Qxd3 Nf5 h3 h5 Kf1 g6 Kg1 Kg7 Nd2 Qb7 Ne4 h4 g4 Nd4 Qe3 Qe7; white_wins 7; black_wins 4; draws 2;
rn1qk2r/pp2bpp1/2p1pn1p/3p1b2/2P5/1P1P1NP1/PB1NPPBP/R2QK2R b KQkq - acd 43; acs 14400; bm O-O; cce -15; ce 17;
pm O-O {38}; pv O-O; white_wins 7; black_wins 6; draws 16;
- By Merkury (*) [ru] Date 2017-04-17 02:28
I chose 2 positions and made a 24-hour automatic analysis using a 16-core server. Here are the results of the analysis.

https://yadi.sk/d/vx3VEc5l3Gya9y

https://yadi.sk/d/bBlc1ujE3H2xQ9
- - By Merkury (*) [ru] Date 2017-09-06 15:42
Our project is undergoing a major reorganization.
Only powerful servers remained in the project, all slow computers were excluded from the project.
We increased the number of IDeA engines to 92 pieces. Now engines for IDeA can be rented by separate servers (from 16 engines minimum and up to 92 engines maximum).
For rent is available a powerful 28-core (56 threads) Dual Xeon E5-2683v3 server - 45000 kN / s in the starting position on AsmFish. We connect to the programs Chessbase and the Aquarium.

On our website you will find detailed information.
http://www.superchess.org/--2-c8gk
http://www.superchess.org/--3

Prices for server rental are very affordable, hurry to use an available server.
Parent - - By Ghengis-Kann (***) [us] Date 2017-09-06 17:44 Edited 2017-09-06 18:01
For those who think this might be a scam, I don't think so.
You can build this thing that Merkury is describing for about $5,000.

There are very affordable aftermarket Xeon processors available that I used to build a 16 core (32 thread) Xeon server for about $1,500.
This computer can provide up to 32 engines for IDEA or run as a single engine.
It runs about 32,000 KN/s in infinite analysis, so Merkury's claim of 45,000 KN/s off a 28 core machine seems perfectly reasonable.

Hi Merkury!
I have a couple of questions for you.
Pawnslinger pointed out that Stockfish occasionally produces bizarre evaluations when used with IDEA in a thread on this forum entitled "Which Engine for Idea?"
I have also experienced this problem with Stockfish, but not with Komodo or Houdini.
Have you seen this happen and if so do you know why or how to prevent it?

Another issue I have seen with Houdini5 Pro is that the remote computers start generating "Memory Access Error" pop up boxes and occasionally crash completely when the remote engine count is high or the analysis time short. It is not entirely clear if this is specific to Houdini since I haven't tried Komodo on this network yet.

I run 40 remote engines on a network with the Xeon box described above and 2 other 4 core PCs.
This error occurs on all of them in approximate proportion to the number of engines, but never on the computer that is running Aquarium.
It seems reasonably stable using (10 seconds AND 24 plies, wait for next depth, max time 900 seconds), but the errors still pop up now and then.
If I reduce the depth so more positions per second are analyzed the errors occur more frequently and IDEA will start dropping engines.

Have you seen these errors, and if so do you have any way of preventing them?

Thanks,
Ghengis-Kann
Parent - - By pawnslinger (***) [us] Date 2017-09-07 00:59
I have not had the problem you mention, but I did notice something interesting when I upgraded my system from a 6yr old Intel chip to a new Ryzen chip... more cores and much faster cores.  Typically I had to increase the depth of analysis from 0 seconds and depth 18 plies to 24 or more plies, depending on the position.  On the old Intel box I usually ran 6 engines and a depth of less than 20 plies, now I run 14 engines and depth usually 24 or greater.

The problem I was having with the old settings on the new system was that the evals were being done too quickly and over-running the ability of my hard drives -- I did not upgrade my hard drives, so they are still slower than I would like.  However since they are sata3 drives, and my new system is sata3 (old system was sata2), the drives do appear to be somewhat faster.  I wish I had the money to get a large fast SSD.  I could only afford a 256gb SSD for my boot drive on the new system... not to be used for anything except boot and a 6-man tablebase.
Parent - - By Ghengis-Kann (***) [us] Date 2017-09-07 03:23
Hi Pawnslinger.

Maybe I misunderstood your post.

What did you mean by this:
(pasted from the thread I mentioned)

- - By pawnslinger (***) [us] Date 2017-06-05 04:44
Yes, but at least in the case of Stockfish, there are bogus evals generated that way also.  For some reason Stockfish sends +/-300 as a bogus eval.  It later corrects it, but Aquarium does not properly record it, until a minimax overwrites the bad eval... which can sometimes take a LONG TIME.

So I stopped doing it and requested a bug fix quite awhile ago -- which I think was ignored.
Parent - - By pawnslinger (***) [us] Date 2017-09-07 03:45
Actually, Stockfish is not the only engine that I have seen do this.  This occurs when using the Sandbox (or Infinite Analysis) to send the evals directly into the IDeA tree.  The bogus eval seems to be something like a temporary eval that Stockfish would do just as well to not send... because it usually corrects it with the complete eval a little later.  However, IDeA does not pickup the new correct eval until later, after a minimax overwrites the thing.  In the meanwhile, of course, this outrageous eval is screwing up the tree.

So I still use Stockfish, but just send the lines to IDeA for eval there... and the bogus evals never happen during IDeA evaluation... only in Sandbox or Infinite Analysis.
Parent - By Ghengis-Kann (***) [us] Date 2017-09-07 12:32
Ok, thanks.
I don't directly use the IA evals and just use it to seed IDEA, so it isn't really a problem for me then.
Parent - - By Merkury (*) [ru] Date 2017-09-07 10:01
Hi, Ghengis-Kann!

In order for remote engines to work well, the following conditions are necessary:

1. Good internet.
2. Use the Aquarium 2016 or Aquarium 2017.
3. Correct settings of the ini-file, priority is required - low! =0
4. Good performance of computers with a minimum of running processes.
5. Correct work with IDeA - analyze no more than 3 projects at a time, a minimum analysis time of 30 seconds, do not add a huge number of red tasks simultaneously.

Our remote engines work very well, when I analyze the positions myself, then 98-100% of everything works. Customers can reboot engines, in case of failure. The engine restarts for 5 seconds.

The engine for infinite analysis works very well, we can connect to the Aquarium or ChessBase or Fritz. Customers can reboot engine, in case of failure. The engine restarts for 5 seconds.The speed of the engine on the 28-core server is very good and the price is 2-3 times lower than in the cloud of Chessbase.

Our internet is very, very good!
Parent - - By Ghengis-Kann (***) [us] Date 2017-09-07 12:43
Good luck with your commercial endeavor.

Your advice matches my own experience that these problems are caused by sending IDEA results faster than it can process them.

I am currently re-balancing my remote engines so the slowest computer runs as a single engine on all cores and the faster computers run a number of threads on each engine that produce approximately the same KN/s as the slowest computer on all cores.

The point of this is to switch from depth-limited to time-limited search.
I do a lot of whole-game analysis and there is a problem where the engines will sit there doing nothing until max time is reached if you ask them to calculate to a depth that goes beyond a forced mate. Time-limited search should also enable me to more easily optimize the number of positions per second I can analyze without causing problems with Aquarium.
Parent - - By pawnslinger (***) [us] Date 2017-09-07 16:41
This problem you point out about depth limited evaluations is quite accurate.  I have seen it also... especially in Houdini.  I reported this bug to Bob Houdart (I think that is his name), but he gave me a bunch of double talk to (a) explain it away, or (b) blame it on the GUI (Aquarium in this case).  Since I found it most severe in Houdini, I stopped using it... pretty much.  Occasionally, I use it in the Sandbox, but not often.  I have been paying for Houdini upgrades, but will probably stop.

I find with time limited evals the search done by IDeA is too uneven... so with Stockfish (my main IDeA engine) I use 0 minimum (of course) and a maximum that is not toooo high.  Usually 240 to 300 seconds.  I am willing to waste up to that much time on a poor move if IDeA gets stuck on it.  But I find that Stockfish and IDeA combo don't run into this issue nearly as often as Houdini does.

As I go thru a game, I find that I must increase the depth limit to keep the rate of evaluation well regulated (i.e. not too fast) and in the end game I increase the max time considerably... in end games I am willing to spend more time chasing poor variations - odd ball looking moves in the end game are often the ones that can save a draw.  And of course, the depth limit in end games is much higher too.
Parent - By Ghengis-Kann (***) [us] Date 2017-09-07 17:50
I ran into another problem with Houdini that pissed me off enough to buy Komodo 11 and switch to it on all of my computers.

The license that came when I bought Aquarium 2017 with Houdini5 Pro was only good for installs on 4 computers, and at the time I was running 5 on my network.
I dropped an old laptop that was kind of pathetic anyway to get Houdini on all of the remaining computers.

We had an electrical storm a few weeks ago that fried the motherboard on my primary computer, and after I replaced it Houdini started producing insane evaluations like -13.06 in the initial position. I'm pretty sure what happened is that the license was tied to specific hardware and the program is designed to do this if it runs on hardware it is not licensed to run on.

Houdart will never see another dime of my money.
Komodo gives me a discount for upgrading to the newer versions and runs on as many computers as I want to put it on with no problems at all.
And of course, Stockfish is free.
- By Merkury (*) [ru] Date 2017-11-01 10:39
Our project continues its development. Now another 28-core (56 threads) Dual Intel Xeon E5-2683v3 server is available. Now the project has 6 servers - four 16-core and two 28-core.
The speed of AsmFish engines for IA (infinite analysis) is 32000 kN\s in the starting position on 16-core servers and 48000 kN\s in the starting position on 28-core servers. The engines for IA we connect to the programs Chessbase, Aquarium and Chess Assistant.
Our supercomputer for IDeA in Aquarium now has 120 engines, the total minimum speed in the initial position is over 150,000 kN\s.

On our website you will find detailed information.
http://www.superchess.org/--2-c8gk
http://www.superchess.org/--3
Up Topic Rybka Support & Discussion / Aquarium / Supercomputer 100 cores & Aquarium
1 2 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill