Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Aquarium / Stockfish 12
- - By pawnslinger (****) Date 2020-09-04 22:08
I just downloaded Stockfish 12.  They said it came with a NNUE file, but I can't find it.  Is it built-into the exe??  Also, the AVX2 package (zip) didn't have any exe file in it.  I assume the file in there was an exe file sans the proper exe extension?!  It looked like it was, but just seems a bit strange.
Parent - By pawnslinger (****) Date 2020-09-04 22:16
Okay, I figured out the missing exe issue... silly me, I downloaded the Linux version by mistake... still the NNUE file missing is a confusion...  I am assuming that they just built it into the exe.  Anybody know for sure?
Parent - - By mattchess (**) Date 2020-09-05 00:01
The default NN is built into the executable so you don't need the file.  But if you want to use an alternative NN you can set the path to that file.

https://tests.stockfishchess.org/nns

I was confused at first and ended up putting the default net in the folder before I realized I did not need to do that.
Parent - By pawnslinger (****) Date 2020-09-05 01:03
Okay, thanks!!  I will take a look at the other NNs.  Nice to know I was not the only one confused.  <grin>
Parent - - By pawnslinger (****) Date 2020-09-05 19:03 Edited 2020-09-05 19:16
This is the first time that I have used the NNUE version of Stockfish.   I did not use any of the dev versions.

My impression is that it is slower running, on my setup I get around 22,000 kNps.  With prior version, Stockfish 11, I would get around 33,000 kNps.

However, the analysis results track very closely with the prior version.  That is, given a set amount of time, say 10 minutes, the 2 versions usually arrive at the same candidate moves for the 1st 3 ply (the number of ply visible in CB's cloud... at least on my system).

So I am a bit skeptical of the SF devs claiming 130 elo improvement.  Perhaps if I ran engine v engine contests the elo improvement would show true.  But in my work, it so far has produced nothing earth shaking. 

In fact it reminds me of Rybka when the dev introduced Monte Carlo simulation.  Slower, but very similar results.
Parent - - By pawnslinger (****) Date 2020-09-05 19:11 Edited 2020-09-05 19:13
Here is an example of what I mean... screenshot from some of my current work, showing SF 12 analysis compared with CB's cloud results showing Fritz 15 and SF 9...
Parent - - By Ghengis-Kann (***) Date 2020-09-05 19:38
Maybe that is the "truth of the position".

Try it on some Tabiya position in the English where there are 17 playable moves.
Parent - - By pawnslinger (****) Date 2020-09-05 20:07
In my Aquarium work, I do not use current versions of any engine.

I am using SF 9 as my primary engine in Aquarium.  Changing engines is very disruptive to my long existing trees.

And I am really only using Aquarium for very niche analyzes these days.  Things like Mar Del Plata.  Aquarium has given me too many disappointments over the years and the devs don't seem interested in supporting it any more.  Features I need have been scoffed at or just ignored.

So, while I still use Aquarium, it has become a very niche and special purpose tool.  I have several Mar Del Plata's underway at the moment, and they all get the AQ treatment, until late middle-game.  Then I switch to CB.  Due to the advantage there of efficient TB usage.

I have not used NN SF for very long, so you could be right... my opinion of it may improve with time.  But so far, in the handful of positions I have used it to analyze, the analysis I showed above is representative.  In fact, Fritz and SF often are very close in their evals.  Komodo does show up in the CB cloud sometimes, and it DOES have its own unique opinion on many positions.
Parent - By Ghengis-Kann (***) Date 2020-09-05 20:47
I crank the dynamism up on Komodo so it over values piece activity and use it to prepare fighting lines for OTB.
Definitely weaker versus other engines but I don't care because I am not preparing to play against Stockfish.

My next plan is to create versions of Stockfish and Lc0 that also over-value piece activity.
Should be pretty straightforward with SF but I am just getting started on figuring out the Github thing.

Leela will require training a custom network from scratch, which should be interesting but will surely present greater challenges.
Parent - By pawnslinger (****) Date 2020-09-06 06:21
Another observation... the new AVX version runs MUCH hotter on my system.   And runs faster too... So a trade-off, heat for speed.  And the Task Manager reports about 6 Percent greater CPU utilization under my loads... which makes sense.  I guess the AVX instructions are more efficient, but burn more energy.
Parent - - By Uly (Gold) Date 2020-09-05 21:58 Edited 2020-09-05 22:03

> So I am a bit skeptical of the SF devs claiming 130 elo improvement.


Oh, it's real, it's very real. I saw it gradually get there, and it was amazing to see 3 years of improvements in less than 2 months. I don't think I've ever seen something like this before, and it's possible it's the biggest jump ever (nothing will jump this high in this time frame ever again.)
Parent - - By pawnslinger (****) Date 2020-09-06 01:27
I come from Missouri... the "Show Me" state.   Heheheh...  Missouri is also famous for their stubborn mules... and I come from a long line of stubborn folks!

So I am skeptical, but open to being convinced (in other words, I am using it and I will see how it goes).
Parent - - By Uly (Gold) Date 2020-09-06 23:06
Fair enough, nextchessmove shows only an improvement of +80 elo (against Stockfish 7):

https://nextchessmove.com/dev-builds

Stills, it's a long way down the road to the chemist's.
Parent - By Leto (***) Date 2020-09-08 16:56
It's showing +326.01 +/- 4.87 over Stockfish 7 now.
Parent - - By Felix Kling (Gold) Date 2020-09-06 04:02
I tried some wild gambits, the new stockfish seems to be much better in defense/king safety evaluation. For example this one:
rnbq1rk1/ppp1bppp/5n2/3p4/3NPB2/2N5/PPPQ1PPP/R3KB1R w KQ -
Parent - - By pawnslinger (****) Date 2020-09-06 05:47
I bet the new SF is much better at these types of positions.  In my work though, I rarely, if ever, run across such positions.  Most of my games are less tactical and more positional.  I can't recall the last time I played anything more wild than an off-beat variation of the Sicilian.... something like the Grand Prix variation.  But that is really quite rare.  However, some folks have tried it on me... and I gave them the Chess equivalent of a "punch in the nose".  Not many folks have the guts to try that sort of opening in my correspondence games.
Parent - - By dickie (**) Date 2020-09-06 08:27
It’s the addition of some positional awareness provided by the net that is responsible for Stockfish 12’s leap in performance. Earlier versions of Stockfish were exceptionally fast searching tactical engines. If the developers are able to accommodate larger nets then there may be further big performance gains to come.

It will be interesting to see the effect on opening repertoires. While it’s possible some current lines may be refuted, I am hoping some more aggressive lines that we are hesitant to play at present may be proven to be perfectly viable. A lot of research and new trees to build this coming winter!
Parent - - By cma6 (****) Date 2020-09-06 17:37
SF-NN nets are about 20 MB. How does that compare with the size of the typical lc0 nets?
Parent - - By pawnslinger (****) Date 2020-09-06 20:35
I do not know the answer... but I think they are much larger.  I believe that SF-NN has A LOT of room to grow.  Many of us have very large RAM capacities!!  20mb is literally nothing in terms of today's RAM sizes.  Even small machines have 8gb of RAM and run very large video games.  So SF-NN is just at the very threshold of this journey.

In the future, we will notice that as core counts increase (i.e. CPU power), so too RAM capacity will have to likewise increase to use that power efficiently.
Parent - - By Felix Kling (Gold) Date 2020-09-06 21:34
It isn't about size (in RAM), but about the trade-off between speed and quality. Larger nets need much more calculation time.
What I find interesting is that the current NNUEs are trained on positions evaluated by the classical stockfish (right?). I guess the next step is to analyse these positions with the new NNUE engine, train the net on that data and repeat... or maybe even using lc0 evaluations!? Btw., does anyone know where to find the positions with evaluations used for creating the NNUE version?
Parent - By Uly (Gold) Date 2020-09-06 22:59

> What I find interesting is that the current NNUEs are trained on positions evaluated by the classical stockfish (right?)


I think that only happened at the beginning. The next nets were being based on "reinforced learning", so Stockfish NNUE was already being used to train future nets. For amazing results, a new net was released every 5 hours and was 2 elo stronger than the last one! (remember back when 10 elo would take a full year to get? We were getting that in a day!)

Unfortunately that "saturated" and more improvements have been from knowing when to use NNUE and when to use classical eval (the very first of such an idea brought +30 instant elo over what already was +90!)

People are predicting this is the end of classical eval, as NNUE can be trained over any classical eval and surpass it with a net. But I claim this is (still!) only the beginning, because, no matter what, pure NNUE can only run at half the speed of classical eval, nothing stops people from making a classical eval as strong as the best net, and it'll be stronger because of the speed.

There's like two camps right now, the ones that want to train something much stronger, and the ones that will improve classical eval so NNUE can be used on less positions for speed (I have faith in them, Stockfish remained over Leela mainly because of those classical eval improvements, those are smart people.)

Who can predict when will an engine be +100 elo over Stockfish 12? I have no idea, it broke all my scales...
Parent - - By Ghengis-Kann (***) Date 2020-09-06 23:00
Lc0's current tournament network is 134 MB.

It's called J92-130 and you can get it here:
https://github.com/jhorthos/lczero-training/wiki/Leela-Training
Parent - - By pawnslinger (****) Date 2020-09-07 00:57
I am sure I know the answer, but I have to wonder... have to ask...

Can NNs be transported?  Exchanged?  Kind of like a brain transplant.  I bet the answer is "No!".  But it would be great if there was a way to share, without having to go thru some long drawn out learning process.  After all the NN is built already, SF or Lc0 just needs someway to "import" the NN from the other.

Blue sky!!  That's what my old boss used to say about my "wonderings".  May he rest in peace.
Parent - - By Ghengis-Kann (***) Date 2020-09-07 16:23
Not at all.
They are completely different.

NNUE is used to augment the evaluation function of Stockfish, but it the rest of it is still an A/B engine.
With Leela the NN evaluates but also contributes to move choice through MCTS.

I asked about this on the Lc0 Discord and was told that the idea of interchanging the nets won't work because of the difference in the way CPU and GPU communicate to the CPU and RAM.
Parent - By pawnslinger (****) Date 2020-09-07 17:56
As I guessed.

However, some enterprising programmer could write an "import" app.... just daydreaming.  It is the beauty of open source.
Up Topic Rybka Support & Discussion / Aquarium / Stockfish 12

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill