Not logged inRybka Chess Community Forum
Up Topic The Rybka Lounge / Computer Chess / Compiling Large Pages into abrok dev BMI2 SF
1 2 3 Previous Next  
- - By Dragon Mist (****) Date 2015-06-22 00:17 Edited 2015-06-22 09:56
Subject says it all. I am a CC player and could benefit from LP, but insist on using only latest SF dev versions from abrok.eu site, no other modification, no IPMAN, no MZ, no Valentine, no nothing else. I have no idea about compiling, but tried to learn something since SF went open. I failed miserably. Words like github, fork, src, ... mean very little to me, i.e. I have only a vague idea what they might represent. However, reading many posts here and on comments page of abrok.eu, I got the feeling that compiling SF myself could not be that difficult of a task. So I decided to try one last time and learn how to do it myself, and am asking here for some help in accomplishing this task.
From what I know, the process goes something like this:

1. each time there is a new abrok.eu compile available (meaning there is a new dev version available, meaning there is a new src available, whatever that means), I am supposed to download certain bunch of files (link, please!).

2. those files are supposed to be saved in certain folder on my desktop, and also there are some rules for naming that folder (what rules, why desktop and not arbitrary new folder on disk, ...).

3. next, one should compile those files (ok, here things start to complicate, what compiler, where do I get one, what compiler is best i.e. produces fastest executables, how to compile for BMI2, how to compile with Large Pages support - there are also other question but are of secondary importance, like, why/how is SF 6 faster than its dev counterpart, what compiler was used for SF 6 that is not used for regular abrok.eu compiles, and why are abrok.eu compiles deliberately compiled with supposedly slower compiler??)

4. now one should fire up the exe produced by step 3 (fire up in dos mode or within chess GUI, for how long, the longer the better?)

5. in the end comes recompile (or last compile, or whatever, now that first compile has been seen in action)

So you see, a couple of answers required, with a grain of patience, and I could be on my way. Thanks to anyone who would help in advance.
Parent - By Vegan (****) Date 2015-06-22 02:21
I use the free Arena, and I have played a few games recently but I am older now and spend more time reading than playing games generally.

That is why I run an open source tournament, then I can look at the play and see how good or bad a given engine is
Parent - - By jpqy (**) Date 2015-06-22 07:22
Hi Dragon,

Try to follow Stockfish support: Compiling Stockfish on Windows

http://support.stockfishchess.org/kb/advanced-topics/compiling-stockfish-on-windows

I have learn it from there..

JP.
Parent - - By Dragon Mist (****) Date 2015-06-22 09:15
Thank you jpgy. I know about that little tutorial, and am pretty sure I could follow the instructions and get the exe, but that does not address my goal. Compiling myself by following the instructions there I get nothing more than I already have on abrok.eu (much easier to just download new version when available).

In particular, it does not solve the following:

- large pages (basicaly the main reason I am trying to learn to compile myself)
- no choice of compiler (says just go with 4.9.2) for to get maximum speed
- does not mention bmi2 (but I guess it's just different switch in "make profile-build ARCH=x86-64-modern COMP=mingw" command, probably something else instead of modern)
- there is no mention of running the program after compiling it, for the sake of optimizing.
Parent - - By Sesse (****) Date 2015-06-22 10:45
Generally, choose the newest version of GCC. There's a lot of cargo-culting around, and any information older than a few months is more likely to be wrong than right.

It's x86-64-bmi2 instead of x86-64-modern.

profile-build already takes care of running Stockfish for you and then recompiling based on that data.

As for patching the source code, you're sort of on your own. :-)

/* Steinar */
Parent - By Dragon Mist (****) Date 2015-06-22 10:56

> Generally, choose the newest version of GCC. There's a lot of cargo-culting around, and any information older than a few months is more likely to be wrong than right.
>


That's what I noticed, too, even being a total noob. :lol:

> It's x86-64-bmi2 instead of x86-64-modern.


So I got that one right.

> profile-build already takes care of running Stockfish for you and then recompiling based on that data.
>


That's a new one to me. So, forget about 2 runs of exe, just single run of GCC.

> As for patching the source code, you're sort of on your own. :-)


Yeap. But I see people do that (include LP) in an instant, so kind of hope someone knowledgeable would come out and share with me.

Thanks Sesse, you've been very helpful!
Parent - - By jpqy (**) Date 2015-06-22 11:08
If you can make a compile..you need notepad++ for example to edit the source : https://notepad-plus-plus.org/
So,if i see a interesting patch i can include it myself and make a compile from it.

For adding LP ..here you have a link : https://github.com/Zerbinati/StockfishMZ/commit/d3fe676a904dc1c636fcbf7b1d78b1eb93c5e84d

Compiler i use and gives me faster compiles then abrok:
for 32 compile
x32-4.7.3-release-posix-sjlj-rev1.7z
http://sourceforge.net/projects/ming...ds-posix/sjlj/

for 64 compile
x64-4.7.3-release-posix-sjlj-rev1.7z
http://sourceforge.net/projects/ming...ds-posix/sjlj/

to make bmi2 compile use:
make profile-build ARCH=x86-64-bmi2 COMP=mingw
make -f MakeFile build ARCH=x86-64-bmi2 COMP=mingw

After compile you use : strip stockfish.exe
Then first rename your stockfish.exe to any name you want..i use the date like: Stockfish 220615 x64 bmi2
now you can clean again and type : make clean
Your Stockfish is ready to put in any folder ,like engine folder ..start your Gui and install as new engine
Done.

JP.
Parent - - By Dragon Mist (****) Date 2015-06-22 11:35
Great! I see a lot of questions still unanswered (well, unclear to me, anyway :eek:), but seems like we're getting there. I shall try the procedure and post it in a separate post later, with all the dilemmas and problems that might arise. Just one question now: what is the purpose of this second run,

> make -f MakeFile build ARCH=x86-64-bmi2 COMP=mingw


that is, shouldn't just

> make profile-build ARCH=x86-64-bmi2 COMP=mingw


be enough, as per tutorial and Sesse post?
Parent - - By jpqy (**) Date 2015-06-22 12:48
You have to choose one line for compiling
With these two lines you get two compiles
The line with profile-build should give you a little faster one..that's it.
Use them seperate..i should give you only one line..and not to confuse you..

JP.
Parent - By Dragon Mist (****) Date 2015-06-22 13:19
No problem and no confusion, thank you for explaining. So, I should use only one of the 2, and preferably the one with profile-build (my goal is to get the fastes exe possible). Thanks jpqy.
Parent - - By Dragon Mist (****) Date 2015-06-22 16:01
Horrors have begun. While

> http://support.stockfishchess.org/kb/advanced-topics/compiling-stockfish-on-windows


simply tells you to "install MinGW (32-bit)", it does not tell you that the process of installation is far from user friendly, that there are steps where you are asked question beyond your comprehension, and most frustrating, at some point you have to choose which of the "packages" (whatever that is) you want to install. It would be an easy choice (All) if there are not like hundreds of packages, and you have to select each and every one manualy! This is the first point where I am close to give up. Can anyone explain or make it easier. I have opted for Basic Setup, and selected all packages (couse there are only 7, had I gone for all packages I would have to click myself to death).
Parent - - By Dragon Mist (****) Date 2015-06-22 16:29
The next step is even more fun. Tutorial says to download  MinGW (64-bit), GCC 4.9.2.. After you do that, you get x86_64-4.9.2-release-posix-seh-rt_v3-rev1.7z file in your download folder. Note the .7z extension. When you try to extract it with 7-Zip, it refuses to do so, claiming some compression method is not supported. However, if you use say WinRar, it does the job without complaints. On a more personal note, this extraction was the longest in the history of the universe. I am afraid to check free space on my C disk. But at least I have now folder structure as it should be, so let's say I'm okay for now. :grin:
Parent - - By Dragon Mist (****) Date 2015-06-22 16:40
Ok, installed Notepad++, downloaded latest src, now I'm supposed to modify several files just like it was done here https://github.com/Zerbinati/StockfishMZ/commit/d3fe676a904dc1c636fcbf7b1d78b1eb93c5e84d?diff=unified.
Fingers crossed.
Parent - - By Amos (**) Date 2015-06-22 18:34
Hi
http://www.mingw.org/wiki/getting_started
You must install MinGW otherwise not working properly
and you have to set the environment variable day
example:
%Path%;C:\MinGW\mingw64\bin;C:\MinGW\mingw32\bin;C:\MinGW\msys\1.0\bin

For compile new c++11
Download:
For 32 compile
x32-4.7.3-release-posix-sjlj-rev1.7z
http://sourceforge.net/projects/mingwbuilds/files/host-windows/releases/4.7.3/32-bit/threads-posix/sjlj/
Install MSYS
http://www.mingw.org/wiki/msys
Download MSYS 1.0.11
http://downloads.sourceforge.net/mingw/MSYS-1.0.11.exe
For 64 compile
x64-4.7.3-release-posix-sjlj-rev1.7z
http://sourceforge.net/projects/mingwbuilds/files/host-windows/releases/4.7.3/64-bit/threads-posix/sjlj/

MinGW command: (mingw32-make or make)
mingw32-make -f MakeFile profile-build ARCH=x86-64-bmi2 COMP=mingw
mingw32-make -f MakeFile profile-build ARCH=x86-64-modern COMP=mingw
mingw32-make -f MakeFile profile-build ARCH=x86-64 COMP=mingw
mingw32-make -f MakeFile profile-build ARCH=general-64 COMP=mingw

mingw32-make -f MakeFile profile-build ARCH=x86-32 COMP=mingw (sse)
mingw32-make -f MakeFile profile-build ARCH=x86-32-old COMP=mingw
mingw32-make -f MakeFile profile-build ARCH=general-32 COMP=mingw

strip stockfish.exe
rename engine (important before launching the command "clean")

mingw32-make clean or make clean
__________________
Besides notepad need to download the plugin COMPARE
Notepad++ Compare plugin
http://sourceforge.net/projects/npp-compare/

For add Large Pages
change:
main.cpp
Makefile
types.cpp
tt.cpp
tt.h
add special file "opt.cpp"
Add the code in green for each file listed
look here
https://github.com/Zerbinati/StockfishMZ/commit/47621af00cf50cc9a55911ce3b8ac3fcb2dc33eb
Parent - - By Dragon Mist (****) Date 2015-06-22 18:45
Many thanks Amos, seems I got it right already.

1. installed MingGW
2. Decompressed MinGW 64bit 4.9.2 into MinGW folder
3. set environment variable Path
4. changed all files as per Zerbinati instructions
5. added Zerbinati opt.cpp
6. executed make profile-build ARCH=x86-64-bmi2 COMP=mingw
7. strip stockfish.exe
8. rename SF exe
9. make clean
10. run uci and bench

It all worked fine. Thanks for your effort.
Parent - - By APassionforCriminalJustice (***) Date 2015-07-13 05:59
Dragon, what are your game results showing with LP versus no LP? Yes I know that a speedup is present, but do you see any ELO gain. I just made an LP compile; I would like to know what you are seeing thus far.
Parent - By Dragon Mist (****) Date 2015-07-13 18:27 Edited 2015-07-13 18:32
I have no test results, as I do not do engine-engine tests. I use engine for analysis only.

BTW (I've always wondered) how do you test an engine with LP?? Multiple games means multiple load-unload, which inevitably leads to loss of LP functionality. So a couple of games you have it, the rest you don't, results skewed. Right?
Parent - - By APassionforCriminalJustice (***) Date 2015-07-13 05:57
Amos, could you possibly give the code that you use to enable AVX for Stockfish? I would imagine that these are only changes made in the makefile?
Parent - - By Amos (**) Date 2015-07-13 06:48
Hi, for AVX compile use command line: "mingw32-make -f MakeFile profile-build ARCH=x86-64-modern COMP=mingw" and change:
misc.cpp

types.h

Makefile
Parent - - By APassionforCriminalJustice (***) Date 2015-07-13 09:57
Thank you Amos... I appreciate the code. Will "make profile-build ARCH=x86-64-modern COMP=mingw" not work to compile AVX? I use msys on windows not the mingw32-make... would appreciate your answer if you have one.... wonder if "make" would work.
Parent - By Amos (**) Date 2015-07-13 11:24
"mingw32-make" or "make" it depends on what you use

mingw32-make -f MakeFile profile-build ARCH=x86-64-modern COMP=mingw
or
make -f MakeFile profile-build ARCH=x86-64-modern COMP=mingw
Parent - - By Sesse (****) Date 2015-07-13 13:08
Note that the effect of compiling with -mavx will be pretty near nil, since there's no actual AVX-using code in Stockfish (so all you get is the three-operand version of some very few instructions). And the USE_AVX #define (which I suppose is there just to output “AVX” in the version string?) is pretty redundant compared to just checking for __AVX__, which is set by the compiler.

/* Steinar */
Parent - - By Amos (**) Date 2015-07-13 13:38 Edited 2015-07-13 13:56
Hi Sesse
-mavx + DUSE_AVX for avx Version
string misc.cpp and types.h for infostring
I do not find correct?
suggestions?
perhaps ..

-march=native (It would seem better)

Parent - - By Sesse (****) Date 2015-07-13 13:45
I don't understand what you're trying to say.

/* Steinar */
Parent - - By Amos (**) Date 2015-07-13 13:58
I added -march = native
Parent - - By Sesse (****) Date 2015-07-13 14:13
That will indeed imply -mavx, if you are compiling on a machine with AVX.

You won't see much speedup from Stockfish, though.

/* Steinar */
Parent - - By Amos (**) Date 2015-07-13 15:51
exactly .. just 0.8% :roll:

Parent - - By Sesse (****) Date 2015-07-13 21:05 Edited 2015-07-13 21:14
Note the “p-value: 1”. Either FishBench has no idea of statistics, or your test is worthless.

Edit: I looked up the source code. The former is the case (it blindly does a one-sided z-test where a two-sided t-test should be done, and generally messes up things). In any case, with so few runs it would be very hard to claim a value as precise as 0.8%; there's a lot of noise. It could easily be anywhere between, say, 0.1% and 1.7%.

/* Steinar */
Parent - By Amos (**) Date 2015-07-13 21:16
I agree with you
Parent - - By Sesse (****) Date 2015-06-22 16:42
Generally go with the default packages. Stockfish doesn't require too much.

(On Linux, generally you can install the compiler with a single line ;-) Nice with a user-friendly operating system like Windows, no?)

/* Steinar */
Parent - - By Dragon Mist (****) Date 2015-06-22 16:54
:lol:

Stay with me, please, problems are arising left and right. It occured to me, what if the original src files that requires change for LP as given here on Zerbinati were modified by other pathces in the meantime? That seems to be the case in the very fist line I was going to change. Here, a small little psqt is extra?? I'll just add opt.o at the begining of the line and hope it's ok.
Parent - By Dragon Mist (****) Date 2015-06-22 17:13
File opt.cpp doesn't seem to exist on src original. Of course (slap in the forehead), it is entirely for Large Pages! So I have to create new file and ... oh, wait, why not download Zerbinati src and take opt.cpp from there and paste in into SF original src? Yes, I am that smart! :twisted:
Parent - - By Dragon Mist (****) Date 2015-06-22 17:20
I think I figured this one out, this is just a list of files that will be used, so no problem.
Parent - By Dragon Mist (****) Date 2015-06-22 17:44
Done! Made exe, stripped, cleaned, now let's fire it up and see if it actualy works!
- - By Dragon Mist (****) Date 2015-06-22 18:06 Upvotes 1
Amazingly, it works!!!! Like a charm, and with speed up beyond any of my expectation. Yesterday I compared regular SF dev on my machine with 3m 30s from starting position infinite and it produced 7,100 kN/s. I compared it with LP dev SF kindly provided by troy (but AVX version, not bmi2) and it did 7,948 kN/s. Now, compile I just created following the instructions on stockfish.org and with some help here, does 8,306 kN/s!!! What bothers me is the size of exe, it is only 888 kb which is the smallest I've seen (even heavily optimized SF6 is around 905 kb). But it seems to work! Is this true, I mean, could it be I actualy didn't mess up something?? If so, next step is to purchase water cooling, as it did lit up my cores to 74 C.
Parent - - By jpqy (**) Date 2015-06-22 18:18
It looks you did it right!
I'm surpriced after you adding LP you didn't get errors..and it even makes you a compile..
So,congratulations..else for sure it would not works :wink:

In notepad++ you have a function under Plugins->Compare->Compare
When you have latest source and tomorrow comes a new out..and you open in both sources the same file..then click on Compare
and he will show you directly the differences in these lines..and you can copy and paste the needed lines..
when nothing is changed he will tell you that files are same.
Same you can do with the patches who are still running..

Enjoy your compilings..

JP.
Parent - - By Dragon Mist (****) Date 2015-06-22 18:37
Thank you jpqy, and thanks for the help. I too, can hardly believe this works. I expected errors all around, but it's just that none occured. Starting it in dos mode, typing uci and then bench, it run just fine and ended tests. I installed it in GUI and again, all workind exceptionaly well, checked a couple of positions all range (opening, middle, endgame) and the speed up is always around 16-18% compared to normal SF dev non LP. I understand I shall have to learn more about Notepad++ for easier usage, and watch out for the changes for the new builds, but, so far so good.
Parent - - By Amos (**) Date 2015-06-22 18:43
Drago Mist
follow my guide I wrote above
and impossible that you've compiled version AVX
send me the source code that you check it
AVX version for you to add support in the Makefile
To you it appears on the motor because AVX and inserted the string
in misc.cpp
Parent - - By Dragon Mist (****) Date 2015-06-22 18:47
Not sure I understand you, I compiled bmi2 and not avx version, since I have Haswell. I'm sending you exe in PM.
Parent - - By Amos (**) Date 2015-06-22 18:53
Ah ok..
use 4.7.3 is faster than any other release! :wink:
Parent - - By Dragon Mist (****) Date 2015-06-22 18:57 Edited 2015-06-22 19:52
In fact, here's link to my compile for anyone to see if it works

https://www.sendspace.com/file/777eva

Would like to hear some feedback.

As for 4.7.3., now that I've downloaded 4.9.2, how do I switch to 4.7.3?

Src here:

https://www.sendspace.com/file/cqyh16
Parent - - By Sesse (****) Date 2015-06-22 19:28
Do note that Stockfish's license (GNU GPL) requires you to include source code when you give out compiles. This is even more true in your case, since you're compiling from modified source code (so it is not readily available from somewhere else).

/* Steinar */
Parent - - By Dragon Mist (****) Date 2015-06-22 19:32
I have no intention to publish whatever I do, and exactly for the reason you mentioned (afraid of copyrights and stuff), this was one time only. In any case, that means if I make compile public, I must also include src files?
Parent - - By Sesse (****) Date 2015-06-22 19:47
Yes. And you just made the compile public, and at that point, there's no turning back. Source has to be there, with no delay.

(Of course, you may continue to modify it on your own and then don't publish _those_ changes. Until you publish a compile again.)

/* Steinar */
Parent - - By Dragon Mist (****) Date 2015-06-22 19:53
There, I edited the original post with link to a src also. Is this now ok?
Parent - - By Sesse (****) Date 2015-06-22 20:09
Looks good to me.

(The easiest is to just put both in the same archive, so it's easier for the next guy distributing to keep up the license, but you're in compliance the way you did it, too.)

/* Steinar */
Parent - By Dragon Mist (****) Date 2015-06-22 20:10
Thanks for the advice (I do not intend to publish anything anymore, anyway). I suck at this legal stuff. :red:
- - By Amos (**) Date 2015-06-22 18:54
you have to send me SRC if you want control you entered everything ..
not executable
Parent - - By Dragon Mist (****) Date 2015-06-22 18:58
Parent - - By Amos (**) Date 2015-06-22 19:09
Very good compliment!
Everything perfect the first time .. :wink:
Up Topic The Rybka Lounge / Computer Chess / Compiling Large Pages into abrok dev BMI2 SF
1 2 3 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill