Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / (Experimental) 64-bit Rybka on Linux
- - By Sesse (****) Date 2007-07-10 23:30
I couldn't resist tinkering a bit -- I'm a big fan of trying to do different things every now and then, so I took a break off the Perl coding I've been doing lately and decided to try to get 64-bit Rybka going on Linux. (WINE doesn't support 64-bit executables, and probably won't for quite a while due to ABI differences, so Linux users have been stuck with the 32-bit version, which is, well, slower.)

About 24 hours after the first line of code was written, there is now:

pannekake:~/dev/foo> ./rybkahack
Expected image base: 0x400000
Relative address of entry point: 0x16dc20
First section in RAM at 0x2b2daf1c31f0
Mapping up section: .text (address=0x401000, real_address=0x401000, fileptr=0x400, size=0x17f000, relocs=0)
Mapping up section: .rdata (address=0x580000, real_address=0x580000, fileptr=0x17f400, size=0x22ea00, relocs=0)
Mapping up section: .data (address=0x7af000, real_address=0x7af000, fileptr=0x3ade00, size=0x67400, relocs=0)
Mapping up section: .pdata (address=0x8d6000, real_address=0x8d6000, fileptr=0x415200, size=0xba00, relocs=0)
Mapping up section: .rsrc (address=0x8e2000, real_address=0x8e2000, fileptr=0x420c00, size=0x200, relocs=0)
imports at 0x7adf60, iat at 0x580000
import dll name 0x3ae43a (KERNEL32.dll), rva to thunk=0x180000
patching 0x180000 from 0x3ae248 to CreateFileA
patching 0x180008 from 0x3ae256 to MapViewOfFile
patching 0x180010 from 0x3ae266 to UnmapViewOfFile
patching 0x180028 from 0x3ae29e to GetLastError
patching 0x180038 from 0x3ae2ba to CreateFileMappingA
patching 0x180040 from 0x3ae2d0 to CloseHandle
patching 0x180048 from 0x3ae2de to GetTickCount
patching 0x180050 from 0x3ae2ee to Sleep
patching 0x180068 from 0x3ae31a to GetNumberOfConsoleInputEvents
patching 0x180070 from 0x3ae33a to GetCurrentProcess
patching 0x180078 from 0x3ae34e to FlushConsoleInputBuffer
patching 0x180080 from 0x3ae368 to WriteFile
patching 0x180088 from 0x3ae374 to GetCommandLineA
patching 0x180090 from 0x3ae386 to GetConsoleMode
patching 0x180098 from 0x3ae398 to SetConsoleMode
patching 0x1800a0 from 0x3ae3aa to CreateProcessA
patching 0x1800b0 from 0x3ae3d0 to GetStdHandle
patching 0x1800b8 from 0x3ae3e0 to GetSystemInfo
patching 0x1800c0 from 0x3ae3f0 to CreatePipe
patching 0x1800c8 from 0x3ae3fe to DuplicateHandle
patching 0x1800d0 from 0x3ae410 to GetCurrentProcessId
patching 0x1800e0 from 0x3ae448 to HeapFree
patching 0x1800e8 from 0x3ae454 to HeapAlloc
patching 0x1800f0 from 0x3ae460 to GetProcAddress
patching 0x1800f8 from 0x3ae472 to GetModuleHandleA
patching 0x180110 from 0x3ae4a2 to EnterCriticalSection
patching 0x180118 from 0x3ae4ba to LeaveCriticalSection
patching 0x180120 from 0x3ae4d2 to GetSystemTimeAsFileTime
patching 0x180128 from 0x3ae4ec to GetVersionExA
patching 0x180130 from 0x3ae4fc to GetProcessHeap
patching 0x180138 from 0x3ae50e to HeapSetInformation
patching 0x180140 from 0x3ae524 to HeapCreate
patching 0x180148 from 0x3ae532 to GetModuleFileNameA
patching 0x180150 from 0x3ae548 to GetCPInfo
patching 0x180158 from 0x3ae554 to GetACP
patching 0x180168 from 0x3ae56a to FlsGetValue
patching 0x180170 from 0x3ae578 to FlsSetValue
patching 0x180188 from 0x3ae59a to SetLastError
patching 0x180190 from 0x3ae5aa to GetCurrentThreadId
patching 0x180198 from 0x3ae5c0 to FlsAlloc
patching 0x1801a8 from 0x3ae5dc to WideCharToMultiByte
patching 0x1801d0 from 0x3ae648 to SetUnhandledExceptionFilter
patching 0x1801e8 from 0x3ae68e to ReadFile
patching 0x1801f0 from 0x3ae69a to SetHandleCount
patching 0x1801f8 from 0x3ae6ac to GetFileType
patching 0x180200 from 0x3ae6ba to GetStartupInfoA
patching 0x180220 from 0x3ae706 to InitializeCriticalSection
patching 0x180230 from 0x3ae732 to FreeEnvironmentStringsA
patching 0x180238 from 0x3ae74c to GetEnvironmentStrings
patching 0x180240 from 0x3ae764 to FreeEnvironmentStringsW
patching 0x180248 from 0x3ae77e to GetEnvironmentStringsW
patching 0x180250 from 0x3ae798 to QueryPerformanceCounter
patching 0x180278 from 0x3ae7f8 to HeapSize
id name Rybka 2.3.2a mp
id author Vasik Rajlich
option name Hash type spin min 2 max 4096 default 32
option name Max CPUs type spin min 1 max 2048 default 2048
option name Display PV Tips type check default false
option name CPU Usage type spin min 1 max 100 default 100
option name Win Percentage to Hash Usage type check default false
option name Display Current Move type check default true
option name NalimovPath type string default <empty>
option name NalimovCache type spin min 1 max 256 default 1
option name NalimovUsage type combo default Rarely var Frequently var Normally var Rarely var Never
option name Preserve Analysis type check default false
option name Clear Hash type button
option name Ponder type check default true
option name MultiPV type spin default 1 min 1 max 100
option name UCI_LimitStrength type check default false
option name UCI_Elo type spin default 1200 min 1200 max 2400
option name Server Buffer type check default false
option name UCI_AnalyseMode type check default false
option name UCI_Opponent type string default <empty>
option name UCI_EngineAbout type string default
option name Contempt type spin default 0 min -100 max 100
option name Outlook type combo default Neutral var Very Pessimistic var Slightly Pessimistic var Neutral var Slightly Optimistic var Very Optimistic var Ultra Optimistic
option name Rate Of Play type combo default Normal var Ultraslow var Slow var Normal var Fast var Ultrafast
option name Time Usage type combo default Varied var Constant var Varied
option name Emergency Time Buffer type combo default Medium var Small var Medium var Large
position startpos e2e4 e7e5
go depth 10
info depth 5
info depth 5 score cp 6 hashfull 0 time 6 nodes 500 nps 85333 pv b1c3 b8c6
info depth 5 time 8 nodes 779 nps 99712
info depth 6
info depth 6 score cp 18 hashfull 0 time 20 nodes 1891 nps 96819 pv b1c3 g8f6 d2d4
info depth 6 time 23 nodes 2279 nps 101465
info depth 7
info depth 7 score cp 17 hashfull 0 time 30 nodes 2904 nps 99123 pv b1c3 g8f6 d2d4 d7d6
info depth 7 time 38 nodes 4036 nps 108759
info depth 8
info depth 8 score cp 12 hashfull 0 time 81 nodes 8875 nps 112197 pv b1c3 g8f6 g1f3 b8c6 d2d4
info depth 8 time 102 nodes 11742 nps 117880
info depth 9
info depth 9 score cp 9 hashfull 0 time 161 nodes 17779 nps 113078 pv b1c3 g8f6 g1f3 b8c6 e2e3 e7e6
info depth 9 time 196 nodes 22725 nps 118726
info depth 10
info depth 10 score cp 13 hashfull 0 time 278 nodes 29843 nps 109925 pv b1c3 g8f6 g1f3 b8c6 e2e3 e7e6 f1c4
info depth 10 time 318 nodes 35615 nps 114684
info time 318 nodes 35615 nps 114684
bestmove b1c3 ponder g8f6

It's essentially a giant hack: It maps the Rybka executable into RAM at the offset it expects (0x400000 -- seems like it's not relocatable, which caused me quite a bit of headaches), replaces all the imports with its own stubs (all of which are written in assembler, since gcc can't compile code to the Win64 ABI), jumps to the entry point and hopes everything goes OK. (It also NOPs out a few code sequences...) It definitely won't work with anything but Rybka (and probably 2.3.2a only), and it's not very well-tested, but it actually works, and it is 60-70% faster than running the 32-bit version in WINE. :-)

It doesn't work with SMP (I haven't figured out how to do shared memory most Windows-like on Linux yet, from there it should be relatively easy), and it crashes on exit (no idea why; I haven't implemented ExitProcess yet, but that's not where it crashes). It probably also can't do tablebases yet, but I haven't tested.

In any case, it's not ready for general release yet (the code is a giant kludge, it needs bugfixing and it requires a hacked version of the linker to build), but I thought I'd share if there are other Linux-interested people out here. I guess I'll release a semi-working version shortly, assuming Vas doesn't mind. (In any case, it should be unneccessary to point out that such a beast would be totally unsupported!)

/* Steinar */
Parent - By thingummywut Date 2007-07-11 06:34
Wow... that's impressive. Keep us posted!
Parent - - By Felix Kling (Gold) Date 2007-07-11 09:02
What exactly did you do? Did you change the .exe file or did you write a script that makes it possible to run rybka x64?

If you changed the .exe file, sharing would of course be illegal, but you could offer a description of what you did so that every customer can do this.

Anyway, I thought Vas was working on different versions for Rybka, also a Linux version (since it should be easy to do), so I would suggest to wait a bit :) .
Parent - - By Sesse (****) Date 2007-07-11 10:31
The latter -- it's a wrapper (although I don't normally use the word "script" about something written in C and assembler :-) ). Once you have that, it's easy to change it into a single binary (just concatenate them together and have the wrapper read itself), but obviously, copyright rules distribution of such a beast out. You'll definitely need the regular Rybka binary to run it, and the wrapper contains no Rybka code (well, except for one instruction it needs to search for, which I'm quite sure comes from the Microsoft C runtime; in any case, a single assembler instruction is hardly copyrightable). Still, given that it runs in the same process space and messes with Rybka's view of the world, I'm not entirely comfortable releasing it unless Vas actually approves. :-)

Still no SMP support, by the way -- not only is the shared memory giving me headaches, but there's also something about the pipes the sub-Rybkas communicate over that eludes me still. There's tablebase support now, though:

pannekake:~/dev/foo> ./foo
setoption name NalimovPath value /srv/tablebase
setoption name NalimovUsage value Rarely
info string setting nalimov usage to rarely
position fen
5K2/3NB3/8/8/8/8/1k1p4/8 w - - 0 1

go depth 1
info depth 1 score mate -41 nodes 12 pv d7f6
info time 2 nodes 12 nps 6144 tbhits 68
bestmove d7f6 ponder d2d1q

/* Steinar */
Parent - - By gala.martin (**) Date 2007-07-11 14:21
All of this just looks great. It should be easy to build up a linux native rybka from source, and V.R. did not do that mainly to avoid post-selling issues (new bugs etc).

I did not properly understand what you are doing, but it would be really nice to have a downloadable script or something that could generate a rybka engine running in linux. I think that a script patching the .exe file should be legal (if you do not have the file, it is worthless), or at least it should be tolerated by the author (come on! You are not that strict, are you?). Using wine is not so painful as in the past, but you are forced to use windows chessGUIs and, as you pointed ou, 64bit is unsopprted.

So, if you work out something legal, I would be grateful if you can share it.

Parent - - By Sesse (****) Date 2007-07-11 14:59
Vas has stated that there will be a Linux version of Rybka, but his schedule is probably already quite packed. When it comes out, however, it will be a much better solution than this. This is basically a horrid, unsupported hack that will eat your hamster, crash when you're low on time and lure you into losing your queen at random ;-) It's more for the fun of it than for making a long-term usable solution. (I've got SMP support now, though -- it leaks the shared memory segment on exit, but it runs fine on both the dual and the quad I've been able to test it at. The main unsupported feature now is, amazingly, "quit"; Rybka seems to want to do "something" to "some memory area" in a deallocation loop that produces a crash. It's not easy trying to figure out what 1.5MB of uncommented assembler code does without knowing the C version :-) )

As for how it works: Think of it as working more or less exactly like WINE, except that it's tailored to one specific application (Rybka) and thus can be a whole lot less general. WINE will eventually gain 64-bit support too, but it's a long way forward.

By the way, you're not limited to Windows interfaces even if you run 32-bit Rybka with WINE; you can just specify "wine rybka.exe" in your favourite Linux interface and it should run just nicely. That's one of the neat things about UCI; interface/engine communication is made in a very transparent, engine- and platform-neutral way.

/* Steinar */
Parent - - By gala.martin (**) Date 2007-07-11 15:39
I see. Thanks for the tips. My feeling is that native linux rybka will be available from version 3. So far, anyway, fruit and glaurung are free strong alternatives running native.  If you use are not a "computerchess competitor", they are good enough.
Parent - - By Vasik Rajlich (Silver) Date 2007-07-12 07:39
Generally, I'd like to say that we'll support Linux from Rybka 3. However, schedules have a way of slipping, so no promises yet.

Parent - - By exigentsky (***) Date 2007-10-16 05:14
What about OS X? I switched recently and really want Rybka. ;)
Parent - - By Felix Kling (Gold) Date 2007-10-16 12:02
If you have a 64-bit system you could try the microwine port for OS X by Sesse...
Parent - By Vasik Rajlich (Silver) Date 2007-10-17 10:07
As for a native port - eventually, yes. Windows is first, though.

Parent - - By Marcel (*) Date 2007-07-11 15:51
Can you also make an hack for MAC OSX?
Parent - By Sesse (****) Date 2007-07-11 15:59
There's no good reason why it couldn't be done (assuming you have a 64-bit Intel Mac), but I don't have a Mac and I'm not particularly interested in the platform.

/* Steinar */
Parent - By Sesse (****) Date 2007-10-14 10:07
An update here: You might want to try microwine 0.5, as it has Darwin support. It's rather alpha, though.

/* Steinar */
Parent - - By Vasik Rajlich (Silver) Date 2007-07-12 07:38

this looks amazingly cool :)

Of course, please feel free to release it. In fact, after a few other guys give it a shot, I'd be very happy to offer it for download from our web site (as an unsupported tool of course).

If there is anything which is easy which I can do or tell you to make your life easier, please let me know.

Parent - - By Sesse (****) Date 2007-07-12 10:37
Vas: Thanks for the enthusiasm! :-)

It's online now, at [url= ][/url]. I'd be very happy for testing/feedback -- note that it still crashes from time to time, and tracking down these issues might prove hard, so don't use it for anything you care about.

As for easy things that could help, there are probably a few, but I believe it's not worth spending time on -- if you have spare moments and the energy it'd be much better spent just making the native Linux port. (When a layer like this can be made in ~48 hours, I think it indicates that a sourceful port won't be terribly much work.)

/* Steinar */
Parent - - By Felix Kling (Gold) Date 2007-07-12 12:19
I put a link on .
If you want to add sth. or if I should mention your real name instead of "Sesse", just tell me :)
Parent - - By Sesse (****) Date 2007-07-12 12:27
Please add that they should really read the README :-) There are some pitfalls, especially in the case where Rybka is stopped abruptly. (You might also want to emphasize that it's not supported by you guys at all -- I don't really think you want to support the combination microwine+Rybka as it currently stands. I've noted this in the README, though.)

Also, yes, I prefer to go by my full name (Steinar H. Gunderson). I thought it was available via the forum, but obviously not. :-)

/* Steinar */
Parent - - By Felix Kling (Gold) Date 2007-07-12 12:46
OK, now it should be good.
I think the word "experimental" indicates that there's no support by Convekta and there's your note in the readme, so I guess there won't be problems.

I would like to know if it really works for other 64-bit Linux users (I'm using a 32-bit Ubuntu since my computer is quite old), as I am no programmer this looks a bit like a miracle for me ;)

Another question is if you can use it in one of the Windows GUIs running with Wine (since they expect an .exe file and no command), I would only know how to start it in e.g. Jose...
Parent - By Sesse (****) Date 2007-07-12 13:41
If they can't take any arguments, I guess the easiest thing would be making a shell script, along the lines of

#! /bin/sh
exec /home/sesse/microwine-1.0/microwine /home/sesse/Rybkav2.3.2a.exe "$@"

and chmod +x that, and give it as the "engine".

If this doesn't work, let me know, and I'll make a version of microwine that can combine into the Rybka executable.

/* Steinar */
Parent - - By Sesse (****) Date 2007-07-13 16:48
> If there is anything which is easy which I can do or tell you to make your life easier, please let me know.

I actually thought of something, now that things seem to have stopped crashing on exit (and hopefully otherwise too :-) ). Does Rybka depend on critical sections for anything important? Currently, I don't implement them at all, but it would be worth it if they're used to synchronize access to the hash or something. (The stdio layer uses them, but that doesn't matter as long as you're not using multiple threads.)

/* Steinar */
Parent - - By Vasik Rajlich (Silver) Date 2007-07-14 06:49
Hi Steinar,

no, Rybka herself does not perform any synchronization between processes at all.

Actually, I am not 100% sure that something like the CRT would not do this, but this would really surprise me.

Parent - - By Sesse (****) Date 2007-07-14 20:58
The CRT definitely does it -- when you use, say, fwrite, the FILE* object is being locked -- at least as long as you're using the multithreaded runtime. But it doesn't matter as long as you're only using one thread (and the processes don't mind unsynchronized access to the shared memory segments), so I'll just leave them as stubs without worrying.

I'm still waiting for other testers' experiences, but for my own use this is starting to work rather well; I haven't seen a single crash since the bugfix in 0.2, and with 0.3 (coming soon) the memory leaks on ctrl-C/crash should be gone too. Now I only need somebody crazy to reverse-engineer the .ctg format... :-)

/* Steinar */
Parent - By Vasik Rajlich (Silver) Date 2007-07-16 10:29
Yes, I guess this makes sense. Anyway, yes, Rybka never starts a second thread inside the same process, and the communication between the processes is completely unsynchronized.

Parent - - By Sesse (****) Date 2007-07-13 14:31
Version 0.2 is out, Felix, could you update the URL?

Citing directly from the changelog:

Version 0.2, 2007-07-13:

* Use calloc instead of malloc, since that matches HeapAlloc's behavior
   better. Rybka expects some 512-element (?) table to be initialized
   to all zero, and when it's not, crashes occur at exit. From what I
   can see, this should also fix the crashes that have happened runtime,
   but this needs more testing. Update README accordingly.
* Implement ExitProcess, now that the process actually gets this far
   without crashing.
* Add this changelog.

/* Steinar */
Parent - By Vasik Rajlich (Silver) Date 2007-07-14 06:58

thanks very much. I have updated the links and comments on our installation page.

Parent - - By Sesse (****) Date 2007-07-26 13:23
I finally got to release 0.3; . From the changelog:

Version 0.3, 2007-07-26:

* In CreateFileMapping, record the ID of the mapping, and install a
   signal handler that cleans up all such mappings at abnormal end (ie.
   from most common trappable signals). This should fix most issues where
   Rybka would leak shared memory. Update README accordingly.
* Add a note that changing engine priority is not supported.
* Check the length of the file instead of hardcoding it as a #define.
* munmap and close the executable file after we're done with it.
* Drop freeshm from the distribution, as ipcrm does exactly the same just
   as well, and is a standard tool.

/* Steinar */
Parent - By Felix Kling (Gold) Date 2007-07-26 14:11
I just updated the website.
Parent - - By revengeska (**) Date 2007-07-26 19:10
I'd be happy to test it(I'm running Fedora 7 x64).  I just don't know what to do after I run the ./microwine Rybkav2.3.2a.x64.exe command:P
Parent - By Felix Kling (Gold) Date 2007-07-26 21:14 Edited 2007-07-26 21:16
I guess you would have to type in "UCI" etc. .
Or you could try to use a GUI like "Jose", there you can put in a command line, it should work there...
I'm not sure how it's possible to use it under windows GUIs with WINE

to quote Steinar:

"If they can't take any arguments, I guess the easiest thing would be making a shell script, along the lines of

#! /bin/sh
exec /home/sesse/microwine-1.0/microwine /home/sesse/Rybkav2.3.2a.exe "$@"

and chmod +x that, and give it as the "engine".

If this doesn't work, let me know, and I'll make a version of microwine that can combine into the Rybka executable.

/* Steinar */"
Parent - By Vasik Rajlich (Silver) Date 2007-07-28 07:52
Thanks Steinar :)

Up Topic Rybka Support & Discussion / Rybka Discussion / (Experimental) 64-bit Rybka on Linux

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill