Not logged inRybka Chess Community Forum
Up Topic The Rybka Lounge / Computer Chess / Changes to tablebase.sesse.net
1 2 Previous Next  
- - By Sesse (****) Date 2009-08-27 20:10
Hi,

I've made some changes to the single-connection enforcer at tablebase.sesse.net; now it will simply refuse extra downloads if you've already got a connection going, with an HTTP error code. Currently it's not active for everyone for various reasons, but that will probably happen later -- hammering clients will also be blocked out at some future point.

Let me know if it breaks anything for legitimate clients.

/* Steinar */
Parent - - By Sesse (****) Date 2009-08-27 22:16
OK, I got tired of the hammering clients sooner than I'd thought, so now the hammer protection is also in place (hammer too much and you will be automatically nullrouted).

/* Steinar */
Parent - - By Mark (****) Date 2009-08-30 18:14
Has your tablebase site been down the last couple days?  I can't seem to connect. Thanks for providing this service.  I'm about half way done!
Parent - - By Sesse (****) Date 2009-08-30 19:56
No, it's been more up than ever -- my guess is that you were taken by the limiter. What's your IP address?

/* Steinar */
Parent - - By Mark (****) Date 2009-08-30 22:08
It's 192.168.1.4

Not sure why I got bumped off.  I've only been downloading one file at a time and only a couple GB this year.

Thanks!
Parent - - By Sesse (****) Date 2009-08-30 22:10
That's your internal IP. Try whatismyip.com.

/* Steinar */
Parent - - By Mark (****) Date 2009-08-30 22:12
69.140.58.244

Thanks again...
Parent - - By Sesse (****) Date 2009-08-30 22:34
OK, I've looked through the logs, and they seem pretty inconsistent (some logs show that IP making lots of requests, and some don't show it at all). I've made some changes, cleared the ban list and we'll try again. :-)

/* Steinar */
Parent - - By Mark (****) Date 2009-08-30 22:53
Well, after I had a problem in the middle of a download, I probably tried half a dozen times to re-download the same file, so maybe that was it.  Also, the last couple days I tried unsuccessfully to get to the site a number of times.
Parent - - By Sesse (****) Date 2009-08-31 09:56 Edited 2009-08-31 10:04
It needs over 50 unsuccessful attempts in a relatively short time frame to trigger, so I'd guess it was more a bug. Once you're blocked, you're blocked, though -- nothing you do then will do anything to or from.

Edit: I've found and fixed one bug that probably is related to this (it would correctly detect more than one request, but it'd ban you way too quickly for it).

/* Steinar */
Parent - - By Vegan (****) Date 2009-09-01 02:47
emule is the defacto distribution for tablebases. Tablebases do tend to overwhelm emule but the latest updates are more stable.

p2p is the easiest way to download 1.25 TB of data in 2804 files.

One link, for all the files.

Mind you I could make a torrent for people too. Torrents are 100x more popular then emule.
Parent - - By Sesse (****) Date 2009-09-01 05:04
Sure, I won't keep you from using eMule if that's the way you prefer. Personally I'm glad I won't ever have to touch it again, though. :-)

Torrents is an interesting proposal, but it's really difficult to seed a torrent that big without completely trashing your I/O subsystem.

/* Steinar */
Parent - - By Vegan (****) Date 2009-09-02 05:51
Seeding a torrent would be problematic unless I can find adequate numbers of seeds to make it feasable. I suggest 20 at a minimum who could commit bandwidth to make it work.
Parent - - By Sesse (****) Date 2009-09-02 17:14
20 people seeding a 1TB torrent would still be >50GB working set for each (assuming perfect division), which is hardly feasible at high speeds. You'd be hitting disk like crazy (torrent seeding is exceedingly random-access, as compared to a web server where you at least get some sequentiality).

/* Steinar */
Parent - - By ErO Sennin (**) Date 2009-09-08 11:41
I'm sorry, but this decision to limit the number of downloads and not let the use of download managers is bad, imagine having to do the download 1.2 tb, and 1 download at a time, certainly take years and much patience, I was thinking purchasing a faster Internet to accelerate it, but after that I see that it is better to buy the  NALIMOV DVD's of the chessbase. I do not understand why this decision, 3 months ago, was quick and easy, what happened to make all these changes?

-> Sorry I'm using the translator of google ;) <--
Parent - - By Sesse (****) Date 2009-09-08 22:40 Edited 2009-09-08 22:46
If one download at a time is slower than multiple downloads at a time, one of two is happening:

1. Your TCP stack is misconfigured, and you should fix it, and/or
2. Some pipe somewhere is full, and all multiple downloads would do is push others away so you get more bandwidth relative to others.

I have no sympathy for either case. :-)

Edit: FWIW, this is not a “change”. From day one the policy has been that using multiple connections at a time will get you banned, and the site has been very clear about that. If you've gotten around, it was simply a matter of the old tools not being good enough -- the new ones are not perfect either, but significantly better (and the machine is doing a lot better as a result), and reject people in a generally nicer way.

/* Steinar */
Parent - - By Mark (****) Date 2009-09-09 00:33
Out of curiosity, how many people can download at a time, or is the bandwidth just divided up evenly among all the requesters?
Parent - By Sesse (****) Date 2009-09-09 03:52
The latter. The machine is on gigabit, so it would be pretty rare that all of it was filled up with chess. (It has happened for some of the other things it's serving, though, but that's mostly event-based, not regular traffic.)

There is a global limit on the server, which is something like 250 connections, but that should almost never get hit -- until someone comes along with a download accelerator that's set to like a thousand threads and starts up new connections until the server is full. (Yes, this used to happen at irregular intervals before I instated the current one-session-each enforcing.) If there would ever actually be 250 people downloading at once, I'd up that limit; it's mostly there as a safety check.

/* Steinar */
Parent - - By Sunny (***) Date 2009-09-08 13:38
Steinar
I downloaded krnkrp from you website as emule is bit slow, but when I load them and restart the GUI, GUI displays as error that can't load Rybka. Total number of files for krnkrp are 16 I downloaded from your site. What could be the reason for the failure?
Parent - - By Sesse (****) Date 2009-09-08 22:41
Have you verified that the files are downloaded okay? There's .md5 files in each directory; you can check your download using that.

I'm afraid I can't help you more than that -- if the files are okay, it's a general GUI/engine problem.

/* Steinar */
Parent - - By TheNightFlier (*) Date 2009-09-09 14:27

> Have you verified that the files are downloaded okay? There's .md5 files in each directory; you can check your download using that.


Use MD5checker that does it automatically for all the tablebases.
It is a nice utility contained in a folder inside Frenzee Feb08 that you can find at http://www.computerskak.dk/
Parent - By Sunny (***) Date 2009-09-29 14:24
This helped me. One of the filke was corrupt. I downloaded whole set thrice and with same problem and after using MD5Checker I downloaded the currupt file and then it worked fine. Thanks a lot
Parent - - By GCP (***) Date 2009-09-09 11:49
Thanks a lot for providing this service. I didn't know about it until this post, but it allowed me to get some egtbs I was missing. Too bad my ISP has quota. I hope the site stays up until I can get everything :)

PS. wget says the mimetype is text/plain, but it doesn't seem to corrupt the downloads on Linux. It might corrupt transfers on Windows, though.
Parent - - By Sesse (****) Date 2009-09-09 17:03
Hm, good point, perhaps I should use application/octet-stream or something along those lines.

I really hope browsers have stopped doing CR/LF transfers over HTTP, but you never know...

/* Steinar */
Parent - - By ErO Sennin (**) Date 2009-09-10 11:49
Well before I used a download manager, it was good because I did not have to worry about keeping monitoring the computer, now I get between 30 to 30 minutes checking that the download completed ...
Parent - - By Sesse (****) Date 2009-09-10 12:36
There's nothing saying you can't use a download manager, or whatever tool downloads multiple files serially for you -- what you cannot is set it to download using multiple connections at once.

/* Steinar */
Parent - - By Mark (****) Date 2009-09-10 14:33
I've never tried a download manager before, but it looks like there are a lot of free ones available.  I assume ones that are advertised as "accelerators" would be prohibited.  Just a basic one to allow serial downloads would be great, but I'm almost afraid to experiment for fear of getting banned...
Parent - By Sesse (****) Date 2009-09-10 14:41
Well, the threshold for getting banned is quite high, so unless you pick a very aggressive one some experimentation should be safe. (And I tend to unblock people who ask, especially in the case of “I was trying to find non-malicious software” like this :-) ) Any reasonable download manager should really show if it's using one or more downloads at a time...

/* Steinar */
Parent - By Rubén Cómes (****) Date 2009-09-29 03:35
Hello Sesse!

Sorry for my bad and little english. I hope you can understand.
First of all thank you very much for a good service.
Unfortunately I am late to read the conditions and prohibitions of the service.
I do apologize for the mistake of trying to download improperly. I had no malice, just ignorance on my part.
I am using FileZilla to download the TBs.
I do not know technically how it works, but the person I remotely installed and configured it, did it to download several files at once. As I read this wrong.
At present, the person who helps me to download FileZilla tells me that configured to only download a file.
The problem is that apparently my IP is blocked or something.
I have more than one week trying to log in, but it is not possible.
If possible remedy this would be much obliged. Obviously I reiterate my apology and will never again occur.
My IP: 168.226.108.199
Many thanks. I am at your service for what might help.
Regards
Rubén - Argentine
Parent - - By Sesse (****) Date 2009-10-11 18:21
It turns out the power went out in the server room today, and the UPS was not powerful enough to keep the server up. Thus, it's been restarted, and the ban list doesn't persist across restarts, so I guess those who were previously banned have a second attempt. :-)

/* Steinar */
Parent - By Sunny (***) Date 2009-10-12 02:00
That means Crawly can start downloading TBS from your site. :)
- - By ErO Sennin (**) Date 2009-09-10 14:02
which program is safe? I used the Orbit downloader, it's very good, but now I can not use it.
Parent - - By Sesse (****) Date 2009-09-10 14:37
I have no idea -- I don't use Windows. Anything that uses only one thread, or can be configured to use one, should be safe, though.

/* Steinar */
Parent - - By ErO Sennin (**) Date 2009-09-10 14:46
this server is private?
Parent - - By Sesse (****) Date 2009-09-10 14:50
Define “private”. It's public as in anyone can access the service as long as they abide by the one simple rule “only one download at a time”. It's private as in it's my own hardware and thus I do whatever I want with it. :-)

/* Steinar */
Parent - - By ErO Sennin (**) Date 2009-09-10 14:54
how you acquired all Nalimov's
Parent - - By Sesse (****) Date 2009-09-10 17:40
Patient downloading from various sources over the course of almost a year.

/* Steinar */
Parent - - By ErO Sennin (**) Date 2009-09-10 17:56
you can share with us some of these sources? if you can send me into a private:)
Parent - - By Fulcrum2000 (****) Date 2009-09-10 18:00
As he mentioned 'Patient downloading' I guess emule.
Parent - - By Sesse (****) Date 2009-09-10 18:43
Yes, eMule, various HTTP and FTP sites found via Google (I've long since forgotten where, and I'd guess some of them are down now), and one or two persons who offered to upload some files privately.

/* Steinar */
Parent - - By Vegan (****) Date 2009-09-11 00:41
I managed to kill 3 disks so far this year. Ouch. Once I scrape up some more disks, I can post tablebases on-line via eMule which is 100x easier to use. I made a super easy single emule collection for the whole 6 pieces. Once I get a new fat disk, I will put it up on the web server for those who want to abuse the net downloading tablebases. I can also send a disk anywhere on the planet (that has mail) if you prefer.
Parent - - By Sesse (****) Date 2009-09-11 09:09
Okay, if you think eMule is 100x easier, go ahead -- I won't stop you. Personally I think it's 100x more cumbersome, which is why I preferred to do it this way :-) To each his own, and it's a good thing humans can be different.

PS: The tablebases are also available over FTP, for those who prefer using FTP clients.

/* Steinar */
Parent - - By ErO Sennin (**) Date 2009-09-11 11:42
Well the 2 methods has its advantages, the emule is good because it does not require monitoring but the speed of the download time is not good. The downloaded it via ftp now requires monitoring but the speed is higher and more stable.
The best way is to use the 2 modes, when the monitoring PC, download
via ftp, when you can not monitor use emule.
Parent - - By Sesse (****) Date 2009-09-11 14:42
By the way, are you the guy at 91.77.12.145? (It doesn't match up with the nationalities, but that's the only one who has used FTP for tablebases to my site today.) If so, could you take a look at http://tcpmeasure.sesse.net/ and report the output to me? The TCP parameters on your side look a bit unusual (but it might just be me).

/* Steinar */
Parent - By ErO Sennin (**) Date 2009-09-11 17:20
I'm not downloading this week, but next week will see my ip, this address not open heating may be due to the blockade of the server of my work. We are installing an internet link faster here in my work.
Parent - - By Vegan (****) Date 2009-09-29 05:51
If there were a lot more seeds eMule would be far faster. I still am sharing tablebases.
Parent - - By Rubén Cómes (****) Date 2009-09-29 07:25
Hello Sesse!

Sorry for my bad and little english. I hope you can understand.
First of all thank you very much for a good service.
Unfortunately I am late to read the conditions and prohibitions of the service.
I do apologize for the mistake of trying to download improperly. I had no malice, just ignorance on my part.
I am using FileZilla to download the TBs.
I do not know technically how it works, but the person I remotely installed and configured it, did it to download several files at once. As I read this wrong.
At present, the person who helps me to download FileZilla tells me that configured to only download a file.
The problem is that apparently my IP is blocked or something.
I have more than one week trying to log in, but it is not possible.
If possible remedy this would be much obliged. Obviously I reiterate my apology and will never again occur.
My IP: 168.226.108.199
Many thanks. I am at your service for what might help.
Regards
Rubén - Argentine
Parent - - By Sesse (****) Date 2009-09-29 09:53
Hello,

That entire ISP was banned a while ago after someone repeatedly switched IPs to get around the blocking.

/* Steinar */
Parent - - By Rubén Cómes (****) Date 2009-09-29 11:05
Steinar, can you say me if I can log in your web?
How can I do?
Thank you very much!
Regards
Rubén
Parent - - By Sesse (****) Date 2009-09-29 11:11
You cannot. Your ISP is banned.

/* Steinar */
Up Topic The Rybka Lounge / Computer Chess / Changes to tablebase.sesse.net
1 2 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill