Not logged inRybka Chess Community Forum
Up Topic Rybka Support & Discussion / Rybka Discussion / Is Cluster currently the strongest chess entity?
1 2 Previous Next  
- - By RubiksChess (**) Date 2012-08-07 09:46
I suppose top CC players could beat the cluster at CC time controls and I'm sure there are lots of centaurs that can beat it at even faster time controls.

But my question is if it is currently the strongest non-centaur entity at classical and faster time controls.

I haven't heard of anything better but the cluster rental website makes no mention if it being the strongest.
Parent - - By Ray (****) Date 2012-08-07 12:19

> I suppose top CC players could beat the cluster at CC time controls and I'm sure there are lots of centaurs that can beat it at even faster time controls.


Complete rubbish, like all your postings.
Parent - - By Uly (Gold) Date 2012-08-07 13:06
I don't find it completely rubbish, the main reason a Corr chess player can beat the cluster is that one can get into a position Cluster software doesn't understand, but one helped with some other software does, then the huge computation of the cluster would be wasted by the engine checking irrelevant variations and pruning the good ones, because the engine just doesn't understand.
Parent - - By RubiksChess (**) Date 2012-08-07 13:40
Would you also agree about the Centaur part of my statement?

I mainly assume that is so because I haven't heard any great discussion along the lines of "computer so strong it is better unaided" which I assumed would be a pretty exciting transition.
Parent - By Uly (Gold) Date 2012-08-07 14:04

> Would you also agree about the Centaur part of my statement?


I think at least one hour/move is necessary for this (showing superiority over the unassisted engine). This will remain more or less true unless there are really fast interactions, but the Cluster has shown superiority to Centaurs at 90 minute games or so.

> I haven't heard any great discussion along the lines of "computer so strong it is better unaided"


I think that's a given for short control games, for instance, at bullet time controls any interaction by the centaur is going to cost him the game on time. Similarly, blitz and standard time controls (15 0) are ruled out, if only because at the end of the game the centaur will not have time to interact well, or he'll need to play faster all along, which only allows shallow analysis which will lead to making worse moves that what an engine would have played unattended, and this is critical. Results at Freestyle time controls were tried and a previous Cluster version proved better than the Centaurs.

So, to have superiority the centaur needs more time to outsmart the cluster, and sadly, this would require having the game adjourned and played in sessions because the centaur can't be expected to play a full 1 hour/move game in one sitting, or the centaur would need to play at full corr time controls so he has time to sleep while the game is running.
Parent - By RubiksChess (**) Date 2012-08-07 13:41
Well that and the fact that I've seen some pretty incredible performances from good Centaurs.
Parent - - By RubiksChess (**) Date 2012-08-07 13:38
There's an issue that arises when you stop caring about what's true or logical and instead go for personal attacks.

The issue is that you end up being wrong a lot or at the very least hampering your understanding of things.
Parent - - By Ray (****) Date 2012-08-07 17:52
Not a personal attack. A comment about the content of your posts not an attack on you personally. And I still think the statement is dead wrong.
Parent - - By RubiksChess (**) Date 2012-08-08 04:09
You think that the cluster can defeat top correspondence players at correspondence time controls?

Or you think that the cluster can defeat top centaurs at faster time controls?

Or both?
Parent - - By Ray (****) Date 2012-08-08 05:07
Both.
Parent - - By RubiksChess (**) Date 2012-08-08 05:45
Not sure why you think that.

Correspondence chess is played on a level computers aren't even able to brute force their way into yet. At least top tier correspondence chess.
Parent - - By Ray (****) Date 2012-08-08 06:10

> Correspondence chess is played on a level computers aren't even able to brute force their way into yet.


I don't agree with that statement, especially when you are talking about the cluster.
Parent - - By Uly (Gold) Date 2012-08-08 09:04
Ray, you've never seen the Cluster playing at corr time controls, it could as well hit a wall in its strength at which point his move will not be better, while the corr chess player can keep improving his move, this basically gives the Cluster a time handicap, and the longer the time control, the worse this becomes.

This is a problem that all unassisted engines have and that's a reason their performance on corr chess sucks, I don't see why it would be different from the Cluster.
Parent - - By Kappatoo (*****) Date 2012-08-08 09:33
Is there any indication that at some point, more thinking time does not improve the quality of an engine's moves?
Parent - - By Uly (Gold) Date 2012-08-08 10:23
Check this game:

http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=24693

Count the times that the main move changed because the engine had 8 hours, from those times, count the ones where the move that was switched in was actually better than what the engine would have played with less time.

While you're on it, you can count the times where the engine plays a move that wasn't in the top 3 in MultiPV, at lower depth.

You'll notice several hours of analysis time were wasted time, where the engine did nothing useful, and would have played the same without so many time. You'll also see that the moves the engines came up with were easy to predict at shallower depth, so that a corr player could have predicted the move in ponder time and prepare best reply against them and against the non-played ones, and then improve on that when is its time to move.

The more you increase the engine's time, the more time that is wasted in this fashion, because the search strategy is optimized for short time control games but not for much longer time controls in where an extra ply will do nothing.
Parent - - By Kappatoo (*****) Date 2012-08-08 12:08
None of this is evidence for your claim.

Generally speaking, today's top engines change their main move quite rarely from one move to the next. This is not just a phenomenon applying to bigger depths. So:
Do you have any data how often the engine's first move changes from one ply to the next at smaller depth? Do you have any data on how this changes at bigger depth? Do you have any data on how this relates to Elo?

You are the one who claims that engines 'hit a wall' at a specific point, so you should at least have some data to back this up. Recall the thread where Vas said that he tested Elo gains from ply n to n+1 up to high depths and noticed no significance flattening of the curve.
Parent - By Chess_Rambo (***) Date 2012-08-08 13:27

> Recall the thread where Vas said that he tested Elo gains from ply n to n+1 up to high depths and noticed no significance flattening of the curve.


The engine needs exponentially more time for each additional ply. It won't hit a wall, but eventually it will be stuck in a quagmire. :smile:
Parent - - By Uly (Gold) Date 2012-08-08 15:01

> You are the one who claims that engines 'hit a wall' at a specific point


The wall is diminishing returns, and that's well known. After the engine has found the best move, there's nothing left to do, all the time that you give afterwards will be wasted. The corr player can use that time effectively getting ready for the moves the Cluster will play, and building a tree of variations much more powerful than the engine's hash table, that the Cluster can't even share so most tables of the engine's slaves will be cleaned as soon as the variations that they analyzed can no longer be reached on the game.

While the time taken for the engine to play better moves becomes exponential, the time taken by the corr player to play better moves remains linear; while in an absurd 32 days/move game, the Cluster may not get more strength than what it would have played on a 16 day/move game, the corr player could find optimal moves for all possible* continuations of the Cluster from current position 8 moves in advance.

(* possible not meaning "brute force"/"being ready for everything", but "what the Cluster would actually play"/"being ready for anything that would actually happen". That is, after 1.e4 e5 a move like 2.Ba6 would be regarded as impossible)
Parent - - By Banned for Life (Gold) Date 2012-08-08 17:16
While the time taken for the engine to play better moves becomes exponential, the time taken by the corr player to play better moves remains linear;

No, no, no. As a tree based game, chess is an inherently exponential problem. If the correspondence player is better at pruning out poor moves, he'll end up with a smaller branching factor (meaning a smaller exponent), but there is no chance of this becoming linear...
Parent - - By Kappatoo (*****) Date 2012-08-08 17:19
It would be really weird if it was linear. Just think about the implications for the Elo gain compared to an engine when going from, say, 1 day per move to two days per move.
Parent - - By Banned for Life (Gold) Date 2012-08-08 17:26
I think what Uly is trying to say is that people can prune better than engines, and as long as this is true, they can help out by keeping the engine from wasting too much time on useless branches of the tree. I think this is a valid point. Engines still spend a high percentage of time in a lot of positions trying really stupid things! :lol: It would be interesting to have a tool where a player could go through the tree and mark moves to be included/excluded from the search (This might be feasible for three to five plies). This might have the potential of speeding up the search by an order of magnitude (just a guess).
Parent - - By Kappatoo (*****) Date 2012-08-08 18:23
You may recall that I said some time ago that I am no longer sure about this. I had played a correspondence tournament and found that it was extremely hard to improve upon what an engine suggests after, say, a few hours. I had the impression that a couple of my opponents virtually always played engine moves, but still I was struggling to win at least one of these games.
I am aware that many people on this forum believe that their contribution to their correspondence games is huge, but I have become skeptical about this.
Parent - - By Banned for Life (Gold) Date 2012-08-08 19:13
If you look at the current state of high level correspondence chess, you will see many people getting mostly draws, but there are still a small percentage getting a high percentage of wins. Your thesis would explain the former, but not the latter, particularly for people getting a high percentage of wins without impressive hardware.

I'll also note that "always played engine moves" is not really much of an indictment. There is a reasonably high percentage of moves where there are several top moves which many engines will award almost identical evaluations. In many cases, the moves lead to very different games. Chess engines are not known for their strength in picking game positions based on whether they are winnable or not, and in fact don't even have an internal representation for this concept (i.e. an evaluation of zero can mean a dead draw, or a knife edge position where the engine can't determine who's better off).
Parent - By Uly (Gold) Date 2012-08-09 01:30

> there are still a small percentage getting a high percentage of wins.


I hope someone takes the time to go over the lines played of those players and show how they played moves that the engines wouldn't pick after several hours of analysis or moves where the engine would pick that move by luck.

I think games start with some preset chances of white losing, getting a draw, or black losing, and as the game goes on those chances are refined. Stronger players would start with better chances than opponent because of their moves they will play (mainly, superior opening choices), and white would start with better chances than black (unless the strength of the players was too far apart).

Players not being deterministic, will have a chance to make the game easier or harder to the opponent. Unless the losing player is lousy*, what the winning player is doing, is increasing the chance that his opponent makes a game losing blunder, move by move, until the chances are crushing and the winning player gets its dream position.

(* lousy players are going to be defeated even by pure engine moves, so it doesn't matter much what approach is used)

I haven't seen any evidence that the Cluster is as good for this as strong chess players, and what I've seen is that it has failed to win games against entities that were much weaker than corr players and that a corr player could have easily beaten, just by picking a different opening.

The core factor I see here is that engine developers have never tested their engines on "days per move" time control, yet, for some reason users assume and expect that the engine will keep increasing its level indefinitely, and while this may be true on engine v engine games, against a corr player most of the engine's time will be wasted.

Note: there's a sister thread on computer chess, with interesting discussion:

http://rybkaforum.net/cgi-bin/rybkaforum/topic_show.pl?tid=25471

From there, I claim the quality of engine analysis is shown by looking at its PV produced after several hours, and it never reaches uniformity, the moves at the tail always remain low quality and the corr player can easily create a much higher quality line of the same length,  in shorter time.
Parent - By Geomusic (*****) Date 2012-08-14 08:10
I'll be excited for the day when MC output analysis becomes incorporated into Aquarium.
Parent - By suj (***) Date 2012-08-14 10:46
The key is pruning/branching and going through the tree. That essentially is centauring!
Parent - - By Shaun (****) Date 2012-08-08 09:35
I would like to see the results of a (decent) unattended computer at correspondence chess, has anyone gathered such results?

I play on ICCF and have not been doing too badly but sometimes wonder if I am adding anything? Would the computer on it's own perform better? Obviously I hope not but....
Parent - By Uly (Gold) Date 2012-08-08 10:39
All you need to do is check your brilliant moves that you played that no engine would have found after several hours of analysis.

Also note that engines from years into the future are going to play moves that destroy engines from today, just like engines from today destroy engines from the past. And the same follows for corr chess players, I don't think players with software and hardware from 2007 could hold their games against the best there is today, and I think at some point the current Cluster hardware strength is going to look like Deep Blue looks to us now.

So the moves that can defeat any of your corr opponents are there to be found, but engines unattended will not be able to find them, it's up to you to help them. If you really don't think you're adding anything it's time to look for a different hobby.
Parent - By oudheusa (*****) Date 2012-08-08 11:28
I think strong corr players will hold cluster to a draw, at least.
Aside engine strength the first challenge will be the opening book; imo there is no opening book available that matches the skills of a strong corr player.
Parent - - By CumnorChessClub (***) Date 2012-08-08 16:21
I have been doing a test on a corr site (not ICCF or FICGS) since 01 Jan 2012 with an unaided engine with the following conditions:

Book: based on my corr games from 2011 up to move 21
Time control for engine: 8 hours/40 moves + 4 hours/20 moves + 2 hours+30s/ for rest of game (all opponents 30 days+ per move)

Results so far: score=89%, Score white=88%, score black=90%, ELO average=1937(19), Perf=2300

Now engine has a rating of 2200+ and has started playing opponents in the 2200+ range there has been an increase in the number of draws.

The engine is not in the top 5 but is in the top 10 of most current ratings lists.
Parent - By Shaun (****) Date 2012-08-09 10:46
Thanks for the info...

It will be interesting to see how the performance changes with stronger opponents - please keep us updated.

Shaun
Parent - - By Ray (****) Date 2012-08-08 10:42
Well, I tend to think the corr chess player will hit a wall long before the cluster will. But yes, we are in unknown territory. One advantage is that the corr player has access to several engines.
Parent - - By Uly (Gold) Date 2012-08-08 11:19

> Well, I tend to think the corr chess player will hit a wall long before the cluster will.


What makes you think that? For one thing, the corr chess player doesn't suffer from "branching factor" and is not restricted by iterations.

Compare:

48 hours/move.

Engine takes 26 hours to finish an iteration. Next iteration expected to finish after 52 hours, outside of the time control. Engine does not have time to switch move even if by next iteration a better move would be chosen. The engine plays move as if it only had 26 hours for this move, wasting 22 hours this move.

Corr player doesn't have any of these problems, he can make better extensions and pruning to find the above move earlier because he can focus on his plan instead of wasting time checking poor variations with centipawn score closer to main variation, and can give equal time to worthwhile variations (those that have more of a chance to become main move) unlike the engine that will focus mostly on its main move (wasting hours until the best move is finally extended and it's found, but this takes the engine until the next iteration to do so!)

I'm going to repost famous position by Richard Vida, to exemplify how there are positions where corr players' strategy could be fundamentally superior:

3k4/p7/Pp6/1Pp5/2P5/2P2K2/8/3B4 w - - 0 1


How much time does it take to a corr player to see this position is drawn? Even a human without engine assistance can devise a strategy to draw against anything white plays!

How much time does it take to the Cluster to see this is drawn?
Parent - - By RubiksChess (**) Date 2012-08-10 08:16
I have to disagree with you here. Though I fully agree with most of what you've said.

A corr player does indeed have a branching factor. I used to play a lot of correspondence chess on Gameknot, redhotpawn, and ficgs. I was one of the top players there for awhile (was #1 on Gameknot for about a year before I completely burned out and quit chess for a couple years)

I can say without a doubt that correspondence-level analysis (the real kind. where i might have a few boards out with completely different looking positions on them yet all referring to the same game. and where I would have pages of notes and spend nearly every waking hour (at work, in the shower) and a lot of sleeping hours doing analysis) has a very real branching factor.

Some components are linear. But most of those are the types of situations that are fairly linear for an engine as well. There are situations that are linear for me and not for engines. And I imagine there were engine-linear situations that I didn't handle linearly.

But the overall average was a branching factor in the 2-3 range.

At the time, strong players using engines like Fruit or Rybka with really strong hardware were about 200 elo beneath me. When they would be suggested, investigated, exposed, then banned, I would do post mortems on all the games I played with them as well as the games where they scored wins/losses against strong players and it was clear that the engine had serious weaknesses.

Overall, my conclusions were that a human prunes significantly better than an engine, but not enough to have his process described as linear. And I think the real advantages are still in position evaluation and endgame stuff. The player I outed for being a cheater was easily maneuvered into a completely dead position when I trapped a rook. At the time I don't think engines were sophisticated enough to understand that a piece like that, imobilized in late-middlegame/early-endgame, might as well be considered captured.

Subtle positional understanding, game volume assessment (understanding what leads to quiet/loud positions), god-vs-perfect-play (understanding the difference between the drawn 0.00 and the double-edged 0.00), and endgames with long king marches are the types of advantages that it will take the longest for engines to overcome. The branching factor is a trivial aspect. That's only a matter of better hardware and better search.

But I don't think engines will surpass CC players via mastery of the subtle topics. They'll achieve the necessary speed before they need to. Chess has never been as much about AI development as it should have been in my opinion. (also 7 and 8 man tables, better books, and more use of monte carlo won't hurt)
Parent - - By Barnard (Bronze) Date 2012-08-10 17:32

>strong players using engines like Fruit or Rybka with really strong hardware were about 200 elo beneath me. When they would be suggested, investigated, exposed, then banned


banned,why?
Parent - - By Banned for Life (Gold) Date 2012-08-10 18:02
At the time being referred to, using chess engines was not allowed in most cc tournaments.
Parent - By Barnard (Bronze) Date 2012-08-11 04:04
that must be very long time ago,because im playing since near 7 years ago,and engines were allowed
Parent - - By suj (***) Date 2012-08-13 14:54
the real problem for correspondence players playing a timed match is how do you get to a drawn position as the cluster is always going to outsearch you.
Parent - - By Banned for Life (Gold) Date 2012-08-13 17:34
There are still areas where a deeper search isn't enough. King safety is probably the easiest to identify. I strive for positions where I'm not going to castle in my cc games because engines will generally provide a reasonably large negative for this, which may or may not be justified. In a sharp position, this misevaluation can be significant. At a minimum, this ensures a lot of decisive games.
Parent - - By Uly (Gold) Date 2012-08-13 20:53

> misevaluation can be significant.


And there are several misevaluations that engines have, and can be taken advantage of. I've seen several positions in where the end-tail evaluations of the engines after extending the lines for a long time are 0.00 score to very dangerous positions, or 0.40is scores to positions where actually black has the edge, and I doubt the Cluster can do anything about those because the leaf nodes are going to return a wrong score, and most of the search will be focused on "phantoms" (positions that are drawn but that the opponent will not play into because he knows that black has the advantage, but the cluster could think it has to find wins by white on the drawn variations; in other words, the Cluster thinking the best the opponent has is a draw and will aim for it, but since this isn't true, the Cluster will be pruning the moves where it wrongly gives itself a nice advantage and focus on refuting the drawish variations that the opponent will not play.)

I actually think that someone with access to the Rybka Cluster software could have an easier time for this as he could just play out the variations and focus on those that the software misevaluated. While the Cluster can't be outsearched, it can be sidesearched :grin:

The current difficulty of this is that the evaluation of future positions by the Cluster software is unknown, I think that being able to rent one core just to see how Rybka Slaves evaluate the positions could be critical, that's why I think it's important to know how strong is the software, and how much it differs from Rybka UCI in its evaluation. If it was Rybka UCI software with the power of the Cluster, you'd just need to guide the game into these positions with wrong evals and most of the Cluster Cores will be used for irrelevant positions.
Parent - - By Banned for Life (Gold) Date 2012-08-13 21:22
I would guess that the cluster's eval is not that much different than what is available in R4. The main difference is that it searches simultaneously both much wider and much deeper, so for all practical purposes, it won't be out searched. This makes it imperative to play into positions that will be wrongly evaluated, regardless of search depth. Positions where king safety is considered important by the engine (whether it really is or not) should be an obvious focus area.
Parent - - By Uly (Gold) Date 2012-08-13 22:10

> I would guess that the cluster's eval is not that much different than what is available in R4.


I'm not sure, as Rybka 3 evaluation and Rybka 4 evaluations were radically different, and the positions that one misevaluated were different than the other misevaluated, with Rybka 3 misevaluating more positions but being able to evaluate some positions better than Rybka 4 (and, this is a reason I still use Rybka 3 daily in my analysis, there are still some positions where it's the best, or at least where Human or Dynamic are the best).

Rybka 3 also evaluated positions radically differently from Rybka 2.3.2a (with the latter being much worse all around.)

With Vas changing the engine's evaluation so radically it would not surprise me if the positions Cluster software misevaluated were very different from the ones of Rybka UCI, so that the opponent would be for a surprise when the Cluster evaluates very well the position he was aiming for.

Then again, I just saw Freestyle Chess Champion Eros Riccio performance against the Cluster. Okay, those were just 3 games, but I had expected several tries to be needed before a successful strategy against the Cluster could be found, including lots of lost games to prune what doesn't work, yet the first win against the Cluster happened too soon.

It makes me answer the OP question on intuition:

Is Cluster currently the strongest chess entity?

No.

I hold slower time controls are only going to help the centaur.
Parent - - By Banned for Life (Gold) Date 2012-08-13 23:27
As I recall, for R4, Vas simplified the evaluation function and rewrote the King Safety component. No clue what he's done since. As noted on the other page, the games between Eros and the cluster were three years ago, so a lot might have changed since then...
Parent - By Uly (Gold) Date 2012-08-14 15:57

> As noted on the other page, the games between Eros and the cluster were three years ago, so a lot might have changed since then...


Yes, my bad, my comments were based on wrongly thinking his results were recent.

This would now be an very interesting match, since Eros has improved and also has access to much better hardware (hopefully?) and software than what was available in 2009. It would be the ultimate Centaur Vs. Machine challenge (since HEM gave up saying the cluster was way too strong).
Parent - - By suj (***) Date 2012-08-14 10:15
if playing a match it would be silly for someone to have access to the very same engine. I know how important it is to play with an engine where evals are unknown.

To play as centaur with an unknown cluster evals just adds quite a lot of elo. Even the mighty Eros has trouble with it as centaur!

This is one reason I never send out any evals on the sjeng cluster during any games on playchess.
Parent - - By Uly (Gold) Date 2012-08-14 16:03

> if playing a match it would be silly for someone to have access to the very same engine.


Oh? Why do you want to limit the centaur's software if you're not limiting the Cluster's hardware? If there's a software publicly available that can predict the Cluster's moves then it makes no sense to prohibit its use to the centaur. For a start, it would mean Cluster moves aren't that special.
Parent - - By suj (***) Date 2012-08-15 11:41

>For a start, it would mean Cluster moves aren't that special.


Totally agree on this though cluster search would be still better due to its depth and time to depth. The actual strength or advantage to the player with cluster access will be vastly reduced.

>Why do you want to limit the centaur's software if you're not limiting the Cluster's hardware?


The actual advantage is the cluster software supllemented with the hardware.

Try getting to play the cluster say 200+cores and give any top centaur a 64 core cluster of the same engine and the standalone cluster will get trashed big time.
I have tried this.
Parent - By Uly (Gold) Date 2012-08-15 21:53

> The actual advantage is the cluster software supllemented with the hardware.


For the Rybka Cluster this hasn't been tested, the strength of the software, and whether it's stronger or not than top UCI engines (there's a ~40% chance that Rybka UCI 4th is best) is unknown.
Parent - By suj (***) Date 2012-08-14 10:51
yes for sure. Typical example is b97 uncastled king and moving the e5 pawn. At one point all engines evaluated pushing the e5 pawn as even and few moves later it was dead. king safety has been improved a lot thanks to new theory and hardware and perhaps all better programming and chess knowledge coming down from authors themselves.
Parent - By tturgut (**) Date 2013-02-24 01:36
I was just reading this message (which is more than 6 months old...)

I think top level cc players would be my clear favorite. (against cluster using infinite analysis)

But a modified cluster -using Triple Brain (or 5-6 engines combined) +  iDeA may beat strong cc players. But that is not pure cluster..

Human is still my favorite...

best regards,
Up Topic Rybka Support & Discussion / Rybka Discussion / Is Cluster currently the strongest chess entity?
1 2 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill