Not logged inRybka Chess Community Forum
Up Topic The Rybka Lounge / Computer Chess / AZ vs SF - game 99
1 2 3 Previous Next  
Parent - By udokaiser (**) Date 2018-01-09 23:25
Me too.
Parent - - By Carl Bicknell (*****) [gb] Date 2018-01-10 09:57
With Giraffe the E.F was the NN, so I'm not sure how that could be handcoded. With A0 the whole thing is a NN - that would be mind boggling to code. I wonder if Deep Mind themselves really understand how it choose its moves.
Parent - By Chris Whittington (**) [fr] Date 2018-01-10 10:20
There must be some at Deepmind with a meta-view, in general terms.
by handcoded, I mean making a system based on scoring features in a linear polynomial, might be possible, I am at early stages of experimenting
Parent - - By turbojuice1122 (Gold) [us] Date 2017-12-27 20:36
Obviously, far more than 4 hours of computer time was used--it's just that 4 hours of computer time was used to actually learn to play well.  Millions of man-hours were probably spent developing a program that can learn to play well, so it's not so outlandish as might be imagined.

Nonetheless, I would still like to see an independent match--for example, AZ against something like the Thoresen's computer alternately running Stockfish, Komodo, and Houdini with a top opening book and good operator.
Parent - - By Banned for Life (Gold) Date 2017-12-27 22:08
Millions of man-hours were probably spent developing a program that can learn to play well

The upper limit would be hundreds of thousands of hours, and this would include all research done at DeepMind since its inception...
Parent - - By Chris Whittington (**) [fr] Date 2017-12-27 22:52
A little bird told me DeepMind took this project very seriously and threw their entire workforce onto it for several months. Entire workforce is supposed to be 100 PhDs. Took many iterations to find the network parameters that worked. Top bleeding edge AI guys in London are on around 250K. That computes to a lot of salary. No way to recoup that sort of figure selling Chess, Go and Shogi programs. On the other hand the PR marketing value of all those column inches is incalculable. The teamsters will have gained a lot of experience in the art of the how of deep learning networks. Cool that Deepmind is in London.
Parent - - By Banned for Life (Gold) Date 2017-12-28 05:29
I don't doubt that both DeepMind and Google wanted to make this a high priority, especially after they got such great PR from their Go effort which was front page news all over the world.

Doing some quick calculations, if you had 100 PhDs spending full time <2000 hours per year since the founding in 2010, you might have 1.5 million man hours worked at DeepMind. Of course DeepMind didn't start with 100 PhDs, and hasn't spent 100% of it's time working on games, even though they are excellent PR not only for DeepMind but also for Google.

I would guess that the total number of hours isn't larger than 300K, which is still a huge investment with no financial return. Given a salary of 250K (dollars I assume rather than pounds?) which is about 125 per hour, times a burden rate of 2 (it's probably higher than that for google), that would be $75 million dollars. That's a pretty large speculative program for PR only, even for Google...

The UK seems like an early leader in machine learning. This is probably due to the need to compensate for the lack of natural intelligence in the UK! :lol: (just kidding)
Parent - - By Chris Whittington (**) [fr] Date 2017-12-28 11:50
Pounds.

U.K. Has been traditional leader in developments by single genius working more or less alone.

It may be we have higher levels of genes for autism/aspergers, and we tolerate weirdo cranky people. Or produce weirdos in the education system. Public boarding schools seem quite good at that.

It maybe that deepmind also did AZ for fun reasons. I certainly did chess in later years for fun, while other things made the money. Hassibis is so rich he doesn't need to work,  can see him sponsoring "interesting" over commercial with the idea that something may come out of the interesting or not. Probably it also keeps the highly desirable maths physics neurology guys there, they like fun too.

There is billions going into AI now, the difference with boom bust of old days is that the technology is practically useful, and makes serious money. this AI boom is based, no longer on the dream of chess, which proved to be not widely useful, but on go and now on generality and which (deep learning) is proven useful big time
Parent - By Banned for Life (Gold) Date 2017-12-28 16:43
I'm sure that they had no problem finding guys to work on the board game applications, but it wouldn't have been possible if Google, which paid a half a billion dollars for DeepMind a few years ago, wasn't on board. I'm sure they must have had some heated discussions early on, but after the recent highly publicized successes in Go and Chess, I'm sure pretty much everyone is happy, and I've read that this has also given DeepMind a lot of additional cachet in the all important machine learning market.

Your point on Hassibis is probably right on. He's been interested in board games for a long time, and no doubt Google wants to keep him happy and on board, and he could certainly start a new company if he was so inclined and his contract allows for it (which is doubtful for several years). Google is probably keeping him happy both by indulging his interest in board games and also investing massively in his team which competes with Google Brain for resources.

AI is certainly booming right now, but the returns, while almost certain, are mostly in the future, and sometimes it takes longer than you expect for the future to arrive. I say this as a guy who is still waiting for graphene to become commercially viable...
Parent - - By Carl Bicknell (*****) [gb] Date 2017-12-28 09:54 Upvotes 1

> On the other hand the PR marketing value of all those column inches is incalculable.


Exactly.

And how do you enhance that? By going for the strapline "Alpha Zero crushes Stockfish after 4 hours of training"

Never mind about:
- The silly TC
- A 32 core machine with HT on
- Tiny Hash
- No opening book
- or tablebases
- A version over a year old

As in 1997 this feels like a PR stunt, despite the fact that underlying it is something very, very impressive.
Parent - By Chris Whittington (**) [fr] Date 2017-12-28 11:55 Upvotes 1
No point in being cynical, this technology is changing the world under your feet. Revolution. To paraphrase Marx in a Hegelian negation sort of way. The world is changing, the important thing is to understand it.
Parent - By Kreuzfahrtschiff (***) [de] Date 2018-01-11 01:03
@ Carl:

best post i have seen here, i cannot believe that so many other chess players (if they really are) are that blind to AZ

this is a stupid machine who cannot win against the best normal engines today and its no revolution, because chess is already solved
Parent - - By Banned for Life (Gold) Date 2017-12-24 17:28 Edited 2017-12-24 17:55
Ed,

It could be a giant hoax (and I love giant hoaxes! :lol:), but here's what it says right up in the abstract:

In contrast, the AlphaGo Zero program
recently achieved superhuman performance in the game of Go, by tabula rasa reinforcement
learning from games of self-play. In this paper, we generalise this approach into
a single AlphaZero algorithm that can achieve, tabula rasa, superhuman performance in
many challenging domains. Starting from random play, and given no domain knowledge
except the game rules, AlphaZero achieved within 24 hours a superhuman level of play in
the games of chess and shogi (Japanese chess) as well as Go, and convincingly defeated a
world-champion program in each case.


The most significant aspect is the tabula rasa part and the learning from self play. If the effort was really inspired by learning from existing chess engines, it would be much less significant...
Parent - - By Rebel (****) Date 2017-12-24 22:31
As said before, I can believe AlphaGo, I have too little knowledge of the game to have an opinion of AlphaGo Zero.
Parent - - By Chris Whittington (**) [gb] Date 2017-12-24 23:30
Shannon number for Go is way bigger than for Chess. But Go works, and you say chess can't citing Shannon number.
Parent - - By Rebel (****) Date 2017-12-25 09:40
In GO pattern recognition is doable, there are no patterns in chess. Part of its beauty.
Parent - By Chris Whittington (**) [gb] Date 2017-12-25 11:22 Upvotes 1
there are patterns in chess.
there are also probably patterns in chess beyond our human comprehension.
chess heuristics are a kind of pattern.
patterns don't have to have 2D visualisations to be patterns.
bishop pair is a pattern, for example.
all squares aroundthe king controlled is a pattern (a visionable one).
and so on.
Parent - - By Peter Grayson (****) [gb] Date 2017-12-24 02:18

> That's an interesting remark. Care to elaborate?


Yes Ed, the first game I looked at was a French Defence posted on Youtube. Following the game through it quickly became noticeable not only were Stockfish 8's moves confirmed but significantly how many of AZ's moves were being predicted by Stockfish 8 in the game time frame of 60s per move with 1 Gb hash. i.e. a high ponder hit rate. If you check my added engine output from Stockfish 8 in my first reply to your posted game you will see that SF8 predicted a high percentage of AZ's moves giving a similarly high ponder hit rate.

Of course the Youtube video presentation made no commentary regarding this matter but for me that is a very important piece of information especially if the presenter had commented "on many occasions, Alpha Zero played Stockfish 8's predicted move". Looking at two further games and now the one you posted, if I had not known AZ was Stockfish 8's opponent I would have guessed it was a fairly close Stockfish 8 derivative. A matter of opinion though and perhaps spending much time analysing the games may confirm the deviations were key and the rest of the moves were most sensible.

Peter
Parent - - By Rebel (****) Date 2017-12-24 10:52
Ah, good old ponder hits. Mighty interesting to invest time into that and calculate the percentage. But to make your theory believable Komodo | Houdini should score lower ponder hits.
Parent - - By Peter Grayson (****) [gb] Date 2017-12-24 12:11

> But to make your theory believable Komodo | Houdini should score lower ponder hits.


I would not consider it so much a theory but more my perception from looking through the games on my machine. Incidentally, I am not necessarily convinced that just 1 Gb hash was a particularly bad setting because when I gave Stockfish 16 and then 32 Gb hash it seemed to slow the engine.

I do not understand peoples' comments that it is not about the hardware? If AZ had lost overall, we probably would be ignorant the match had ever taken place. Therefore anything that may influence the game outcome is an important consideration to be taken into account on the worth of the AZ programming process.

I have read comments the AZ hardware gave it a 10 to 16 x time advantage, some say it was nothing; because of the different nature of the engines and hardware there is no definitive measure for comparison. However, if anyone truly believes hardware difference would not influence the moves they need to think again. When I analysed move 32.c4 and Komodo 11.2.2 found it and stuck with it after just 13 seconds, I subsequently ran the position on my old Q9550 machine with 1 Gb hash. Komodo on the slower hardware also found 32.c4 but in 119 seconds indicating approximately a 9x speed up for the 32 threads machine. The point being the 32 thread machine would likely find the move inside the time control cut off but the slower machine would not. Hardware then is a big player in the current engines' capabilities.

One telling comment in the AZ paper on page 4 was "In chess, AlphaZero outperformed Stockfish after just 4 hours (300k steps)" that confirms the "100 game" match did not take place until the Google people were satisfied the AZ development was in a position to win. Therefore any suggestion the game was played without previous heavy testing and development against Sockfish 8 is untrue and it suggests to me AZ may have been tuned or minimally influenced by Stockfish.

For me the hardware and time control issues detract from the AZ result. The lack of facility for the Stockfish people to prepare the engine for the match is also disappointing assuming they would choose to do so. The hash size less so but it looks clear the match was not played by an engine tested purely against itself.

From a consumer perspective, until such time the AZ engine and whatever hardware is needed to run it becomes commercially available and is affordable the AZ news item is little more than of passing interest.

Peter
Parent - By Banned for Life (Gold) Date 2017-12-24 17:38
From a consumer perspective, until such time the AZ engine and whatever hardware is needed to run it becomes commercially available and is affordable the AZ news item is little more than of passing interest.

Peter,

It would be surprising if you couldn't buy better hardware for under $10K by 2020. The details of the tabula rasa machine learning algorithms will certainly remain under wraps for as long as DeepMind can keep it's trade secrets from dissipating, but an inference engine based on a NN that can run on commercially available hardware using a DeepMind developed NN configuration and weights could be released if it was seen to be worth the effort. The payback to DeepMind in this case would be in good publicity, but they have courted this in the past...

Alan
Parent - By Chris Whittington (**) [gb] Date 2017-12-24 17:54
"it looks clear that the match was not played by an engine tested purely against itself"

This "theory" gets more and more ridiculously convoluted post by post. Obviously AZ was not TESTEDagainst itself. It was tested against state of the art, stable AB opposition engine, which happened to be Stockfish.

AZ was TRAINED on TRAINING data that AZ generated by playing games against itself. No other data from any other source, only its own games and FIDE rules.

As is normal when training an ANN, training progress is measured every so often by VERIFICATION on different data from the Training data. The verification data was provided from another set of games, games played against Stockfish by the latest ANN. Verification has to be against a known strength stable opponent. Obviously.

Over many cycles they get a plot of ELO difference (current AZ vs Stockfish). If everything works this should provide a rise to positive ELO difference and a more or less continual climb after that until plateau. This takes a non-trivial amount of time using a lot of expensive Google hardware. A few hours.

If this doesn't work, they have to start again from the beginning, different network architecture, number of hidden layers, convolutions bla bla. Nobody has a "science" for these parameters, it's going to be magical tinkering, or else looping through all the various ways an ANN can be built. At a cycle time of maybe four hours that's going to be very time consuming and very expensive and, more importantly, breaks the maxim of NO game specific, GENERAL purpose, Deep learning algorithm for Go, Chess, Shogi. They need a network that works in all cases, only difference: count of inputs, the rules of the game and the output count. Are they even allowed to vary the convolutions and hidden layers between GO Shogi Chess? The complete paper will tell us presumably, but it seems doubtful.

What would seem more likely, is, based on experience with network architecture for Go, plus their general smartness, designed an architecture they thought would cross transfer to Chess and Shogi, ie be general purpose, ran some testing to confirm the ANN would at least learn something in each domain and make progress, and then hurled it at the three game types, using state of the art opposition for each game type to verify progress.

Tuned against Stockfish? Minimally influenced by Stockfish? When all learning data was AZ-AZ self play and a "tuning" cycle (more a does our system work or not cycle) of several hours using very valuable hardware resources over and over again to "tune". Please.

The idea that hidden layer count of 23 would work against Stockfish and not Houdini, while Houdini would fall to hidden layer count of 24 which then didn't work against Stockfish?! Come on. Either the ANN plays powerful chess or it doesn't. I really don't accept one can tune network architecture against specific opposition. You can tune the weights against specific opposition, but AZ has no facility to train weights other than by self-play.
Parent - - By Rebel (****) Date 2017-12-24 22:48
One telling comment in the AZ paper on page 4 was "In chess, AlphaZero outperformed Stockfish after just 4 hours (300k steps)" that confirms the "100 game" match did not take place until the Google people were satisfied the AZ development was in a position to win. Therefore any suggestion the game was played without previous heavy testing and development against Sockfish 8 is untrue and it suggests to me AZ may have been tuned or minimally influenced by Stockfish.

Good point, as the paper also states that the final steps were 700k.
Parent - - By Chris Whittington (**) [gb] Date 2017-12-25 11:45
He uses words wrongly. As Confucius say "If names be not correct, language is not in accordance with the truth of things."

AZ modified weights were VERIFIED against Stockfish. AZ was not developed against Stockfish, it was developed against itself, in three domains.

Confucius go on to say: "If language be not in accordance with the truth of things, affairs cannot be carried on to success."

His "..... therefore ..... then AZ tuned or influenced by Stockfish" is a conclusion based on names being not correct.

The only conclusion valid from the point he is trying so hard to make is: "if Deeepmind couldn't get AZ to work, they probably would have published nothing."

But they did get it to work, in three domains, with generalised learning, no domain specifics (which was the brief) and they did publish the initial paper.

If you believe the Grayson hypothesis, then you have to believe that AZ, learning entirely on self play games, with Stockfish totally absent during learning, has learnt to play chess specificlly against Stockfish and will be unable to repeat the win rate against Houdini or other AB programs. In other words, you would have to believe that many self-play learning cycles, somehow produce Stockfish targeted network weightings. That would be preposterous, not to say idiotic. Bizarre at minimum. Or perhaps, according to such nonsense, it would randomly have a 6000 ELO rating againts Houdini and a 1500 rating against ChessMaster? Just a sort of luck?!
Parent - - By Banned for Life (Gold) Date 2017-12-25 17:55
I don't doubt the veracity of the paper, but another training method that should be possible (but wouldn't be as impressive) would have been for AZ to start with no knowledge other than the rules of the game, and then be trained by playing against some engine, e.g. SF8. This might provide faster learning in the beginning, but would probably be limiting in the end when AZ surpassed SF8 in playing strength. In any event, it would be interesting to try this and compare with the learning curve for self play...

Another interesting question would be how much improvement in AZ's score against SF8 would be possible if the self learning AZ had been further trained against SF8?
Parent - - By Chris Whittington (**) [gb] Date 2017-12-25 18:17
how would that work?

every time Stockfish played e4 it would win
every time untrained AZ played e4 it would lose, same for all other opening moves. so how does the MCTS distinguish win rates of moves when on average all moves score 0.5? And if it can't distinguish one move from another it can't learn.
Parent - - By Banned for Life (Gold) Date 2017-12-25 18:26
I am assuming that losing more slowly is better! :lol: So even though it would initially get crushed by SF8, it could still learn which moves allow it to stay alive longer and eventually get to a point where some games are drawn, and then to a point where some games are won. Of course there is always the possibility that it would never learn how to win, but end up playing excellent anti-chess engine moves that led to consistent draws...
Parent - - By Chris Whittington (**) [gb] Date 2017-12-25 19:41
I don't think the AZ learning algorithm is concerned with speed of win. I might be wrong though
Parent - - By Banned for Life (Gold) Date 2017-12-25 19:48
It has the total move count available. It's hard to imagine that it wouldn't prefer winning more quickly or losing more slowly, especially if it isn't following exact lines, but rather working toward positions where it is more likely to be successful.
Parent - - By Chris Whittington (**) [gb] Date 2017-12-25 22:27
Pretty sure that is not part of the training. For every move in the rollout, it tells the network its output should be 0, 0.5 or 1. There's no move count bias. Well, as usual, paper is silent on some details, but I think that's it. It's a brutal training, 1.e4 will get told it has 1 and gets trained, then 0 and gets trained, then 1, then 0.5 and so on. Quite a kick each time. Under that kicking it's hard to see that a kick based on 1.0 would be seen as much different to a kick based on 0.999. I doubt that the final trained network value would be accurate enough anyway for depth of mate to meaningfully show up. Guessing.
Parent - - By Banned for Life (Gold) Date 2017-12-26 05:51
Playouts started fewer moves from a potential mate should lead to more decisive average results, and this should affect the average scores giving preference to better earlier outcomes. That's the point of the playout, no?
Parent - By Chris Whittington (**) [gb] Date 2017-12-26 13:23
ummmmm .... yes, probably this is true, the earlier the mate win is found, the greater its effect on node valuation and move choice back up the tree. this will reflect in the training of the ANN. good spotted.
Parent - - By Peter Grayson (****) [gb] Date 2017-12-28 01:01 Upvotes 1
From your post above

>The idea that hidden layer count of 23 would work against Stockfish and not Houdini, while Houdini would fall to hidden layer count of 24 which then didn't work against Stockfish?! Come on. Either the ANN plays powerful chess or it doesn't. I really don't accept one can tune network architecture against specific opposition. You can tune the weights against specific opposition, but AZ has no facility to train weights other than by self-play.


and now,

>If you believe the Grayson hypothesis, then you have to believe that AZ, learning entirely on self play games, with Stockfish totally absent during learning, has learnt to play chess specifically against Stockfish and will be unable to repeat the win rate against Houdini or other AB programs. In other words, you would have to believe that many self-play learning cycles, somehow produce Stockfish targeted network weightings. That would be preposterous, not to say idiotic. Bizarre at minimum. Or perhaps, according to such nonsense, it would randomly have a 6000 ELO rating against Houdini and a 1500 rating against Chess Master? Just a sort of luck?!


Nowhere have I claimed Alpha Zero is not a strong computer chess program. I have said "The question then arises, would it have achieved such a good result against Houdini or Komodo? We will probably never know."

You fall foul of your own criticism!

The top engines are within 20 to 30 Elo of each other when run under tournament conditions and using a variety of fixed opening lines to present an overall indication of playing strength. Time controls may influence the differences as will the selected opening lines and hardware.

There is also something strange about these wins.
The two Ruy Lopez, Berlin Defence are not actually achievable with Stockfish 8 for two reasons.
1. With clean hash,  I cannot get it to play 1..e5 in response to 1.e4. The engine looks at 1..e5 for a very short time, but within a second changes to 1..e6 and sticks with it. Then ...
2. Forcing 1.e4..e5, 2.N3..Nc6 (OK), 3.Nf3 and now the same problem. The engine will only look at 3..a6 but does not consider 3..Nf6 so no Berlin defence. Where did that come from? Perhaps the engine behaviour is different with 64 threads?

Unfortunately for Stockfish 8 it does not play the French so well therefore it was little surprise it lost at least two of the French openings. It struggles against Komodo and Houdini too.

Six of the 10 wins came from the same opening line that was a Queen's Indian defence, all getting to 4.g3 by transposition.

rnbqkb1r/p1pp1ppp/1p2pn2/8/2PP4/5NP1/PP2PP1P/RNBQKB1R b KQkq - 0 4


The latest Stockfish engine that I have tested date stamped 111217 when tested in a 600 game match, 300 per engine, had no trouble against Stockfish 8 either coming out 8 games to the good in a 100, therefore my conclusion has not changed. I see nothing special about the Alpha Zero engine that is not achievable by today's current engines. That statement cannot be interpreted as you tried to change to make out that I stated stated Alpha Zero was not a strong engine.

We really need to see the other 90 games that would reveal just how broad or narrow were  the opening lines. I suspect the latter.

One curiosity was some randomness in AZ's opening choice when after 1.d4..Nf6 it chose both 2.c4 and 2.Nf3. Does that mean Alpha Zero may play with some variability? Is that built into the scheme of things?

Peter
Parent - By Chris Whittington (**) [fr] Date 2017-12-28 12:00
Is this a grudge match?!

To answer your question, I believe they add diriglet noise to the ply one move choice in order to get variability. So, yes, that's built in.
Parent - - By MarshallArts (***) [us] Date 2017-12-28 23:23
Didn't SF have White in the Berlin Defense games?
Parent - By Peter Grayson (****) [gb] Date 2017-12-29 11:12

> Didn't SF have White in the Berlin Defense games?


Yes, you are right. I copied and pasted the games from the paper and because the headers were not pgn conformant, all the games were given the same engine pairing header. My mistake so I withdraw the Ruy Lopez, Berlin comment as not applicable.

Peter
Parent - By Peter Grayson (****) [gb] Date 2017-12-23 18:07
Houdini 6.03 and Komodo 11.2.2 also find 21.Be3 in under one minute too.

3: AlphaZero - Stockfish,  2017
rnb2r2/p3b2p/1ppq1p1k/6p1/Q6P/2P3P1/P4PB1/R1B1R1K1 w - - 1 0


Analysis by Houdini 6.03 Pro x64-popc:

21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Bf5 24.hxg5+ fxg5 25.Bg2 Rd8 26.Rd1 Rd7 27.Bf3 b5 28.Qb3 Qe5 29.Rxd7 Nxd7 30.Qd5 Kg7 31.Bg2 Qe6 32.Bc1 Kg6 33.Qc6
  =/+  (-0.65)   Depth: 21/51   00:00:01  87012kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Bf5 24.hxg5+ fxg5 25.Bg2 Rd8 26.Rd1 Rd7 27.Bf3 Rxd1+ 28.Qxd1 Nd7 29.Bc1 Qf6 30.Qa4 Kg6 31.Be4 Qe6 32.Bxf5+ Qxf5 33.Qxa7 h5 34.Qb7 Qe6 35.c4 h4 36.Bb2 hxg3 37.fxg3 Qe1+ 38.Kg2 Qe2+ 39.Kh3
  =/+  (-0.62)   Depth: 22/51   00:00:03  153mN
21.Rxe7
  =/+  (-0.55 !)   Depth: 23/51   00:00:04  220mN
21.Rxe7
  =/+  (-0.48 !)   Depth: 23/51   00:00:05  225mN
21.Rxe7 Qxe7
  =/+  (-0.55 --)   Depth: 23/54   00:00:06  273mN
21.Rxe7
  =/+  (-0.46 !)   Depth: 23/54   00:00:06  280mN
21.Rxe7 Qxe7
  =/+  (-0.58 --)   Depth: 23/54   00:00:06  292mN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Bf5 24.Bg2 Nd7 25.hxg5+ fxg5 26.Qxa7 Rf7 27.Qc7 Kg7 28.Qc6 Qe2 29.Qf3 Qxf3 30.Bxf3 Ne5 31.Bd5 Re7 32.Rf1 Kg6 33.Bc1 h6 34.f4 Nd3
  =/+  (-0.58)   Depth: 23/54   00:00:07  341mN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Bf5 24.Bg2 Nd7 25.hxg5+ fxg5 26.Qxa7 Rf7 27.Qc7 Kg7 28.Qc6 Qe2 29.Qf3 Qxf3 30.Bxf3 Ne5 31.Bd5 Re7 32.Rf1 Kg6 33.Bc1 h6 34.f4 Nd3
  =/+  (-0.58)   Depth: 23/54   00:00:08  367mN
21.Rxe7
  =/+  (-0.51 !)   Depth: 24/56   00:00:10  446mN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Bf5 24.Bf3 Rd8 25.hxg5+ fxg5 26.Rd1 Rxd1+ 27.Qxd1 Nd7 28.Bc1 Ne5 29.Be2 Nf7 30.Bc4 Kg7 31.Bd5 Ne5 32.Qd2 h6 33.f4 Nd3 34.fxg5 Nxc1 35.gxh6+ Kg6 36.Qxc1 Qe5 37.Qd2 Qxg3+ 38.Qg2
  =/+  (-0.50)   Depth: 24/56   00:00:12  535mN
21.Be3
  =/+  (-0.43 !)   Depth: 24/56   00:00:12  536mN
21.Be3
  =/+  (-0.32 !)   Depth: 24/56   00:00:12  536mN
21.Be3
  =  (-0.17 !)   Depth: 24/56   00:00:14  603mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 24/56   00:00:20  855mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 24/56   00:00:20  855mN
21.Be3 Bf5 22.Be4 b5 23.Qc2 Bg4 24.Bf5 Bxf5 25.Qxf5 Qd5 26.Bxg5+ fxg5 27.hxg5+ Kg7 28.Rxe7+ Rf7 29.Qf6+ Kf8 30.Rxf7+ Qxf7 31.Qh8+ Qg8 32.Qf6+ Qf7
  =  (0.00)   Depth: 25/56   00:00:29  1252mN
21.Be3 Bf5 22.Be4 b5 23.Qc2 Bg4 24.Bf5 Bxf5 25.Qxf5 Qd5 26.Bxg5+ fxg5 27.hxg5+ Kg7 28.Rxe7+ Rf7 29.Qf6+ Kf8 30.Rxf7+ Qxf7 31.Qh8+ Qg8 32.Qf6+ Qf7
  =  (0.00)   Depth: 25/56   00:00:29  1252mN
21.Be3 Bf5 22.Be4 Qe6 23.Bc2 Bxc2 24.Bxg5+ Kg7 25.Bh6+ Kg8 26.Qxc2 Qf7 27.Rad1 Re8 28.Qe2 b5 29.Qg4+ Kh8 30.Qe4 Qg8 31.Qf5 Qf7 32.Qe4
  =  (0.00)   Depth: 26/57   00:00:43  1837mN
21.Be3 Bf5 22.Be4 Qe6 23.Bc2 Bxc2 24.Bxg5+ Kg7 25.Bh6+ Kg8 26.Qxc2 Qf7 27.Rad1 Re8 28.Qe2 b5 29.Qg4+ Kh8 30.Qe4 Qg8 31.Qf5 Qf7 32.Qe4
  =  (0.00)   Depth: 26/57   00:00:43  1837mN
21.Be3 Bf5 22.Be4 Qe6 23.Bc2 Bxc2 24.Bxg5+ Kg7 25.Bh6+ Kg8 26.Qxc2 Qf7 27.Bxf8 Bxf8 28.Qa4 Nd7 29.Qxc6 Rd8 30.Qc7 Re8 31.Rxe8 Qxe8 32.Qxa7 Ne5 33.Qxb6 Nf3+ 34.Kf1 Qe4 35.Rd1
  =  (0.00)   Depth: 27/57   00:00:54  2287mN
21.Be3 Bf5 22.Be4 Qe6 23.Bc2 Bxc2 24.Bxg5+ Kg7 25.Bh6+ Kg8 26.Qxc2 Qf7 27.Bxf8 Bxf8 28.Qa4 Nd7 29.Qxc6 Rd8 30.Qc7 Re8 31.Rxe8 Qxe8 32.Qxa7 Ne5 33.Qxb6 Nf3+ 34.Kf1 Qe4 35.Rd1
  =  (0.00)   Depth: 27/57   00:00:54  2288mN
21.Be3 Bf5 22.Be4 b5 23.Qc2 Bg4 24.Bf5 Bxf5 25.Qxf5 Qd5 26.Bxg5+ fxg5 27.hxg5+ Kg7 28.Rxe7+ Rf7 29.Qf6+ Kf8 30.Rxf7+ Qxf7 31.Qh8+ Qg8 32.Qd4 Qd5 33.Qh8+ Qg8
  =  (0.00)   Depth: 28/67   00:01:16  3196mN
21.Be3 Bf5 22.Be4 b5 23.Qc2 Bg4 24.Bf5 Bxf5 25.Qxf5 Qd5 26.Bxg5+ fxg5 27.hxg5+ Kg7 28.Rxe7+ Rf7 29.Qf6+ Kf8 30.Rxf7+ Qxf7 31.Qh8+ Qg8 32.Qd4 Qd5 33.Qh8+ Qg8
  =  (0.00)   Depth: 28/67   00:01:16  3197mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 29/67   00:01:25  3572mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 29/67   00:01:28  3686mN
21.Be3 Bf5 22.Be4 Qe6 23.Bc2 Bxc2 24.Bxg5+ Kg7 25.Bh6+ Kg8 26.Qxc2 Qf7 27.Bxf8 Bxf8 28.Qe4 Na6 29.Qxc6 Rb8 30.Rad1 Nc5 31.Qf3 Bg7 32.Qf5 Rf8 33.Rd6 Kh8 34.Rdd1 Kg8
  =  (0.00)   Depth: 30/67   00:01:34  3933mN
21.Be3 Bf5 22.Be4 Qe6 23.Bc2 Bxc2 24.Bxg5+ Kg7 25.Bh6+ Kg8 26.Qxc2 Qf7 27.Bxf8 Bxf8 28.Qe4 Na6 29.Qxc6 Rb8 30.Rad1 Nc5 31.Qf3 Bg7 32.Qf5 Rf8 33.Rd6 Kh8 34.Rdd1 Kg8
  =  (0.00)   Depth: 30/67   00:01:35  3966mN

Analysis by Komodo 11.2.2 64-bit:

21.hxg5+ Kg6
  =/+  (-0.40 !)   Depth: 6   00:00:00  14003kN
21.hxg5+ Kg6
  =  (0.06 !)   Depth: 6   00:00:00  14010kN
21.hxg5+ fxg5 22.Qh4+ Kg7 23.Bxg5 Bxg5 24.Qxg5+ Qg6 25.Qe3
  =  (0.16)   Depth: 6   00:00:00  14022kN
21.hxg5+ fxg5
  =  (0.10 --)   Depth: 7   00:00:00  17628kN
21.hxg5+ fxg5
  =  (-0.01 --)   Depth: 7   00:00:00  17637kN
21.hxg5+ Kg7
  =  (-0.23 --)   Depth: 7   00:00:00  17649kN
21.hxg5+ fxg5 22.Qh4+ Kg7 23.Bxg5 Bxg5 24.Qxg5+ Qg6 25.Re7+ Kg8 26.Qh4
  =/+  (-0.38)   Depth: 7   00:00:00  17669kN
21.hxg5+ fxg5 22.Qh4+
  =/+  (-0.28 !)   Depth: 8   00:00:00  18777kN
21.hxg5+ fxg5 22.Qh4+
  =  (-0.01 !)   Depth: 8   00:00:00  18784kN
21.hxg5+ fxg5
  =/+  (-0.46 --)   Depth: 8   00:00:00  18795kN
21.hxg5+ fxg5 22.Qh4+ Kg7 23.Bxg5 Bxg5 24.Qxg5+ Qg6 25.Re7+ Kg8 26.Qe3 Nd7
  =/+  (-0.58)   Depth: 8   00:00:00  18818kN
21.hxg5+ fxg5
  =/+  (-0.64 --)   Depth: 9   00:00:00  18846kN
21.hxg5+ fxg5 22.Qh4+
  =/+  (-0.47 !)   Depth: 9   00:00:00  18861kN
21.hxg5+ fxg5 22.Qh4+ Kg7 23.Bxg5 Bxg5 24.Qxg5+ Qg6 25.Re7+ Rf7 26.Rxf7+ Kxf7 27.Qf4+ Kg7 28.Re1 Be6
  =/+  (-0.58)   Depth: 9   00:00:00  19159kN
21.hxg5+ Kg7
  =/+  (-0.64 --)   Depth: 10   00:00:00  19217kN
21.hxg5+ Kg7
  -/+  (-0.73 --)   Depth: 10   00:00:00  19290kN
21.hxg5+ fxg5
  =/+  (-0.59 !)   Depth: 10   00:00:00  19513kN
21.hxg5+ fxg5 22.Qh4+ Kg7 23.Bxg5 Bxg5 24.Qxg5+ Qg6 25.Re7+ Kg8 26.Qe3 Qf6 27.Re1 Kh8 28.a3 Bf5
  =/+  (-0.47)   Depth: 10   00:00:00  19891kN
21.hxg5+ fxg5 22.Qh4+ Kg7 23.Bxg5 Bxg5 24.Qxg5+ Qg6 25.Re7+ Kg8 26.Qe3 Qf6 27.Re1 Kh8 28.c4 Bf5 29.c5
  =/+  (-0.46)   Depth: 11   00:00:00  20204kN
21.hxg5+ fxg5
  =/+  (-0.57 --)   Depth: 12   00:00:00  20511kN
21.Qe4
  =/+  (-0.54 --)   Depth: 12   00:00:00  20513kN
21.Qe4 Rf7
  =/+  (-0.63 --)   Depth: 12   00:00:00  20565kN
21.Qe4 Rf7
  -/+  (-0.75 --)   Depth: 12   00:00:00  20616kN
21.hxg5+ Kg7 22.Qh4 fxg5 23.Bxg5 Bxg5 24.Qxg5+ Qg6 25.Re7+ Kg8 26.Qe3 Bf5 27.Rd1 Qf6 28.Rd4 Kh8 29.a3 Na6
  =/+  (-0.65)   Depth: 12   00:00:00  20638kN
21.hxg5+ fxg5
  -/+  (-0.72 --)   Depth: 13   00:00:00  21171kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.59 !)   Depth: 13   00:00:00  21358kN
21.Rxe7 Qxe7
  =/+  (-0.66 --)   Depth: 13   00:00:00  21637kN
21.Rxe7 Qxe7
  -/+  (-0.78 --)   Depth: 13   00:00:00  21952kN
21.hxg5+ fxg5 22.Qh4+ Kg7 23.Bxg5 Bxg5 24.Qxg5+ Qg6 25.Re7+ Kg8 26.Qe3 Qf6 27.Rd1 Bf5 28.Rd4 Kh8 29.Bf1 b5 30.a3 Rg8 31.Bg2 Na6
  -/+  (-0.71)   Depth: 13   00:00:00  22015kN
21.hxg5+ Kg7 22.Bf4
  =/+  (-0.65 !)   Depth: 14   00:00:00  22132kN
21.hxg5+ Kg7 22.Bf4
  =/+  (-0.56 !)   Depth: 14   00:00:00  22802kN
21.hxg5+ fxg5
  =/+  (-0.65 --)   Depth: 14   00:00:00  23024kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.44 !)   Depth: 14   00:00:00  23083kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.28 !)   Depth: 14   00:00:00  23262kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Kg7 24.hxg5 fxg5 25.Qd4+ Qf6 26.Qe3 Bd7 27.Rd1 Kh8 28.Bc1 Bc6 29.Bxc6 Nxc6 30.Rd7 Ne5
  =/+  (-0.28)   Depth: 14   00:00:00  23297kN
21.Rxe7 Qxe7
  =/+  (-0.34 --)   Depth: 15   00:00:00  23416kN
21.Rxe7 Qxe7
  =/+  (-0.43 --)   Depth: 15   00:00:00  23456kN
21.Rxe7 Qxe7
  =/+  (-0.55 --)   Depth: 15   00:00:00  23542kN
21.Rxe7 Qxe7
  -/+  (-0.71 --)   Depth: 15   00:00:00  24673kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Kg7 24.hxg5 fxg5 25.Qd4+ Qf6 26.Qxf6+ Kxf6 27.Bb2 Be6 28.Be4 Bc4 29.a4 Nd7 30.a5 Ne5 31.Bxh7 Rd8 32.Be4
  -/+  (-0.72)   Depth: 15   00:00:00  24725kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.66 !)   Depth: 16   00:00:00  24952kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.58 !)   Depth: 16   00:00:00  25212kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.47 !)   Depth: 16   00:00:00  25438kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Kg7 24.Bg2 Ba6 25.Rd1 Qe8 26.Qxe8 Rxe8 27.Bc1 h6 28.hxg5 hxg5 29.f4 g4 30.Bd5 Be2 31.Re1
  =/+  (-0.37)   Depth: 16   00:00:00  26352kN
21.Rxe7 Qxe7
  =/+  (-0.43 --)   Depth: 17   00:00:00  26645kN
21.Rxe7 Qxe7
  =/+  (-0.51 --)   Depth: 17   00:00:00  26974kN
21.Rxe7 Qxe7
  =/+  (-0.62 --)   Depth: 17   00:00:00  27639kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.51 !)   Depth: 17   00:00:00  28541kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Kg7 24.Bg2 h6 25.Rd1 Bf5 26.Bc1 Re8 27.Be3 Qf7 28.Bd5 Be6 29.c4 Bxd5 30.Rxd5 gxh4 31.Qd1 hxg3
  =/+  (-0.49)   Depth: 17   00:00:00  28597kN
21.Rxe7 Qxe7
  =/+  (-0.55 --)   Depth: 18   00:00:00  29779kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.48 !)   Depth: 18   00:00:00  30684kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.37 !)   Depth: 18   00:00:00  30968kN
21.Rxe7 Qxe7 22.Ba3
  =  (-0.22 !)   Depth: 18   00:00:00  31546kN
21.Rxe7 Qxe7
  =/+  (-0.35 --)   Depth: 18   00:00:00  32333kN
21.Rxe7 Qxe7 22.Ba3
  =  (-0.17 !)   Depth: 18   00:00:00  35013kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Kg7 24.hxg5 fxg5 25.Qc2 Bf5 26.Qd2 Qf6 27.Re1 Rd8 28.Bd5 Nc6 29.c4 Nd4 30.Qe3 Nc2 31.Qe7+ Kg6 32.Qxf6+ Kxf6 33.Bb2+ Kg6 34.Re7 Rd7 35.Re8 Nb4
  =/+  (-0.35)   Depth: 18   00:00:01  36377kN
21.Rxe7 Qxe7
  =/+  (-0.41 --)   Depth: 19   00:00:01  36725kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.34 !)   Depth: 19   00:00:01  37484kN
21.Rxe7 Qxe7 22.Ba3
  =  (-0.23 !)   Depth: 19   00:00:01  38452kN
21.Rxe7 Qxe7
  =/+  (-0.33 --)   Depth: 19   00:00:01  44755kN
21.Rxe7 Qxe7
  =/+  (-0.54 --)   Depth: 19   00:00:01  46621kN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.35 !)   Depth: 19   00:00:01  47130kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Bf5 24.Rd1 Kg7 25.hxg5 Qe2 26.Bg2 fxg5 27.Rf1 Rf7 28.Bc1 Bc2 29.Qa3 Bd3 30.Bxg5 Qg4 31.Qc1 Bxf1 32.Bxf1 Rf3 33.Bh6+ Kg8
  =/+  (-0.31)   Depth: 19   00:00:01  53109kN
21.Rxe7 Qxe7 22.Ba3
  =  (-0.25 !)   Depth: 20   00:00:02  56701kN
21.Rxe7 Qxe7 22.Ba3
  =  (-0.17 !)   Depth: 20   00:00:02  59174kN
21.Rxe7 Qxe7 22.Ba3
  =  (-0.06 !)   Depth: 20   00:00:02  61424kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Kg7 24.hxg5 fxg5 25.Bd5 Bf5 26.c4 Qe2 27.Qb3 Bg4 28.Qb2+ Qxb2 29.Bxb2+ Kg6 30.Re1 Bf3 31.Re7 Bxd5 32.cxd5 Rd8 33.Re6+ Kf7 34.Rf6+ Ke8 35.Re6+ Kf8 36.Rf6+ Ke7 37.Re6+ Kf7
  =  (0.00)   Depth: 20   00:00:02  63098kN
21.Rxe7 Qxe7
  =  (-0.06 --)   Depth: 21   00:00:02  65837kN
21.Rxe7 Qxe7
  =  (-0.14 --)   Depth: 21   00:00:02  67131kN
21.Rxe7 Qxe7
  =  (-0.25 --)   Depth: 21   00:00:02  70569kN
21.Rxe7 Qxe7 22.Ba3
  =  (-0.14 !)   Depth: 21   00:00:02  72955kN
21.Rxe7 Qxe7
  =/+  (-0.28 --)   Depth: 21   00:00:03  82921kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Kg7 24.hxg5 fxg5 25.Bd5 Kg6 26.Bg2 Bf5 27.Rd1 Qe2 28.Bc1 Qxf2+ 29.Kxf2 Bc2+ 30.Qf4 gxf4 31.Rd6+ Rf6 32.Bxf4 Rxd6 33.Bxd6 Nd7 34.Ke3 Bf5 35.a3 h6 36.Bf3 Be6 37.Bf4 Nf6 38.Bb8 Nd5+ 39.Kd3
  =  (-0.22)   Depth: 21   00:00:03  91372kN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Kg7 24.hxg5 fxg5 25.Bd5 Kg6 26.Bg2 Bf5 27.Rd1 Qe2 28.Bc1 Qxf2+ 29.Kxf2 Bc2+ 30.Qf4 gxf4 31.Rd6+ Rf6 32.Bxf4 Rxd6 33.Bxd6 Nd7 34.Ke3 Bf5 35.a3 h6 36.Bf3 Be6 37.Bf4 Nf6 38.Bb8 Nd5+ 39.Kd3
  =  (-0.22)   Depth: 22   00:00:03  95842kN
21.Rxe7 Qxe7
  =/+  (-0.28 --)   Depth: 23   00:00:04  115mN
21.Rxe7 Qxe7
  =/+  (-0.36 --)   Depth: 23   00:00:05  138mN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.27 !)   Depth: 23   00:00:06  144mN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Kg7 24.hxg5 fxg5 25.Bd5 Bf5 26.Qd1 Re8 27.Bb2 Qe2 28.Qb3 Qc2 29.c4+ Kg6 30.Bf3 h5 31.Rd1 g4 32.Qxc2 Bxc2 33.Rd6+ Kh7 34.Bg2 Be4 35.Bf1 Nc6 36.Bg2 Bxg2 37.Kxg2 Re2 38.Rxc6 Rxb2
  =  (-0.23)   Depth: 23   00:00:06  146mN
21.Rxe7 Qxe7
  =/+  (-0.29 --)   Depth: 24   00:00:07  173mN
21.Rxe7 Qxe7
  =/+  (-0.37 --)   Depth: 24   00:00:07  176mN
21.Rxe7 Qxe7
  =/+  (-0.48 --)   Depth: 24   00:00:09  219mN
21.Rxe7 Qxe7 22.Ba3 c5 23.Bxa8 Bf5 24.Bg2 Nd7 25.Rd1 Ne5 26.Rd5 Rd8 27.hxg5+ fxg5 28.Rxd8 Qxd8 29.Bc1 Qe7 30.f4 Ng4 31.fxg5+ Kg6 32.Bf1 Ne5 33.Bg2 Qe6 34.Bd2 a5 35.Bf4 Nf7 36.Kh2 Nxg5 37.Qb3 h6 38.Qd5
  =/+  (-0.47)   Depth: 24   00:00:11  282mN
21.Rxe7 Qxe7
  =/+  (-0.53 --)   Depth: 25   00:00:13  349mN
21.Rxe7 Qxe7 22.Ba3
  =/+  (-0.46 !)   Depth: 25   00:00:18  489mN
21.Rxe7 Qxe7
  =/+  (-0.54 --)   Depth: 25   00:00:19  544mN
21.Be3 Bf5 22.Rad1
  =/+  (-0.35 !)   Depth: 25   00:00:19  545mN
21.Be3 Bf5 22.Rad1
  =  (-0.20 !)   Depth: 25   00:00:19  545mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 25   00:00:19  546mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 26   00:00:19  564mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 27   00:00:20  608mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 28   00:00:22  683mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 29   00:00:24  737mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 30   00:00:26  826mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 31   00:00:29  930mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 32   00:00:33  1052mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 33   00:00:37  1214mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 34   00:00:46  1530mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 35   00:01:01  2052mN
21.Be3 Bf5 22.Rad1 Qa3 23.hxg5+ fxg5 24.Qh4+ Kg6 25.Qh1 Kg7 26.Bc1 Qc5 27.Be3 Qa3
  =  (0.00)   Depth: 36   00:01:14  2509mN
Parent - - By Peter Grayson (****) [gb] Date 2017-12-23 19:04
Komodo 11.2.2 finds 32.c4 is advantageous for White in less than a minute. Therefore all we have here from Alpha Zero is 19.Re1 that engines analyse as second best. There is no evidence Alpha Zero played 19.Re1 identifying a specific advantage for White that suggests the game result was the consequence of little more than a blunder by Stockfish 8 rather than anything special by Alpha Zero. It would of course be nice to be able to determine the speed equivalence of the AZ hardware for the top engines but on the face of it, the chess capability of AZ may offer no more than todays engines on equivalent hardware.

It is easy to get carried away in the furore of this but a more down to earth approach suggests there was nothing special from AZ!

PeterG

3: AlphaZero - Stockfish,  2017
rn3rq1/p5k1/2p2bp1/1p4p1/8/2P1B1PQ/5PK1/3R3R w - - 0 1


Analysis by Komodo 11.2.2 64-bit:

32.Rd6 Be5
  -/+  (-1.12 --)   Depth: 7   00:00:00  6085kN
32.Rd6 Be5
  -/+  (-1.23 --)   Depth: 7   00:00:00  6096kN
32.Rd6 Be5
  -/+  (-1.49 --)   Depth: 7   00:00:00  8788kN
32.Bc5
  -/+  (-1.48 --)   Depth: 7   00:00:00  8790kN
32.Bc5 Rd8 33.Bd6 Kf7 34.Rd3 Qh8 35.Qg4
  -/+  (-1.42)   Depth: 7   00:00:00  8801kN
32.Bc5 Rd8
  -/+  (-1.48 --)   Depth: 8   00:00:00  8814kN
32.Qg4 Qc4 33.Qh3 Kf7 34.Qh7+ Bg7 35.Bd4 Rg8 36.Rhe1
  -/+  (-1.41)   Depth: 8   00:00:00  11533kN
32.c4 bxc4 33.Rb1 Re8 34.Qh6+ Kf7 35.Bxg5 Na6 36.Rbd1
  -/+  (-1.37)   Depth: 8   00:00:00  11553kN
32.c4 Rd8 33.Rxd8
  -/+  (-1.31 ++)   Depth: 9   00:00:00  11827kN
32.c4 Re8 33.cxb5 cxb5 34.Qh6+ Kf7 35.Bxg5 Qg7 36.Rd6 Re6 37.Rxe6
  -/+  (-1.21)   Depth: 9   00:00:00  11908kN
32.Rd6 Re8 33.c4
  -/+  (-1.17 ++)   Depth: 9   00:00:00  11924kN
32.Rd6 Re8 33.c4 Be5 34.Rdd1 bxc4 35.Bxg5 Qh8 36.Rd7+ Kg8 37.Rh7
  -/+  (-1.11)   Depth: 9   00:00:00  11941kN
32.Rd6 Re8
  -/+  (-1.17 --)   Depth: 10   00:00:00  11992kN
32.Rd6 Re8
  -/+  (-1.26 --)   Depth: 10   00:00:00  12050kN
32.Rd6 Re8
  -/+  (-1.38 --)   Depth: 10   00:00:00  12113kN
32.Rd6 Re8 33.Bd2
  -/+  (-1.21 ++)   Depth: 10   00:00:00  12264kN
32.Rd6 Re8 33.Qh6+ Kf7 34.Bxg5 Be5 35.Rd3 Qg7 36.Qh4 Kg8 37.Qe4 Qf7 38.Rf3
  -/+  (-1.39)   Depth: 10   00:00:00  17670kN
32.c4 bxc4 33.Qh6+ Kf7 34.Bxg5 Qg7 35.Qh4 Bxg5 36.Qxg5 Qf6 37.Qc5 Kg8 38.Rd6
  -/+  (-1.35)   Depth: 10   00:00:00  17936kN
32.c4 bxc4
  -/+  (-1.41 --)   Depth: 11   00:00:00  18174kN
32.Rd6 Re8 33.Qh6+
  -/+  (-1.24 ++)   Depth: 11   00:00:00  18190kN
32.Rd6 Re8
  -/+  (-1.36 --)   Depth: 11   00:00:00  18222kN
32.Rd6 Re8
  -/+  (-1.48 --)   Depth: 11   00:00:00  18505kN
32.Rd6 Re8
  -+  (-1.64 --)   Depth: 11   00:00:00  18531kN
32.Rd6 Re8 33.Rd3 Kf7 34.Qg4 Rd8 35.Bd4 Rd6 36.Rf3 Nd7 37.Qxg5 Re8
  -/+  (-1.60)   Depth: 11   00:00:00  18742kN
32.c4 g4 33.Qxg4
  -/+  (-1.45 ++)   Depth: 11   00:00:00  18756kN
32.c4 bxc4 33.Rd6 a5 34.Qh6+ Kf7 35.Rxf6+ Kxf6 36.Qxg5+ Ke6 37.Bc5 Kd7 38.Rd1+ Kc8 39.Bxf8 Qxf8 40.Qxg6 a4
  -/+  (-1.53)   Depth: 11   00:00:00  18793kN
32.c4 bxc4 33.Rd6
  -/+  (-1.47 ++)   Depth: 12   00:00:00  18998kN
32.c4 Be5
  -/+  (-1.31 ++)   Depth: 12   00:00:00  19054kN
32.c4 bxc4 33.Rd6 a5 34.Qg4 Be5 35.Rxg6+ Kxg6 36.Qxg5+ Kf7 37.Qxe5 Qg6 38.Bc5 Re8 39.Qf4+ Kg8
  -/+  (-1.39)   Depth: 12   00:00:00  19186kN
32.Rd6 Re8 33.Qh6+
  -/+  (-0.73 ++)   Depth: 12   00:00:00  19203kN
32.Rd6 Re8 33.Qh6+ Kf7 34.Bxg5 Be5 35.Rd3 Qg7 36.Qh4 Bxc3 37.Bh6 Qe5 38.Qg4 Qf5 39.Qxf5+ gxf5 40.Rxc3
  -/+  (-0.73)   Depth: 12   00:00:00  19234kN
32.Rd6 Bxc3 33.Qh6+
  =/+  (-0.67 ++)   Depth: 13   00:00:00  19366kN
32.Rd6 a5
  =/+  (-0.47 ++)   Depth: 13   00:00:00  19373kN
32.Rd6 Re8 33.Qh6+
  =  (-0.28 ++)   Depth: 13   00:00:00  19903kN
32.Rd6 a5 33.Bd4
  =  (0.00 ++)   Depth: 13   00:00:00  19909kN
32.Rd6 a5 33.Bd4 Bxd4 34.Qh6+ Kf7 35.Qxg5 Bxc3 36.Qf4+ Kg7 37.Qh6+ Kf7 38.Qf4+
  =  (0.00)   Depth: 13   00:00:00  19924kN
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kf7 37.Qf4+ Ke6 38.Qg4+
  =  (0.00)   Depth: 14   00:00:00  20078kN
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kf7 37.Qf4+ Ke6 38.Qg4+
  =  (0.00)   Depth: 15   00:00:00  20199kN
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kf7 37.Qf4+ Ke6 38.Qg4+
  =  (0.00)   Depth: 16   00:00:00  20391kN
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kf7 37.Qf4+ Ke6 38.Qg4+
  =  (0.00)   Depth: 17   00:00:00  21183kN
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kf7 37.Qf4+ Ke6 38.Qg4+
  =  (0.00)   Depth: 18   00:00:00  21473kN
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kf7 37.Qf4+ Ke6 38.Qg4+
  =  (0.00)   Depth: 19   00:00:00  22043kN
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kf7 37.Qf4+ Ke6 38.Qg4+
  =  (0.00)   Depth: 20   00:00:01  23434kN, tb=1
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kd6 37.Bf4+ Kc5 38.Be3+ Kd6
  =  (0.00)   Depth: 21   00:00:01  25735kN, tb=4
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Ke7 37.Bg5+ Kd6 38.Bf4+ Kc5 39.Be3+ Kd6 40.Bf4+
  =  (0.00)   Depth: 22   00:00:01  30271kN, tb=10
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Ke7 37.Bg5+ Kd6 38.Bf4+ Kc5 39.Be3+ Kd6 40.Bf4+
  =  (0.00)   Depth: 23   00:00:01  32362kN, tb=10
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kd6 37.Bf4+ Kc5 38.Be3+ Kd6
  =  (0.00)   Depth: 24   00:00:01  39021kN, tb=11
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kd6 37.Bf4+ Kc5 38.Be3+ Kd6
  =  (0.00)   Depth: 25   00:00:02  48624kN, tb=28
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kd6 37.Bf4+ Kc5 38.Be3+ Kd6
  =  (0.00)   Depth: 26   00:00:03  77383kN, tb=213
32.Rd6 Re8 33.Qh6+ Kf7 34.Rxf6+ Kxf6 35.Qxg5+ Ke6 36.Qg4+ Kd6 37.Bf4+ Kc5 38.Be3+ Kd6
  =  (0.00)   Depth: 27   00:00:06  136MN, tb=1473
32.c4 Re8 33.Rd6
  =  (0.06 ++)   Depth: 27   00:00:13  288MN, tb=7835
32.c4 Re8 33.Rd6
  =  (0.14 ++)   Depth: 27   00:00:14  297MN, tb=8854
32.c4 Re8 33.Rd6
  =  (0.25 ++)   Depth: 27   00:00:15  326MN, tb=10088
32.c4 Re8 33.Rd6
  +/=  (0.40 ++)   Depth: 27   00:00:17  378MN, tb=10967
32.c4 Re8 33.Rd6
  +/=  (0.61 ++)   Depth: 27   00:00:20  421MN, tb=12903
32.c4 Re8 33.Rd6
  +/-  (0.90 ++)   Depth: 27   00:00:29  628MN, tb=38586
32.c4 Re8 33.Rd6 bxc4 34.Qh6+ Kf7 35.Rxf6+ Kxf6 36.Qxg5+ Ke6 37.Rd1 Qf7 38.Qh4 Qf5 39.Bg5 Kf7 40.Qh7+ Ke6 41.Re1+ Kd5 42.Rxe8 Qxg5 43.Qc7 Kd4 44.Qg7+ Kc5 45.Re5+ Qxe5 46.Qxe5+ Kb6 47.Qd4+ Kc7 48.Qf4+ Kb6 49.Qxc4 a5 50.Qd4+ Kc7 51.Qa4 g5 52.Kh3 Nd7 53.Kg4 Nb6 54.Qd4 a4 55.Qe5+ Kb7 56.Qe7+ Ka6 57.Qa3
  +/-  (0.97)   Depth: 27   00:00:34  724MN, tb=45084
32.c4 Re8
  +/-  (0.91 --)   Depth: 28   00:00:36  761MN, tb=45932
32.c4 Re8 33.Rd6 bxc4 34.Qh6+ Kf7 35.Rxf6+ Kxf6 36.Qxg5+ Ke6 37.Rd1 Qf7 38.Qh4 Qf5 39.Bg5 Kf7 40.Qh7+ Ke6 41.Re1+ Kd5 42.Rxe8 Qxg5 43.Qc7 Kd4 44.Qg7+ Kc5 45.Re5+ Qxe5 46.Qxe5+ Kb6 47.Qd4+ Kb7 48.Qxc4 a5 49.Qf7+ Kb6 50.f4 a4 51.Qxg6 a3 52.Qb1+ Kc7 53.Qa2 c5 54.f5 Nc6 55.f6 Kb6 56.f7 Kc7 57.Qxa3 Rxa3 58.f8Q
  +/-  (0.87)   Depth: 28   00:00:43  918MN, tb=52916
32.c4 Re8 33.Rd6
  +/-  (0.93 ++)   Depth: 29   00:00:49  1057MN, tb=62146
32.c4 a5 33.Rd6
  +/-  (1.01 ++)   Depth: 29   00:00:50  1090MN, tb=63228
32.c4 a5 33.Rd6
  +/-  (1.12 ++)   Depth: 29   00:00:54  1161MN, tb=72968
32.c4 a5 33.Rd6
  +/-  (1.27 ++)   Depth: 29   00:01:02  1357MN, tb=107480
32.c4 Re8 33.Rd6 Be5 34.Bd4 Bxd4 35.Rxd4 Rd8 36.Rxd8 Qxd8 37.Qe6 Nd7 38.Rd1 Nf8 39.Rxd8 Nxe6 40.Rxa8 bxc4 41.Rxa7+ Kf6 42.Ra4 c3 43.Kf3 Ke5 44.Rc4 Kd5 45.Rxc3 Nd4+ 46.Kg4 Ne6 47.Rd3+ Ke5 48.Ra3 c5 49.Kf3 c4 50.Ra6 Kd5 51.Ke3 g4 52.Kd2 Nc5
  +/-  (1.22)   Depth: 29   00:01:05  1403MN, tb=110311
32.c4 Re8 33.Rd6
  +/-  (1.28 ++)   Depth: 30   00:01:10  1535MN, tb=120840
32.c4 Re8 33.Rd6
  +/-  (1.36 ++)   Depth: 30   00:01:13  1608MN, tb=124324
Parent - - By Banned for Life (Gold) Date 2017-12-23 20:23 Edited 2017-12-23 20:28
Peter,

AZ used a totally different paradigm and beat the very mature SF8. There is no reason to believe that AZ is mature, and in fact, it is likely it can be significantly improved, as DeepMind's go engine was. It seems inevitable that NN based engines will rule the roost in the future. The main obstacle is that only DeepMind (and not Google Brain! :razz:) has demonstrated that they understand how to structure the problem to allow the requisite machine learning tabula rasa, and they don't seem inclined to share the details (which is not surprising).

As far as hardware differences are concerned, the race is on to develop better hardware to support machine learning and inference. Although Google seems to be in front at this point with its second generation TPUs, there is no reason to believe that NVIDIA, AMD, Intel and others will not be able to compete. Since this market is relatively new, it is almost certain that the rate of progress will be much faster than the traditional CPU market, and that the cost per unit performance will fall much faster.

Furthermore, the discussion of unequal hardware is kind of a red herring anyway, since the machine used for hosting SF8 was within a small factor of optimal...
Parent - - By Peter Grayson (****) [gb] Date 2017-12-23 21:30
Alan,

  I cannot dispute what you say or that the way forward for chess engines is the Alpha Zero model. However, what I was able to demonstrate was that on relatively modest hardware, the Alpha Zero engine offered nothing that is not currently achievable with today's chess engines. The hardware may affect the time to achieve a similar output but having established Stockfish 8 was seeing most of Alpha Zero's replies and Ed's suggestions of moves that were special being played within 60 seconds by Houdini and Komodo, I could find nothing that set out Alpha Zero apart from today's top engines as being something particularly special. Similarly so with AZ's other games.

The AZ performance may have looked more creditable if some information on the engine's move scoring and plydepths achieved had been published with the games that would have given a better indication of knowledge base and search depth.

Peter
Parent - - By Chris Whittington (**) [fr] Date 2017-12-23 22:12
You absolutely did not demonstrate AZ achieved no more than current engines.
You presented a huge amount of data and some conclusions not concludable from the data.
Parent - - By Peter Grayson (****) [gb] Date 2017-12-24 02:23

> You absolutely did not demonstrate AZ achieved no more than current engines.


You must be looking at something completely different to me. In the game Ed posted it was clearly demonstrated AZ's moves were playable by today's top engines under the game criteria.

You have criticised my opinion supported by the posted data but you supply no data to support your criticism. When you are able to do so, your criticism and thus supported argument may be of more interest to me.
Parent - - By Chris Whittington (**) [gb] Date 2017-12-24 10:56
Your varied conclusions:
"AZ heavily influenced by Stockfish"
"AZ achieve no more than current engines"
"nothing special from AZ"
"no evidence AZ identified advantage from Re1 .... game result was blunder by Stockfish"

are just plain wrong and/or without logic.
point one disputes the veracity of the paper, and is too ridiculously wrong to even begin to list reasons. You may as well have concluded "homeopathy is true" or "man eats fish"; there simply isn't the connection in logic from data to conclusion, meanwhile there is the paper which tells you how AZ is trained, specifying self play and FIDE rules known only.
point two not even wrong, again.
point three, flies in face of results, style and so on. More not even wrong.
point four "no evidence" means "you don't have the evidence". The conclusion can't  be drawn.

Lastly, it is not possible to prove AZ is not an outstanding improvement/paradigm shift on AB programs using AB program rollouts or best line analysis. That should be more than obvious.
Parent - By Peter Grayson (****) [gb] Date 2017-12-24 12:50

> meanwhile there is the paper which tells you how AZ is trained, specifying self play and FIDE rules known only.


Indeed the paper reads that way but special consideration should be given to the brief comment on page 4 of the published paper ...
"In chess, AlphaZero outperformed Stockfish after just 4 hours (300k steps)" suggesting the AZ engine was also being run against Stockfish 8 to measure its performance that would have continued until the testing confirmed AZ was stronger than SF8. It would not have made commercial sense playing the match with a weaker than SF8 engine and thus losing it. We would likely be ignorant the match ever took place.

>"AZ heavily influenced by Stockfish"


For SF8 to have such a high ponder hit rate predicting AZ's moves suggests either the game had a limited number of sensible deviant moves or there was similarity in the predictability of moves. The above comment may also be an idicator.

>"no evidence AZ identified advantage from Re1 .... game result was blunder by Stockfish"


We have no evidence of AZ's scoring at all.

One thing having worked 44 years in measurement and control engineering in the steel industry taught me was do not take everything "experts" tell you as being gospel, especially when commercial interests are involved. When we are not allowed to question and query it is time to move on to something else.

> Lastly, it is not possible to prove AZ is not an outstanding improvement/paradigm shift on AB programs using AB program rollouts or best line analysis. That should be more than obvious.


Neither is it currently possible to prove AZ is an outstanding improvement/paradigm shift on AB program. All we have is 10 games selected from a batch of 100 games and without any indication of engine scores, ply depths and anticipated move continuations.

Your remaining comments probably fall into the category of being carried along on the wave of enthusiasm that should be tempered by other chess and programming experts views on the matter that are contrary to yours. I fall into the "doubting Thomas" category but am prepared to be swayed by overwhelming evidence. I see none to date.
Parent - - By MarshallArts (***) [us] Date 2017-12-24 05:04

> Therefore all we have here from Alpha Zero is 19.Re1 that engines analyse as second best.


Not all engines:

http://www.talkchess.com/forum/viewtopic.php?topic_view=threads&p=744707&t=66110

:slim:
Parent - - By Peter Grayson (****) [gb] Date 2017-12-24 11:13
Thanks very much for the link. Interesting that others are finding AZ's "special" moves with their software/hardware set up and noted strong scoring by "Zero-the-Hero"?

In the search for that it lead me to an interesting article

https://www.theregister.co.uk/2017/12/14/deepmind_alphazero_ai_unfair/

when the quoted experts doubt the validity of the results. I still see this as much ado about nothing and probably more of a hardware and conditions optimisation for AZ. It is a pity the Google people did not communicate and work with the Stockfish developers before this match announcement came out of the blue. I still find it strange that people create so much furore over just 10 published games with no eninge output details that could give us some idea of comparative AZ strength.
Parent - - By Chris Whittington (**) [gb] Date 2017-12-24 11:54
Fake News.
That's an article with nothing to add, other than repeating another article by Collados from over a week ago. Collados article is based on "seems like", "seems odd that", and "it's not clear that" and similar bla bla. His  "not clear that"'s would be perfectly clear if he had bothered to understandingly read the original DeepMind paper. Really bad, unsoundly speculative, lazy article from Collados. Unsurprising though that Fake News journalism writes more articles based on it. False unsound reading of data, repeated 100 times by quoting or re-spreading is still false unsound reading of data.
Parent - By Peter Grayson (****) [gb] Date 2017-12-24 14:11
The released paper describing the method is fine albeit with some hints there may have been more in the AZ tuning method than meets the eye but the proof of the method is surely the results and here we have just 10 games with no information on the moves. Without knowing the fine detail of the sparse game releases amounting of those 10 games it is little wonder that comment from both sides is speculative and subjective to say the least.

Therefore comment praising or criticising in nature can only be subjective but is no more fake news than any other speculative article where a correspondent expresses an opinion based on the presented, very sparse evidence.
Parent - - By Banned for Life (Gold) Date 2017-12-23 20:00 Edited 2017-12-23 20:43
Ed,

1. Nf3 Nf6 2. d4 e6 3. c4 b6 4. g3 Bb7 5. Bg2 Be7 6. O-O O-O 7. d5 exd5 8. Nh4 c6 9. cxd5 Nxd5 10. Nf5 Nc7 11. e4 d5 12. exd5 Nxd5 13. Nc3 Nxc3 14. Qg4 g6 15. Nh6 Kg7 16. bxc3 Bc8 17. Qf4 Qd6 18. Qa4 g5 19. Re1 Kxh6 20. h4 f6 21. Be3 Bf5 22. Rad1 Qa3 23. Qc4 b5 24. hxg5 fxg5 25. Qh4 Kg6 26. Qh1 Kg7 27. Be4

A long backward analysis of this game with SF8 indicates that 27... Bg6?? is a serious error. SF8 backward eval after this move is over 1.5.

As an alternative, after 27... Bxe4 28. Qxe4 Kg8 29. Qe6+ Rf7 30. Qg4 Nd7 black seems likely to achieve a draw (SF8 eval is about 0.25). Of course it is possible that SF8 is overlooking some brilliancy from AZ that would destroy this line...
Parent - By Lazy Frank (***) [gb] Date 2017-12-23 21:12
:yell:
Yes, and all A0 brilliancies before goes down in the hell! Don't take off people from dreams!
Up Topic The Rybka Lounge / Computer Chess / AZ vs SF - game 99
1 2 3 Previous Next  

Powered by mwForum 2.27.4 © 1999-2012 Markus Wichitill