Mastering the game of Go with deep neural networks and tree search

Silver, David, et al. "Mastering the game of Go with deep neural networks and tree search." Nature 529.7587 (2016): 484-489.

Alphago's thesis, the main use of the RL technology, do not know before the use of RL to do Weiqi.

Proposed two networks, one is the strategy network, one is the value network, all through the self-battle realization.

Policy Network:

The strategy network is given the current checkerboard and historical information, giving the probability of each position in the next step. Previous people seem to be using chess to do supervised training, here with RL instead, it seems that the effect than supervised training better. Parameter initialization of a policy network is a parameter with supervised training network.

Value network:

The value network is given the current chessboard and historical information, giving the advantage probability to one's own. It was originally used to replace Monte Carlo's stochastic simulations, but it was found that the combination of the value network and the stochastic simulation estimate was better. Personally think if the value of the network if the training is good enough, perhaps there is no need to simulate the estimate. Of course, the simulation here is not completely random, it seems that a supervised training out of the shallow network to simulate chess.

The strategy network can reduce the width of the Monte Carlo search tree, and the value network reduces its depth.

The paper defeated the human professional for the first time (five-section fan Hui)

In addition, the method has a distributed version and stand-alone version, the official to stand alone version of the judge is and fan Hui A level, distributed version can reach more than 5 professional level. The distributed version uses 40 search threads, 1,202 CPUs, and 176 GPUs. Stand-alone edition is 40 search threads, 48 CPUs and 8 GPUs. According to this configuration, it should be within 10 years, a single laptop can run a professional 3 or more of the Go program, which is a good news for Weiqi learners.

Alphgo Let RL fire, let go fire, let Ke jie fire, power or huge. Go more easily formalized, the rules are relatively simple, but the search space is a bit large, but there are many problems in the reality of complex rules, incomplete information, large state space, large decision-making space, the need for joint decision-making. Alphago is still developing, and there should be papers to follow.

Mastering the game of Go with deep neural networks and tree search