Policy Networks and Value Networks

Search Instead of Instant Answers

AlphaGo did not simply ask one neural network to name the best move. When it had to play, it repeatedly explored possible continuations from the current board position. The search began with a root representing the current game state, gradually added promising possibilities, and then used the statistics collected in that partial tree to select the move.

One MCTS Iteration

Monte Carlo Tree Search is a repeated search process over a partial game tree. During one iteration, the system explores a promising node, adds children to the tree, simulates play, and backs up information through the edges that were traversed. The iteration adds information to the search rather than requiring the complete game tree to be constructed in advance.

• Selection: identify a promising node in the current partial tree.
• Expansion: add children to that node.
• Simulation: simulate play from the newly expanded part of the tree.
• Backpropagation: carry information from the simulation back through the traversed edges.

A Tree That Learns Through Repetition

MCTS repeats its iterations while computational resources remain. Each iteration adds information to the partial game tree. The search therefore does not make its decision from a single simulation; it builds up statistics through repeated exploration. When the search stops, the accumulated statistics at the root are used to choose a move.

Rollouts and Learned Values

AlphaGo used a modified search called APV-MCTS. APV-MCTS combined two kinds of information about leaf states: information from rollout results and learned value estimates. A rollout supplies information by simulating play, while the value network evaluates positions at the leaves used by the modified search. Combining these sources gives the search more than a single kind of evidence for judging a candidate continuation.

Four Neural Network Responsibilities

AlphaGo used four distinct deep convolutional neural networks. The networks did not all perform the same job, and they were not all directly invoked by APV-MCTS during live move selection. Their weights had been trained before live play and remained fixed during the matches described by the source.

Choosing the Root Move

The final move is selected from the root of the partially explored tree. AlphaGo repeatedly runs the modified search from the current board, records statistics for the edges leaving the root, and stops when its available computational resources have been used. In the procedure described by the source, the move played is the action associated with the most visited edge leaving the root.

Common Misreadings

• Treating MCTS as a complete search of the entire game tree.

  The source describes MCTS as repeated search over a partial game tree. The tree gradually adds promising possibilities.

  Fix: Think of MCTS as an incrementally expanded tree whose statistics improve through repeated iterations.

• Treating the policy network and value network as interchangeable.

  The rollout policy network selects moves during simulated games, while the value network evaluates positions at search leaves.

  Fix: Associate policy networks with move prediction or selection and the value network with position evaluation.

• Assuming all four AlphaGo networks are directly called by APV-MCTS during live play.

  The RL policy network helped train the value network rather than being directly invoked by APV-MCTS.

  Fix: Separate training roles from live-search roles.

• Choosing the move from one simulation outcome.

  MCTS repeats iterations and uses accumulated root statistics when the search stops.

  Fix: Trace how information is backed up over repeated iterations, then inspect the root's statistics.

Practice the Trace

Search, Evaluation, Selection

1. MCTS repeatedly explores a partial game tree through selection, expansion, simulation, and backpropagation. APV-MCTS combines rollout information with learned value estimates from leaf states. AlphaGo's SL policy network provided move priors, its rollout policy network selected moves during simulated games, its value network evaluated leaf positions, and its RL policy network helped train the value network. After repeated search, the move played was associated with the most visited edge leaving the root.