Uber open source "neural evolution" visualization tool Vine__uber

Albert AI Tech Review: The improvement of the calculation force may inject vigor into the old algorithm. Over the past two years, the neural Evolution (neuroevolution) approach has been gradually renewed attention, including OpenAI, DeepMind, Google
Several global research institutes such as Brain, sentient and Uber have recently studied in this area, and Uber seem to have put more effort into it.

Figure 1. The change of "neuroevolution" in Google trend

Albert had a deep interpretation of the 5 articles on neural evolution that Uber several days in the previous period. These articles introduce their findings on neural evolution ideas such as genetic algorithms (genetic algorithm), mutation methods (mutation) and evolutionary strategies (evolution strategies), And through the use of neural evolution methods in the depth of intensive learning training to better than SGD and strategic gradient. This work proclaims that the neural evolution approach has entered a new era.

Based on the work of Uber, many researchers have begun to shift their gaze from the SGD or strategic gradient approach to neural evolution. This also promotes the need for relevant tools to some extent, especially in the process of neural evolution, how to observe the evolution process dynamically and visually. To address these issues, Uber developed an interactive data visualization tool for neural evolution--visual Inspector for Neuroevolution (VINE), which was open source yesterday.

Figure 2. Visualization of Vine

To introduce how VINE is visualized, here we need to briefly introduce the basic ideas of neural evolution.

The concept of neural Evolution (neuroevolution) was first proposed in the 80 's, and its basic idea is to learn from the biological evolution process: mutation-> selection-> reproduction-> mutation ... In the field of machine learning, the traditional approach is for researchers to design a neural network based on their own experience, it is then validated by experiments, and the idea of neural evolution is to initialize a group (for example, 100) with slightly different networks, by training, by screening (and also by synthesizing) some of the standard individuals, It then replicates (for example) 100 copies, adding some random disturbances (similar to mutations in biological evolution) to these replica networks, then retraining and then sifting back and forth until the desired network is filtered out.

As can be seen from the instructions above, whether it is initialized, the choice of difference or the definition of random disturbances, or the selection process and standard formulation, can be based on the needs of the experimenter, so the evolution of a variety of methods, such as the previous mentioned genetic algorithm (GA), mutation Method (M) and evolutionary Strategy (ES). We take ES as an example and choose Uber commonly used Mujoco humanoid gait task specifically to VINE how to visualize the process of neural evolution.

Figure 3. Using genetic algorithm (left) and evolutionary strategy (right) to train the simulated robot to walk

In traditional ES applications (such as OpenAI), a set of neural networks called pseudo descendant Clouds (pseudo-offspring Cloud,poc) is usually used to optimize the specific goals of this generation. In particular, the parameters of each individual neural network in the POC are generated by randomly disturbing the parameters of a single "parent" neural network, and then the neural network of each pseudo descendant is trained and evaluated according to the target. In the humanoid gait task, each pseudo descendant neural network controls the movement of a Mujoco robot, scoring the fitness of each network based on the quality of the robot's Walk (called fitness). ES are used to aggregate pseudo descendant parameters based on these adaptive scores (which seems to be a bit like multiple (double) cross Fusion) to build the next father. and then cycle. Basic use of vine

If you want to use VINE, you must record the behavior characteristics of each father and all the pseudo descendants in the evolution process. Here a behavioral trait can be any behavioral indicator of an agent interacting with the environment. For example, in Mujoco, we can simply use the end position of each agent {X,y} as its behavioral feature, and the endpoint position represents the distance that the agent can move from the origin.

With information about behavioral characteristics, you can use visual tools to map father and pseudo descendants to a 2-D plane. To this end, VINE calls the graphical user interface (GUI), whose main component consists of two classes of interrelated graphs: one or more pseudo descendant cloud images (on a separate 2-D plane); a fitness graph

As shown in the following figure. Pseudo-descendant cloud images show the behavioral characteristics of each generation's father and pseudo descendant, while the fitness chart shows how each generation's father's fitness score curve changes.

Figure 4. Pseudo-descendant cloud and fitness diagram examples

In addition to visualizing the pseudo descendant cloud and the father's fitness situation, in VINE, users can also interact with these graphs to explore the overall trend of pseudo descendant clouds and any individual behavior of a father or pseudo descendant in evolution: Users can choose to visualize their father, best performer, and any generation of pseudo descendant clouds, The number and spatial distribution of pseudo descendant cloud with different fitness score in two-dimensional behavior feature plane are explored. Users can also compare between generations to see how a father or pseudo descendant cloud moves on a 2-dimensional behavioral feature plane, and the relationship between them and the fitness score curve in a visual way. As shown in Figure 5, you can even automatically generate a full image clip of the mobile cloud.

Clicking on any point on the cloud can show the behavioral characteristics and fitness scores of the corresponding pseudo offspring.

Figure 5. Visualize the evolution of generational behavior. The colors of each generation change; In a generation, the color strength of each pseudo descendant is based on the percentile of the adaptive score in the generation. user-defined

Of course, the VINE tool also supports some advanced options. For example, we talked about choosing the endpoint position {X,y} as a behavioral feature, if extended a bit, for example, if we split each agent into 1000 time steps from the start point to the endpoint, and record the position of each time node, then we can draw a behavioral eigenvector that records The trajectory of the agent.

Of course, in this case, the behavioral characteristics of the dimension is no longer 2 dimensions, this time need to reduce the dimension of behavioral characteristics of the data dimension to 2 dimensions, such as PCA technology or T-sne technology. In VINE, this process will be completed automatically.

The GUI, of course, can load multiple sets of 2-D behavior features (generated by different reduction techniques) and display them in simultaneous and connected cloud images, as shown in Figure 6. This feature provides a convenient way for users to explore different behavioral characteristics and dimension reduction methods.

Figure 6. Visualize multiple 2D BC and a high dimensional BC as well as a fitness plot.

In addition, users can extend basic visualization by customizing the functionality. In Figure 6, a custom cloud is temporarily displayed, in which you can show high dimensional behavioral characteristics (such as the complete trajectory of the agent) and reduced 2-dimensional behavioral characteristics for certain types of specific areas. Figure 7 is another example of a custom cloud chart that allows users to replay the deterministic/random behavior that the agent produces interactively with the environment.

Figure 7. Vine allows users to view video of deterministic/random behavior generated by any agent.

Of course, as mentioned earlier, VINE is not only used in Mujoco tasks or ES algorithms.

Figure 8 shows a classic game frostbite that applies VINE to the Atar 2600 game, where the behavioral feature is the final state emulator RAM State (an integer value vector that captures the length of all state variables in the game) and uses PCA to map behavioral features to 2 On the sooth surface.

Figure 8. Visual Agent Learning Frostbite

From this diagram, we can observe that as the evolution progresses, the pseudo descendant cloud is moving gradually to the left and appears to converge. By looking at the video of these agents playing this game, we can infer that each cluster corresponds to a semantically meaningful different end state.

In addition, VINE can collaborate seamlessly with other neural evolutionary algorithms, such as the aforementioned genetic algorithms. In fact, the tool is independent of any particular neural evolution algorithm, and what the user needs to do is to modify his neural evolution code slightly to preserve the behavioral characteristics it wants to understand.

Open Source Address: https://github.com/uber-common/deep-neuroevolution/tree/master/visual_inspector

Viauber Engineering Blog, Albert (public number: Yue Move intelligence) AI Technology Review compilation

Http://www.aibbt.com/a/29328.html