To realize their own tensorflow (i)-calculation diagram and forward propagation

Objective

Some time ago because the subject needs to use a period of time tensorflow, feel this framework is very interesting, in addition to can build complex neural network, but also can optimize the other needs of the calculation model, so I always want to learn to write a similar diagram calculation framework. In the last few days, the group will finish the decision to implement a rough version diagram computing framework that mimics the TensorFlow interface to learn the programming of the diagram program and the implementation of forward propagation and reverse propagation. The forward propagation and reverse propagation as well as the gradient descent optimizer are implemented, and an example of an optimized linear model is written.

The code is placed on the GitHub, named Simpleflow, Warehouse Link:

     
     
      
      
Https://github.com/PytLab/simpleflow
     
     

Although it is not very complicated to understand the principles of forward propagation, it is true to write a handwritten discovery that there are many details that need to be learned and processed to optimize the actual model (e.g. the loss function handles the derivation of each compute node matrix). The Simpleflow code does not take into account too many things such as dtype and tensor size checks, because only in order to achieve the main diagram of the calculation function does not consider any optimization, internal tensor operations using the NumPy interface (after all, the purpose of learning and practicing).

For a long time did not update the blog, in the next few inside I will be the realization of the details of the process to sum up, I hope to be able to learn from behind the children's shoes to make a reference.

Body

This paper mainly introduces the realization of calculation graph and forward propagation, mainly involves the construction of graphs and the calculation of the output values on the concrete nodes by the sequence traversal of the constructed graphs and the forward propagation.

First put a simple implementation of the effect bar:

     
     
      
      
Import Simpleflow as SF
      
      

      
      
# Create A graph
      
      
With SF. Graph (). As_default ():
      
      
A = Sf.constant (1.0, Name= ' a ')
      
      
b = sf.constant (2.0, name= ' B ')
      
      
result = Sf.add (A, B, name= ' result ')
      
      

      
      
# Create a session to compute
      
      
With TF. Session () as Sess:
      
      
Print (Sess.run (result))
     
     

Calculation diagram (computational graph)

The calculation graph is an elementary processing method in computational algebra, we can represent a given mathematical expression by a direction diagram, and can quickly and conveniently differentiate the variables in the expression according to the characteristics of the graph. The essence of neural network is a multi-layer compound function, so it can also be expressed by a graph.

This part mainly summarizes the realization of the calculation diagram, in the graph of the map, each node represents a specific operation such as summation, product, vector product, square and so on ... For example, the sum expression f (x, y) is represented by a =x+y graph:

The expression F (x,y,z) =z (x+y) is represented by a mapping to:

Unlike TensorFlow implementations, in order to simplify, in Simpleflow I do not define the tensor class to represent the data flow between nodes in a calculation diagram, but rather directly define the type of the node, which mainly defines four types to represent the nodes in the diagram:

1, Operation: Operation node mainly accept one or two input nodes and then perform simple operations, such as the addition of the above figure and multiplication operations.

2, Variable: No input node node, this node contains data in the operation can be changed.

3, Constant: Similar to variable node, there is no input node, the data in this node in the calculation of the diagram will not change

4, placeholder: There is also no input node, this node data is established through the graph after the user passed in

In fact, all the nodes in the diagram can be regarded as some kind of operation, in which variable, Constant, placeholder are a special operation, but compared with ordinary operation, they have no input, but there will be output (like the XX, yy node in the above image, They themselves output their own values to the + + node, and are typically exported to the Operation node for further computation.

Here we mainly introduce how to implement the basic components of a calculation diagram: nodes and edges.

Operation Node

The node represents the operation, the edge represents the data that the node receives and outputs, and the operation node needs to contain the following attributes:

1, Input_nodes: input node, which holds the connection with the current node of the input node reference

2, Output_nodes: Output node, stored with the current node as input node, that is, the current node's whereabouts

3, Outputvalue: Store the current node value, if it is the Add node, this variable stores two input nodes Outputvalue and

4, Name: Names of the current node

5, Graph: This node belongs to the diagram below we define the operation base class to represent the operation nodes in the diagram (see https://github.com/PytLab/simpleflow/blob/master/simpleflow/ operations.py):

     
     
      
      
Class Operation (object):
' Base class for all operations in Simpleflow.
An operation was a node in computational graph receiving zero or more nodes
As input and produce zero or more nodes as output. Vertices could is an
operation, variable or placeholder.
'''
def __init__ (self, *input_nodes, Name=none):
"Operation constructor."
:p Aram Input_nodes:input nodes for the Operation node.
: Type input_nodes:objects of ' Operation ', ' Variable ' or ' placeholder '.
:p Aram Name:the operation name.
: Type Name:str.
'''
# Nodes received by this operation.
Self.input_nodes = Input_nodes
 
# Nodes that receive this operation node as input.
Self.output_nodes = []
 
# Output value of this operation to session execution.
Self.output_value = None
 
# Operation name.
Self.name = Name
 
# Graph the operation belongs to.
Self.graph = Default_graph
 
# ADD This operation node to destination lists into its input nodes.
For node in Input_nodes:
Node.output_nodes.append (self)
 
# ADD This operation to default graph.
Self.graph.operations.append (self)
 
def compute_output (self):
"" Compute and return the output value of the operation.
'''
Raise Notimplementederror
 
def compute_gradient (self, Grad=none):
"' Compute and return the gradient of the operation WRT inputs.
'''
Raise Notimplementederror

In addition to defining the properties mentioned above in the initialization method, two actions are required:

1. Add the current node's reference to the output_nodes of his input node so that the current node can be found in the input node.

2, the current node to add the reference to the diagram, easy to face the map of the resources to carry out operations such as recycling

In addition, there are two necessary methods for each operation node: Computoutput and Computegradient. They are responsible for calculating the output value of the current node based on the value of the input node and calculating the gradient based on the operation properties and the value of the current node. The calculation of the gradient will be described in detail in the following article, only the calculation of the node output value is introduced in this paper.

Below I want to and operation as an example to illustrate the implementation of the specific Operation node:

     
     
      
      
Class Add (Operation):
' An addition operation.
'''
def __init__ (self, x, Y, Name=none):
"Addition constructor."
:p Aram X:the.
: Type x:object of ' Operation ', ' Variable ' or ' placeholder '.
:p Aram Y:the Second input node.
: Type y:object of ' Operation ', ' Variable ' or ' placeholder '.
:p Aram Name:the operation name.
: Type Name:str.
'''
Super (self.__class__, self). __init__ (x, y, Name=name)
 
def compute_output (self):
"' Compute and return the value of addition operation.
'''
X, y = self.input_nodes
Self.output_value = Np.add (X.output_value, Y.output_value)
Return Self.output_value

Thus, the prerequisite for calculating the value of the current node Output_value is that the value of his input node has been computed before.

Variable node

Similar to the Operation node, the variable node also needs Outputvalue, Outputnodes, and so on, but it has no input node and there is no inputnodes attribute. Instead, you need to determine an initial value when you create it initialvalue:

     
     
      
      
Class Variable (object):
"' Variable node in computational graph.
'''
def __init__ (self, initial_value=none, Name=none, trainable=true):
"Variable constructor."
:p Aram Initial_value:the Initial value of the variable.
: Type Initial_value:number or a ndarray.
:p Aram Name:name of the variable.
: Type Name:str.
'''
# Variable initial value.
Self.initial_value = Initial_value
 
# Output value of this operation to session execution.
Self.output_value = None
 
# Nodes that receive this variable node as input.
Self.output_nodes = []
 
# Variable name.
Self.name = Name
 
# Graph the variable belongs to.
Self.graph = Default_graph
 
# ADD to the currently active default graph.
Self.graph.variables.append (self)
If trainable:
Self.graph.trainable_variables.append (self)
 
def compute_output (self):
"" Compute and return the variable value.
'''
If Self.output_value is None:
Self.output_value = Self.initial_value
Return Self.output_value

Constant nodes and placeholder nodes

The constant and placeholder nodes are similar to the variable nodes, as detailed in the implementation details:

     
     
      
      
https://github.com/PytLab/simpleflow/blob/master/simpleflow/operations.py
     
     

Calculation Diagram Objects

After defining the nodes in the diagram, we need to put the defined nodes in a graph for unified custody, so we need to define a graph class to hold the nodes created to facilitate the unification of the resources of the nodes in the Operation diagram.

     
     
      
      
Class Graph (object):
      
      
' Graph containing all computing nodes.
      
      
'''
      
      
def __init__ (self):
      
      
"Graph constructor."
      
      
'''
      
      
Self.operations, self.constants, self.placeholders = [], [], []
      
      
Self.variables, self.trainable_variables = [], []
     
     

To provide a default diagram, create a global variable to reference the default diagram when importing the Simpleflow module:

     
     
      
      
From. Graph Import Graph
      
      

      
      
# Create a default graph.
      
      
Import Builtins
      
      
Default_graph = Builtins. Default_graph = GRAPH ()
     
     

To mimic the TensorFlow interface, we add a Context Manager protocol method to graph to make it a context manager, and also add a As_default method:

     
     
      
      
Class Graph (object):
#...
def __enter__ (self):
"Reset default graph."
'''
Global Default_graph
Self.old_graph = Default_graph
Default_graph = Self
return self
 
def __exit__ (self, exc_type, Exc_value, EXC_TB):
"" Recover default graph.
'''
Global Default_graph
Default_graph = Self.old_graph
 
def as_default (self):
' Set this graph as global default graph.
'''
return self

This saves the old default diagram object before entering the with code block and assigns the current diagram to the global Diagram object, so that the nodes in the with code block are added to the current diagram by default. Finally, when you exit the with code block, you can restore the diagram. This allows us to create nodes in a diagram in a tensorflow way.

Ok, according to the above implementation we can already create a diagram:

     
     
      
      
Import Simpleflow as SF
      
      

      
      
With SF. Graph (). As_default ():
      
      
A = Sf.constant ([1.0, 2.0], name= ' a ')
      
      
b = sf.constant (2.0, name= ' B ')
      
      
c = A * b
     
     

Forward propagation (Feedforward)

The calculation diagram and the node in the graph are realized, and the forward propagation of the calculation graph is summarized in this part.

Session

First, we need to implement a session to compute a calculated diagram that has already been created, because when we create the node we defined before, we simply create an empty node, and there is no value in the node to compute, that is, the output_value is empty. To emulate the TensorFlow interface, we also define the session as a context manager here:

     
     
      
      
Class session (Object):
' A session to compute a particular graph.
'''
def __init__ (self):
"" Session constructor.
'''
# Graph the session computes for.
Self.graph = Default_graph
 
def __enter__ (self):
"' Context management Protocal method called before ' With-block '.
'''
return self
 
def __exit__ (self, exc_type, Exc_value, EXC_TB):
' The context management Protocal method called after ' With-block '.
'''
Self.close ()
 
def close (self):
' Free all output values in nodes.
'''
All_nodes = (self.graph.constants + self.graph.variables +
Self.graph.placeholders + self.graph.operations +
Self.graph.trainable_variables)
For node in All_nodes:
Node.output_value = None
 
def run (self, operation, Feed_dict=none):
"' Compute the output of an operation. '
# ...

Calculate the output value of a node

Above we can build a calculation diagram, the calculation diagram of each node and its adjacent nodes have a directional connection, now we need to according to the relationship between the nodes in the graph to calculate the value of a node. So how to calculate it? Or, for example, with the x,y,z of our just F (x+y) =z,

If we need to compute the output value of the orange X operation node, we need to compute the output value of the two input nodes connected to it, and then we need to compute the output value of the Green + input node. We can get the output value of all the nodes needed to compute a node by subsequent traversal. In order to facilitate implementation, the subsequent traversal I directly use a recursive way to achieve:

 
 Def _get_prerequisite (operation): 
 
     ' Perform a post-o
      
      Rder traversal to get a list of nodes to is computed in order. 
 
     ' 
 
    postorder_nodes = [] 
 
 
 

      
         # Collection nodes recursively. 
 
    def Postorder_traverse (operation): 
 
        if isinstance (oper
      
      ation, Operation): 
 
            for Input_node in Operation.input_nodes: 
 
                postorder_traverse (Input_node) 
    
        postorder_nodes.append (operation) 
 
 
 
     Postorder_traverse (operation) 
 
 
 
    return postorder_nodes 

Through this function we can get a list of all the nodes needed to compute a node value, then compute the output value of the node in the list, and then we can easily calculate the output value of the current node.

     
     
      
      
Class session (Object):
# ...
def run (self, operation, Feed_dict=none):
"" Compute the output of an operation.
:p Aram Operation:a specific operation to be computed.
: Type operation:object of ' operation ', ' Variable ' or ' placeholder '.
:p Aram Feed_dict:a Mapping between placeholder and its actual value for the session.
: Type Feed_dict:dict.
'''
# Get all prerequisite nodes using postorder traversal.
Postorder_nodes = _get_prerequisite (operation)
For node in Postorder_nodes:
If type (node) is placeholder:
Node.output_value = Feed_dict[node]
else: # Operation and variable
Node.compute_output ()
Return Operation.output_value

Example

Above we implemented the calculation diagram and forward propagation, we can create a calculation diagram to calculate the value of the expression, as follows:

     
     
      
      
Import Simpleflow as SF
      
      

      
      
# Create A graph
      
      
With SF. Graph (). As_default ():
      
      
W = Sf.constant ([[1, 2, 3], [3, 4, 5]], name= ' W ')
      
      
x = Sf.constant ([[9, 8], [7, 6], [Ten, one]], name= ' x ')
      
      
b = sf.constant (1.0, ' B ')
      
      
result = Sf.matmul (w, x) + b
      
      

      
      
# Create a session to compute
      
      
With SF. Session () as Sess:
      
      
Print (Sess.run (result))