How to implement your own TensorFlow in C + +

Original: How to Write Your Own TensorFlow in C + +
Author: Ray Zhang
No, I fly

Absrtact: TensorFlow is the second generation of AI learning system based on Distbelief, whose name originates from its own operating principle, it is completely open source, and the author expounds How to realize his tensorflow with C + + through a small project of his own. This article may seem a bit obscure, and you need to know something about it. The following is the translation.

Before we start, here's the code: Branch with Eigen backend Branch this only supports scalars

I worked on this project with Minh Le. why.

If you are a CS Professional staff, may have heard the phrase "do not make yourself into _" words countless times. CS has encryption, standard library, parser and so on. I think we should also include the ML library now.

No matter what the facts are, it is still an amazing lesson worth learning. People now think that TensorFlow and similar libraries are taken for granted; take them as a black box and let it run. Not many people know what's going on backstage. This is really a non convex optimization problem. Don't stop stirring the pile until it looks right (with the following diagram and machine learning system knowledge to understand this sentence).

TensorFlow

TensorFlow is a deep learning library, open source by Google. In the TensorFlow kernel, there is a large component that strings the action together and acts as a thing called an operator graph . This operation Fu Tu is a g= (v,e) G = (v,e), in some nodes u1,u2,..., un,v∈v u_1,u_2,..., u_n,v \in V and E1,e2,..., en∈e,ei= (ui,v) e_1,e_2,..., e_n \in E, E_ i = (u_i,v) There are some operators that map U1,..., un u_1,..., u_n to v v.

For example, if we have x + y = z, then (x,z), (y,z) ∈e (x,z), (y,z) \in E.

This is useful for evaluating an arithmetic expression. We can get the result by looking for the sinks in the operator graph. sinks are vertices such as v∈v,∄e= (v,u) v \in V, \nexists e = (V, u). In other words, these vertices do not have a forward edge to the other vertices. Similarly, the sources is v∈v,∄e= (u,v) v \in V, \nexists e = (U, v).

For us, the value is always placed in sources, and the value is propagated to sinks. Reverse Mode derivation

If I don't think my explanation is good enough, here are some slides.

Derivation is the core requirement of many models required by TensorFlow, as it is needed to run gradient descent algorithms . Everyone who graduated from high school knows what the derivation is; It just gets the derivative of the function, and if the function is a complex combination of basic functions, then do the chain rule . a super Simple overview

If you have one of these functions:

F (x,y) = x * y

Then the derivation of X will produce:

DF (x,y) dx=y \frac{df (x,y)}{dx} = y

The derivation of Y will produce:

DF (x,y) dy=x \frac{df (x,y)}{dy} = X

Another example:

F (x1,x