Artificial intelligence is so simple (2)--activation function

1. About activating functions

If according to the idea of the previous article, AI can not simulate the curve equation, such as the parabolic equation, in time to add more parameter values, also can not achieve the effect, so need to introduce activation function. To facilitate the introduction of a minimalist function sigmoid function, the prototype of the sigmoid function is as follows:

The graph of this function is shown below:

The sigmoid function partly reflects the idea of two classification, in which the curve of the graph above can be regarded as a probability distribution function, between 0 and 1, and 0.5 can be classified as a threshold value. Of course here we just use the activation function to bend the linear equation. 2, the activation function in the model

In the model above, the original linear results are used to curve the activation function, and the model can be used to simulate a parabolic function test effect. 3. Prepare training Data

Suppose the curve we're going to simulate is

You can prepare the training data in the same way as before.

Import TensorFlow as TF
import matplotlib.pyplot as Plt
import NumPy as np
x = np.random.random (MB) *100-50
  y = x**2
plt.scatter (x,y)
plt.show ()

The curve drawing effect is as follows

Add the activation function to simulate the curve directly below

# Training Model
# Generate test Data
import TensorFlow as tf
import matplotlib.pyplot as Plt
import numpy as NP
import MATP Lotlib.animation as animation
x = Np.linspace ( -1,1,300) [:, Np.newaxis]
y = x**2
W = tf. Variable (. 0,name= ' W ', dtype=tf.float32)
B = tf. Variable (. 0,name= ' B ', dtype=tf.float32)
predict_y =w*x+b Cost
= Tf.reduce_mean (Tf.square (predict_y-y), Name= ' cost ')
train = Tf.train.GradientDescentOptimizer (0.1). Minimize (cost)
init = Tf.global_variables_ Initializer () with
TF. Session () as Sess:
sess.run (init) to
epoch in range:
Sess.run (train)
if epoch%50 = 0:
print ("%s epoch, W =%s, b=%s, loss =%s"% (str (epoch), Sess.run (W), Sess.run (b), SESS.R Un (COST))
Plt.plot (x,y, ' Bo ', X,sess.run (predict_y), ' r--')
plt.show ()

The final effect is shown in the following illustration:

You can see that there's no good simulation curve, the reason is what, because two characteristics w,b is unable to simulate the curve, you can think of the curve as a lot of small segments of the superposition, as a child painting, then we need to add a characteristic value of the model to use 10 neurons to extract the X1 feature. The modified model is shown in the following figure

Rewrite the program according to the model above

Import TensorFlow as TF
import matplotlib.pyplot as Plt
import numpy as NP
import matplotlib.animation as Anim ation
Import sys
x_rand = np.linspace ( -5,5,100)
x = X_rand[:,np.newaxis].astype (np.float32)
y = Np.square (x)
W1 = tf. Variable (Tf.truncated_normal ([1,10],stddev=0.1), name= ' W1 ', dtype=tf.float32)
B1 = tf. Variable ([Tf.zeros]), Dtype=tf.float32,name= ' B1 ') z1 = Tf.add (Tf.matmul
(X,W1), B1)
a1 = Tf.nn.sigmoid (z1 )
W2 = tf. Variable (Tf.truncated_normal ([10,1],stddev=0.1), name= ' W2 ', dtype=tf.float32)
b2 = tf. Variable (Tf.zeros ([1]), dtype=tf.float32,name= ' B2 ')
predict_y = Tf.add (Tf.matmul (A1,W2), b2) Cost
= Tf.reduce_mean (Tf.square (predict_y-y), name= ' cost ')
train = Tf.train.GradientDescentOptimizer (0.1). Minimize ( Cost)
init = Tf.global_variables_initializer () with
TF. Session () as Sess:
    sess.run (init)
    to epoch in range (1000): Sess.run (train
        )
    Plt.plot (x,y, ' Bo ', X , Sess.run (predict_y), ' R '

The output effect is shown in the following illustration:

So we can simulate the curve, we can do a generalization no matter how complex the curve, as long as the eigenvalues are enough to simulate.