Implementation of one of the simplest three-layer neural networks

This paper describes a neural network architecture with only one layer of implicit layer, and has the structure of forward propagation and back propagation, but in the setting of the back propagation, it only considers the simplest case-set the invariant learning rate, does not consider the exponential decay learning rate method, joins the regular item on the basis of the loss function, The sliding average model is added to enhance its robust.

#EX05 below is a complete simple neural network program import TensorFlow as TF from numpy.random import randomstate batch_size=8 #定义训练数据batch的大小 w1=tf.v Ariable (Tf.random_normal ([2,3],stddev=1,seed=1)) w2=tf. Variable (Tf.random_normal ([3,1],stddev=1,seed=2)) #初试化权重矩阵 (generates two random normal distributions of the corresponding size with a mean of 0 and a variance of 1) x=tf.placeholder ( Tf.float32,shape= (none,2), name= ' X-input ') #在第一个维度上利用None, makes it easy to use a small batch size Y_=tf.placeholder (tf.float32,shape= (
none,1), name= ' Y-input ') #在第一个维度上利用None, makes it easy to use small batch sizes #定义前向传播过程 A=tf.matmul (X,W1) #计算隐层的值 Y=tf.matmul (A,W2) #计算输出值 #定义损失函数和反向传播算法 Cross_entropy=-tf.reduce_mean (Y_*tf.log (Tf.clip_by_value (y,1e-10,1.0)) learning_rate=0.001 Train_
Step=tf.train.adamoptimizer (learning_rate). Minimize (Cross_entropy) #在此神经网络中只有基础学习率的设置, there is no exponential decay rate, and there are no regular and sliding average models. #通过随机数生成一个模拟数据集 rdm=randomstate (1) dataset_size=128 X=rdm.rand (dataset_size,2) #以下定义真实的样本标签 (using the following rules: all x1+x2< A sample of 1 is considered to be a positive sample, i.e. a qualified sample, expressed in 1, otherwise the unqualified sample, denoted by 0) y=[[int (x1+x2<1)] for (X1,X2) on X] #以下创建一个session会话来运行TensorFlow程序 with TF. Session () as Sess:init_op=tf.initialize_aLl_variables () Sess.run (init_op) #在此利用以上两行对其中涉及的参数进行统一初始化 print (Sess.run (W1)) Print (Sess.run (W2)) #在此将会打印出训练神经 The value of the parameter before the network #设定训练的轮数 STEPS = STEPS: start= (i*batch_size)%dataset_size end=min ( start+batch_size,dataset_size) #每次选取batch_size个样本进行训练 Sess.run (Train_step,feed_dict={x:x[start:end],y_:y[start: END]}) #通过选取的样本训练神经网络并更新其中的参数 if I%1000==0:total_cross_entropy=sess.run (cross_entropy,feed_dict={x:x,y _:y}) Print ("after%dtraining step (s), cross_entropy on all data is%g"% (i,total_cross_entropy)) print (SESS.R Un (W1)) Print (Sess.run (W2))

The result is:

[[ -0.8113182   1.4845988   0.06532937]
 [ -2.4427042   0.0992484   0.5912243]]
[[-0.85811085]
 [ -0.19662298]
 [0.13895045]]
After0training Step (s), cross_entropy on all data is0.227861
after1000training step (s), cross_entropy on all data is0.0395041
after2000training Step (s), cross_entropy on all data is0.0199546
after3000training step (s), Cross_entropy on all data is0.0117527
after4000training step (s), cross_entropy on all data is0.00781693
[[- 1.7101479  2.2005002  1.576958]
 [ -3.6296377  1.1303781  2.4205081]]
[[-1.8791978]
 [0.62330467]
 [1.771285  ]]