News classification
Contact us
- Add: No. 9, North Fourth Ring Road, Haidian District, Beijing. It mainly includes face recognition, living detection, ID card recognition, bank card recognition, business card recognition, license plate recognition, OCR recognition, and intelligent recognition technology.
- Tel: 13146317170 廖经理
- Fax:
- Email: 398017534@qq.com
Construction of artificial intelligence neural network
Construction of artificial intelligence neural network
First, the completion process of the neural network:
First, prepare the database, extract features, and input it as a neural network (NN).
Second, build the NN construct, from input to output (build the calculation graph first, then use the session to execute) (NN forward algorithm ---> calculate output)
Third, a large amount of feature data is fed to the NN, and the NN parameters are iteratively optimized until the model arrives at the request (NN inverse algorithm--->optimal parameter exercise model)
Fourth, use the well-executed model to complete prediction and classification.
Later, let's take a look at the completion paradigm of forward propagation and back propagation.
Let's first look at the forward propagation: (this part comes from the explanation of the pku tutorial and creates some overview of the predecessors)
Forward propagation is the computational process of building a model, which allows the model to have inferential ability and can provide a corresponding output for a set of inputs.
Before discussing the forward propagation, we must first understand what the number of layers called the neural network is defined.
The number of neural network layers = hiding computing layer + 1 output layer, hiding the computing layer is the intermediate parameter generated in the operation, as shown in the following figure.
Neural network complexity (NN complexity): the number of layers + parameters to represent.
For the sake of convenience, let us give an example:
Consuming a batch of parts with a volume of x1 and a weight of x2, 1, 2 represent the subscript. Volume and weight are the characteristics we choose, feeding them into the neural network, and when the volume and weight data goes through the neural network, we get an output.
Assume that the input eigenvalue is: volume 0.7 weight 0.5
It can be obtained from the constructed neural network. The hiding layer node a11=x1*w11+x2*w21=0.14+0.15=0.29, the same node a12=0.32, a13=0.38, and finally the output layer Y=-0.015 is calculated. The forward propagation process is completed.
Derivation:
The first layer X is the input 1X2 matrix. The input is represented by x. It is a matrix of 1 row and 2 columns, indicating that a set of features is input at a time. This set of features contains two elements of volume and weight.
W front node number, rear node number (layer number) is the parameter to be optimized, its superscript represents the number of layers
The generation of parameters is mentioned in my previous hydrology.
There are two nodes in front of w for the first layer, and three nodes behind w should be a matrix of two rows and three columns.
We say this:
The neural network has several layers (or the current layer network), which refers to the computing layer. The input is not the computing layer, so a is the first layer network, and a is a row and three columns matrix.
We say this: a(1)=[a11, a12, a13]=XW(1)
The second layer of parameters must satisfy the first three nodes, the latter one, so W(2) is a three-row, one-column matrix. We say this:
We multiply each layer of input by the weight w on the line so that the output y can be calculated using matrix multiplication.
First calculate the operational expression of a: a = tf.matmul (X, W1)
Calculate the expression of y: y = tf.matmul(a, W2)
Due to the result of the demand calculation, it is necessary to use the with construct, and all variable initialization and calculation processes should be placed in the sess.run function.
Regarding variable initialization, we write tf.global_variables_initializer in sess.run to complete initialization of all variables, that is, to assign initial values. Regarding the operation in the calculation graph, we can directly fill in the operation node into sess.run, for example, to calculate the output y, directly write sess.run(y).
In practice, we can feed one or more sets of inputs at a time, let the neural network calculate the output y, and first use tf.placeholder to give the input placeholder.
If the first dimension position of a set of data shapes is written once, the second dimension position has several input features;
If you want to feed multiple sets of data at a time, the first dimension of the shape can be written with None indicating that it is empty first, and the second dimension is written with several input features.
This allows for the feeding of several sets of volumetric weights in the feed_dict.
Code example:
#coding:utf-8
Import tensorflow as tf
# Define inputs and parameters
Feeding a set of data in #sess.run
x = tf.placeholder(tf.float32, shape=(1, 2))
W1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1))
W2= tf.Variable(tf.random_normal([3, 1], stddev=1, seed=1))
# Define forward propagation process
a = tf.matmul(x, w1)
y = tf.matmul(a, w2)
#Using the session calculation result to calculate the value of the graph node
With tf.Session() as sess:
Init_op = tf.global_variables_initializer()
Sess.run(init_op)
Print"y in this case is:\n",sess.run(y, feed_dict={x: [[0.7,0.5]]})
1. Upstairs is the status of feeding a set of data in sess.run. Feed data in sess.run with feed_dict={}: the first group has a volume of 0.7 and a weight of 0.5.
#coding:utf-8
Import tensorflow as tf
# Define inputs and parameters
#Use placeholder to define input (sess.run to feed multiple sets of data)
x = tf.placeholder(tf.float32, shape=(None, 2))
W1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1))
W2= tf.Variable(tf.random_normal([3, 1], stddev=1, seed=1))
a = tf.matmul(x, w1)
y = tf.matmul(a, w2)
#call session calculation result
With tf.Session() as sess:
Init_op = tf.global_variables_initializer()
Sess.run(init_op)
Print "the result in this case is:\n",sess.run(y, feed_dict={x: [[0.7,0.5],[0.2,0.3],[0.3,0.4],[0.4,0.5]]} )
Print "w1:\n", sess.run(w1)
Print "w2:\n", sess.run(w2)
2. Upstairs is the state of confluence of multiple sets of data.
Stop and stop, first consider the philosophical meaning.
Look at the back propagation:
The purpose is to optimize the model parameters and use gradient gradients on all parameters to minimize the loss function of the NN model on the exercise data. If the loss function is used, I will explain it in more detail in the later hydrology.
Loss function: (loss): the difference between the predicted value (y) and the known answer (y_)
Mean Square Error MSE: MSE(y, y_) is the sum of the square of each sample and mean difference of our high school mathematics divided by the sample size.
Loss=tf.reduce_mean(tf.square(y_-y))
Backpropagation exercise method: Optimize the purpose of reducing the loss value.
To reduce the loss value for optimization purposes, there are three commonly used optimization methods (optimizers):
Gradient landing, momentum optimizer, adam optimizer.
These three optimization methods can be expressed as tensorflow functions:
Train_step=tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)
Train_step=tf.train.MomentumOptimizer(learning_rate, momentum).minimize(loss)
Train_step=tf.train.AdamOptimizer(learning_rate).minimize(loss)
Where learning_rate represents the learning rate, indicating the magnitude of the parameter update. When applied, we can use a smaller value of the table, such as 0.001.
Learning rate: learning-rate, the extent to which the resolution parameter is updated each time.
Finally, take a look at the source code that the teacher gave last:
#coding:utf-8
#import module to generate an imitation data set. Numpy is python's scientific computing module
Import tensorflow as tf
Import numpy as np
BATCH_SIZE = 8
SEED = 23455
#based on seed to generate random numbers
Rdm = np.random.RandomState(SEED)
#随机数 Returns a matrix of 32 rows and 2 columns representing 32 groups of volume and weight as input data sets
X = rdm.rand(32,2)
#
# as the label of the input data set (correct answer)
Y_ = [[int(x0 + x1 < 1)] for (x0, x1) in X]
Print "X:\n",X
Print "Y_:\n",Y_
#1 defines the input, parameters, and output of the neural network, defining the forward propagation process.
x = tf.placeholder(tf.float32, shape=(None, 2))
Y_= tf.placeholder(tf.float32, shape=(None, 1))
W1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1))
W2= tf.Variable(tf.random_normal([3, 1], stddev=1, seed=1))
a = tf.matmul(x, w1)
y = tf.matmul(a, w2)
#2 defines the loss function and the back propagation method.
Loss_mse = tf.reduce_mean(tf.square(y-y_))
Train_step = tf.train.GradientDescentOptimizer(0.001).minimize(loss_mse)
#train_step = tf.train.MomentumOptimizer(0.001,0.9).minimize(loss_mse)
#train_step = tf.train.AdamOptimizer(0.001).minimize(loss_mse)
#3Generate a session, exercise the STEPS round
With tf.Session() as sess:
Init_op = tf.global_variables_initializer()
Sess.run(init_op)
# Output the current (untrained) parameter value.
Print "w1:\n", sess.run(w1)
Print "w2:\n", sess.run(w2)
Print "\n"
# Exercise model.
STEPS = 3000
For i in range(STEPS):
Start = (i*BATCH_SIZE) % 32
End = start + BATCH_SIZE
Sess.run(train_step, feed_dict={x: X[start:end], y_: Y_[start:end]})
If i % 500 == 0:
Total_loss = sess.run(loss_mse, feed_dict={x: X, y_: Y_})
Print("After %d training step(s), loss_mse on all data is %g" % (i, total_loss))
# Output the value of the parameter after the workout.
Print "\n"
Print "w1:\n", sess.run(w1)
Print "w2:\n", sess.run(w2)
There are several requirements for this waste to clarify:
1.numpy is python's scientific computing module
2. Our rule BATCH_SIZE is the number of input data sets (or range). You can't feed too much data at a time, otherwise the neural network will catch up.
3.seed is pseudo-random, we just use the seed rule to make such a value for demonstration purposes.
4. Apply a random number to generate a data set, Y is a standard
First, prepare the database, extract features, and input it as a neural network (NN).
Second, build the NN construct, from input to output (build the calculation graph first, then use the session to execute) (NN forward algorithm ---> calculate output)
Third, a large amount of feature data is fed to the NN, and the NN parameters are iteratively optimized until the model arrives at the request (NN inverse algorithm--->optimal parameter exercise model)
Fourth, use the well-executed model to complete prediction and classification.
Later, let's take a look at the completion paradigm of forward propagation and back propagation.
Let's first look at the forward propagation: (this part comes from the explanation of the pku tutorial and creates some overview of the predecessors)
Forward propagation is the computational process of building a model, which allows the model to have inferential ability and can provide a corresponding output for a set of inputs.
Before discussing the forward propagation, we must first understand what the number of layers called the neural network is defined.
The number of neural network layers = hiding computing layer + 1 output layer, hiding the computing layer is the intermediate parameter generated in the operation, as shown in the following figure.
Neural network complexity (NN complexity): the number of layers + parameters to represent.
For the sake of convenience, let us give an example:
Consuming a batch of parts with a volume of x1 and a weight of x2, 1, 2 represent the subscript. Volume and weight are the characteristics we choose, feeding them into the neural network, and when the volume and weight data goes through the neural network, we get an output.
Assume that the input eigenvalue is: volume 0.7 weight 0.5
It can be obtained from the constructed neural network. The hiding layer node a11=x1*w11+x2*w21=0.14+0.15=0.29, the same node a12=0.32, a13=0.38, and finally the output layer Y=-0.015 is calculated. The forward propagation process is completed.
Derivation:
The first layer X is the input 1X2 matrix. The input is represented by x. It is a matrix of 1 row and 2 columns, indicating that a set of features is input at a time. This set of features contains two elements of volume and weight.
W front node number, rear node number (layer number) is the parameter to be optimized, its superscript represents the number of layers
The generation of parameters is mentioned in my previous hydrology.
There are two nodes in front of w for the first layer, and three nodes behind w should be a matrix of two rows and three columns.
We say this:
The neural network has several layers (or the current layer network), which refers to the computing layer. The input is not the computing layer, so a is the first layer network, and a is a row and three columns matrix.
We say this: a(1)=[a11, a12, a13]=XW(1)
The second layer of parameters must satisfy the first three nodes, the latter one, so W(2) is a three-row, one-column matrix. We say this:
We multiply each layer of input by the weight w on the line so that the output y can be calculated using matrix multiplication.
First calculate the operational expression of a: a = tf.matmul (X, W1)
Calculate the expression of y: y = tf.matmul(a, W2)
Due to the result of the demand calculation, it is necessary to use the with construct, and all variable initialization and calculation processes should be placed in the sess.run function.
Regarding variable initialization, we write tf.global_variables_initializer in sess.run to complete initialization of all variables, that is, to assign initial values. Regarding the operation in the calculation graph, we can directly fill in the operation node into sess.run, for example, to calculate the output y, directly write sess.run(y).
In practice, we can feed one or more sets of inputs at a time, let the neural network calculate the output y, and first use tf.placeholder to give the input placeholder.
If the first dimension position of a set of data shapes is written once, the second dimension position has several input features;
If you want to feed multiple sets of data at a time, the first dimension of the shape can be written with None indicating that it is empty first, and the second dimension is written with several input features.
This allows for the feeding of several sets of volumetric weights in the feed_dict.
Code example:
#coding:utf-8
Import tensorflow as tf
# Define inputs and parameters
Feeding a set of data in #sess.run
x = tf.placeholder(tf.float32, shape=(1, 2))
W1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1))
W2= tf.Variable(tf.random_normal([3, 1], stddev=1, seed=1))
# Define forward propagation process
a = tf.matmul(x, w1)
y = tf.matmul(a, w2)
#Using the session calculation result to calculate the value of the graph node
With tf.Session() as sess:
Init_op = tf.global_variables_initializer()
Sess.run(init_op)
Print"y in this case is:\n",sess.run(y, feed_dict={x: [[0.7,0.5]]})
1. Upstairs is the status of feeding a set of data in sess.run. Feed data in sess.run with feed_dict={}: the first group has a volume of 0.7 and a weight of 0.5.
#coding:utf-8
Import tensorflow as tf
# Define inputs and parameters
#Use placeholder to define input (sess.run to feed multiple sets of data)
x = tf.placeholder(tf.float32, shape=(None, 2))
W1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1))
W2= tf.Variable(tf.random_normal([3, 1], stddev=1, seed=1))
a = tf.matmul(x, w1)
y = tf.matmul(a, w2)
#call session calculation result
With tf.Session() as sess:
Init_op = tf.global_variables_initializer()
Sess.run(init_op)
Print "the result in this case is:\n",sess.run(y, feed_dict={x: [[0.7,0.5],[0.2,0.3],[0.3,0.4],[0.4,0.5]]} )
Print "w1:\n", sess.run(w1)
Print "w2:\n", sess.run(w2)
2. Upstairs is the state of confluence of multiple sets of data.
Stop and stop, first consider the philosophical meaning.
Look at the back propagation:
The purpose is to optimize the model parameters and use gradient gradients on all parameters to minimize the loss function of the NN model on the exercise data. If the loss function is used, I will explain it in more detail in the later hydrology.
Loss function: (loss): the difference between the predicted value (y) and the known answer (y_)
Mean Square Error MSE: MSE(y, y_) is the sum of the square of each sample and mean difference of our high school mathematics divided by the sample size.
Loss=tf.reduce_mean(tf.square(y_-y))
Backpropagation exercise method: Optimize the purpose of reducing the loss value.
To reduce the loss value for optimization purposes, there are three commonly used optimization methods (optimizers):
Gradient landing, momentum optimizer, adam optimizer.
These three optimization methods can be expressed as tensorflow functions:
Train_step=tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)
Train_step=tf.train.MomentumOptimizer(learning_rate, momentum).minimize(loss)
Train_step=tf.train.AdamOptimizer(learning_rate).minimize(loss)
Where learning_rate represents the learning rate, indicating the magnitude of the parameter update. When applied, we can use a smaller value of the table, such as 0.001.
Learning rate: learning-rate, the extent to which the resolution parameter is updated each time.
Finally, take a look at the source code that the teacher gave last:
#coding:utf-8
#import module to generate an imitation data set. Numpy is python's scientific computing module
Import tensorflow as tf
Import numpy as np
BATCH_SIZE = 8
SEED = 23455
#based on seed to generate random numbers
Rdm = np.random.RandomState(SEED)
#随机数 Returns a matrix of 32 rows and 2 columns representing 32 groups of volume and weight as input data sets
X = rdm.rand(32,2)
#
# as the label of the input data set (correct answer)
Y_ = [[int(x0 + x1 < 1)] for (x0, x1) in X]
Print "X:\n",X
Print "Y_:\n",Y_
#1 defines the input, parameters, and output of the neural network, defining the forward propagation process.
x = tf.placeholder(tf.float32, shape=(None, 2))
Y_= tf.placeholder(tf.float32, shape=(None, 1))
W1= tf.Variable(tf.random_normal([2, 3], stddev=1, seed=1))
W2= tf.Variable(tf.random_normal([3, 1], stddev=1, seed=1))
a = tf.matmul(x, w1)
y = tf.matmul(a, w2)
#2 defines the loss function and the back propagation method.
Loss_mse = tf.reduce_mean(tf.square(y-y_))
Train_step = tf.train.GradientDescentOptimizer(0.001).minimize(loss_mse)
#train_step = tf.train.MomentumOptimizer(0.001,0.9).minimize(loss_mse)
#train_step = tf.train.AdamOptimizer(0.001).minimize(loss_mse)
#3Generate a session, exercise the STEPS round
With tf.Session() as sess:
Init_op = tf.global_variables_initializer()
Sess.run(init_op)
# Output the current (untrained) parameter value.
Print "w1:\n", sess.run(w1)
Print "w2:\n", sess.run(w2)
Print "\n"
# Exercise model.
STEPS = 3000
For i in range(STEPS):
Start = (i*BATCH_SIZE) % 32
End = start + BATCH_SIZE
Sess.run(train_step, feed_dict={x: X[start:end], y_: Y_[start:end]})
If i % 500 == 0:
Total_loss = sess.run(loss_mse, feed_dict={x: X, y_: Y_})
Print("After %d training step(s), loss_mse on all data is %g" % (i, total_loss))
# Output the value of the parameter after the workout.
Print "\n"
Print "w1:\n", sess.run(w1)
Print "w2:\n", sess.run(w2)
There are several requirements for this waste to clarify:
1.numpy is python's scientific computing module
2. Our rule BATCH_SIZE is the number of input data sets (or range). You can't feed too much data at a time, otherwise the neural network will catch up.
3.seed is pseudo-random, we just use the seed rule to make such a value for demonstration purposes.
4. Apply a random number to generate a data set, Y is a standard