News classification
Contact us
- Add: No. 9, North Fourth Ring Road, Haidian District, Beijing. It mainly includes face recognition, living detection, ID card recognition, bank card recognition, business card recognition, license plate recognition, OCR recognition, and intelligent recognition technology.
- Tel: 13146317170 廖经理
- Fax:
- Email: 398017534@qq.com
History of deep learning
History of deep learning
1 History and development trend of deep learning
1.1 History of deep learning
Generally speaking, there have been three waves of deep learning so far: deep learning in the 1940s and 1960s was called cybernetics, and deep learning in the 1980s and 1990s was called connection mechanism (connectionism). ), and began in 2006, revived in the name of deep learning.
The articulation mechanism is presented in the context of cognitive science. Cognitive science is an interdisciplinary approach to understanding the mind and separating multiple levels of dissection. The central idea of the convergence mechanism is that when the network connects a large number of simple computing units at the same time, it can complete intelligent behavior. This insight also applies to neurons in the biological nervous system that act similarly to the hiding unit in the computational model.
The third wave of neural network research began in 2006. Geoffrey Hinton's neural network, called Deep Confidence Network (DBN), can stop exercising effectively using a strategy called greedy layer-by-layer exercise (Hinton et al., 2006a).
1.2 Development of deep learning
The amount of data and model range
The increase in the amount of data makes the techniques of deep learning some techniques to obtain good performance gradually reduced. At present, in complex tasks, it can reach performance comparable to human performance. The arrival of the big data era has made deep learning easier. However, we should pay attention to the application of unlabeled samples in non-monitoring and semi-monitoring learning.
Increase in data volume
With the expansion of the model range, larger networks can achieve higher precision in more complex tasks. Therefore, more scholars introduce more hiding units in the neural network, which makes the model range expand. In addition, hardware (faster CPUs, general-purpose GPUs, hard drive storage) and better distributed computing software infrastructure and faster network connectivity make model-wide expansion possible.
Accuracy, complexity and impact on the ideal world
The ability of deep learning to provide accurate identification and prediction is constantly improving. In addition, deep learning continues to be victoriously applied to more and more popular applications.
The figure above shows the number of connections per neuron. 1, self-compliance linear unit; 2, neural cognitive machine; 3, GPU accelerated convolution network; 4, deep Boltzmann machine; 5, no monitoring convolution network; 6, GPU accelerated multi-layer perceptron; Distributed automatic encoder; 8, Multi-GPU convolution network; 9, COTS HPC no monitoring convolution network; 10, GoogLeNet.
Neural network range extension
1. Perceptron (Rosenblatt, 1958, 1962)
2. Self-compliance linear units (Widrow and Hoff, 1960)
3. Neurocognitive machine (Fukushima, 1980)
4. Early backward propagation networks (Rumelhart et al., 1986b)
5. Recurrent neural networks for speech recognition (Robinson and Fallside, 1991)
6. Multilayer perceptron for speech recognition (Bengio et al., 1991)
7. Mean field sigmoid confidence network (Saul et al., 1996)
8. LeNet-5 (LeCun et al., 1998b)
9. Echo State Network (Jaeger and Haas, 2004)
10. Deep Confidence Network (Hinton et al., 2006a)
11. GPU-Accelerated Convolution Network (Chellapilla et al., 2006)
12. Deep Boltzmann machine (Salakhutdinov and Hinton, 2009a)
13. GPU-Accelerated Deep Confidence Network (Raina et al., 2009a)
14. Monitoring-free convolutional networks (Jarrett et al., 2009b)
15. GPU-Accelerated Multilayer Perceptron (Ciresan et al., 2010)
16. OMP-1 Network (Coates and Ng, 2011)
17. Distributed Autoencoder (Le et al., 2012)
18. Multi-GPU Convolutional Network (Krizhevsky et al., 2012a)
19. COTS HPC Monitoring-Free Convolution Network (Coates et al., 2013)
20. GoogLeNet (Szegedy et al., 2014a)
Deep network's decreasing error rate in ImageNet battle
Deep learning is a method of machine learning. In the past few decades, it has deeply absorbed our knowledge of human brain, statistics and applied mathematics. In recent years, the improvement and applicability of deep learning has been greatly expanded, which has benefited a lot from more powerful computers, larger data sets, and technologies that can train deeper networks. In the years to come, there will be opportunities and opportunities for further progress in deep learning and bringing it to new areas.
1.1 History of deep learning
Generally speaking, there have been three waves of deep learning so far: deep learning in the 1940s and 1960s was called cybernetics, and deep learning in the 1980s and 1990s was called connection mechanism (connectionism). ), and began in 2006, revived in the name of deep learning.
The articulation mechanism is presented in the context of cognitive science. Cognitive science is an interdisciplinary approach to understanding the mind and separating multiple levels of dissection. The central idea of the convergence mechanism is that when the network connects a large number of simple computing units at the same time, it can complete intelligent behavior. This insight also applies to neurons in the biological nervous system that act similarly to the hiding unit in the computational model.
The third wave of neural network research began in 2006. Geoffrey Hinton's neural network, called Deep Confidence Network (DBN), can stop exercising effectively using a strategy called greedy layer-by-layer exercise (Hinton et al., 2006a).
1.2 Development of deep learning
The amount of data and model range
The increase in the amount of data makes the techniques of deep learning some techniques to obtain good performance gradually reduced. At present, in complex tasks, it can reach performance comparable to human performance. The arrival of the big data era has made deep learning easier. However, we should pay attention to the application of unlabeled samples in non-monitoring and semi-monitoring learning.
Increase in data volume
With the expansion of the model range, larger networks can achieve higher precision in more complex tasks. Therefore, more scholars introduce more hiding units in the neural network, which makes the model range expand. In addition, hardware (faster CPUs, general-purpose GPUs, hard drive storage) and better distributed computing software infrastructure and faster network connectivity make model-wide expansion possible.
Accuracy, complexity and impact on the ideal world
The ability of deep learning to provide accurate identification and prediction is constantly improving. In addition, deep learning continues to be victoriously applied to more and more popular applications.
The figure above shows the number of connections per neuron. 1, self-compliance linear unit; 2, neural cognitive machine; 3, GPU accelerated convolution network; 4, deep Boltzmann machine; 5, no monitoring convolution network; 6, GPU accelerated multi-layer perceptron; Distributed automatic encoder; 8, Multi-GPU convolution network; 9, COTS HPC no monitoring convolution network; 10, GoogLeNet.
Neural network range extension
1. Perceptron (Rosenblatt, 1958, 1962)
2. Self-compliance linear units (Widrow and Hoff, 1960)
3. Neurocognitive machine (Fukushima, 1980)
4. Early backward propagation networks (Rumelhart et al., 1986b)
5. Recurrent neural networks for speech recognition (Robinson and Fallside, 1991)
6. Multilayer perceptron for speech recognition (Bengio et al., 1991)
7. Mean field sigmoid confidence network (Saul et al., 1996)
8. LeNet-5 (LeCun et al., 1998b)
9. Echo State Network (Jaeger and Haas, 2004)
10. Deep Confidence Network (Hinton et al., 2006a)
11. GPU-Accelerated Convolution Network (Chellapilla et al., 2006)
12. Deep Boltzmann machine (Salakhutdinov and Hinton, 2009a)
13. GPU-Accelerated Deep Confidence Network (Raina et al., 2009a)
14. Monitoring-free convolutional networks (Jarrett et al., 2009b)
15. GPU-Accelerated Multilayer Perceptron (Ciresan et al., 2010)
16. OMP-1 Network (Coates and Ng, 2011)
17. Distributed Autoencoder (Le et al., 2012)
18. Multi-GPU Convolutional Network (Krizhevsky et al., 2012a)
19. COTS HPC Monitoring-Free Convolution Network (Coates et al., 2013)
20. GoogLeNet (Szegedy et al., 2014a)
Deep network's decreasing error rate in ImageNet battle
Deep learning is a method of machine learning. In the past few decades, it has deeply absorbed our knowledge of human brain, statistics and applied mathematics. In recent years, the improvement and applicability of deep learning has been greatly expanded, which has benefited a lot from more powerful computers, larger data sets, and technologies that can train deeper networks. In the years to come, there will be opportunities and opportunities for further progress in deep learning and bringing it to new areas.