News classification
Contact us
- Add: No. 9, North Fourth Ring Road, Haidian District, Beijing. It mainly includes face recognition, living detection, ID card recognition, bank card recognition, business card recognition, license plate recognition, OCR recognition, and intelligent recognition technology.
- Tel: 13146317170 廖经理
- Fax:
- Email: 398017534@qq.com
Image Classification and Image Segmentation in Deep Learning
Image Classification and Image Segmentation in Deep Learning
From image classification to image segmentation
Convolutional neural network (CNN) has achieved great achievements and widespread applications in image classification and image detection since 2012.
The power of CNN lies in its multi-layer construction that automatically learns features and can learn multiple levels of features: shallower convolutional layers have a smaller perceptual domain, and learn some of the features of partial areas; deeper convolutions The layer has a larger perceptual domain and can learn more general features. These general features are less sensitive to the size, position, and orientation of objects, which helps to identify performance improvements.
These general features are very helpful in classifying, and can be used to determine exactly what kind of objects are contained in an image. However, due to the loss of details of some objects, the detailed outline of the object cannot be given, and each pixel is pointed out. It is very difficult to accurately divide the object.
The traditional CNN-based segmentation approach is usually to classify a pixel by using an image block around the pixel as input to the CNN for exercise and prediction. This approach has several drawbacks: First, the cost of storage is large. For example, if the size of an image block to be used for each pixel is 15×15, the required storage space is 225 times that of the original image. The second is inefficient computing. The neighboring pixel blocks are fundamentally iterative, and the convolution is calculated one by one for each pixel block. This calculation also has a large level of repetition. Third, the size of the pixel block limits the size of the sensing area. Usually the size of the pixel block is much smaller than the size of the entire image, and only the features of some parts can be extracted, resulting in the limitation of the performance of the classification.
In response to this problem, Jonathan Long et al. of UC Berkeley proposed Fully Convolutional Networks (FCN) for image segmentation. The network attempts to recover from the general characteristics the category to which each pixel belongs. That is, the classification from the image level further extends to the pixel level classification.
The principle of FCN
The FCN transforms the full convergence layer in the traditional CNN into a convolutional layer. As shown in the figure below, in the traditional CNN structure, the first 5 layers are convolutional layers, the 6th and 7th layers are a one-dimensional vector of length 4096, and the 8th layer is a one-dimensional vector of length 1000. Correspond to the probability of 1000 categories respectively. The FCN represents these three layers as a convolutional layer whose size (number of channels, width, height) is (4096, 1, 1), (4096, 1, 1), (1000, 1, 1), respectively. All layers are convolutional layers and are called full convolutional networks.
Write a picture here
It can be found that after repeated convolutions (and pooling), the resulting image is getting smaller and smaller and the resolution is getting lower (rough images). So how does the FCN get the class of each pixel in the image? In order to restore the original image resolution from this coarse resolution image, FCN uses upsampling. For example, after 5 times of convolution (and pooling), the resolution of the image is reduced by 2, 4, 8, 16, 32 times in turn. With regard to the output image of the last layer, it is necessary to stop upsampling 32 times to obtain the same size as the original image.
This upsampling is done by deconvolution. The output of the 5th layer (32x magnification) is deconvolved to the original image size. The result obtained is still not accurate enough and some details cannot be recovered. So Jonathan deconvolved the output of the fourth layer and the output of the third layer in turn, requiring 16 times and 8 times the upsampling, respectively, and the result is more precise. The following figure shows the process of this convolution and deconvolution upsampling:
Write a picture here
The following figure is a comparison of the results obtained at 32, 16 and 8 times upsampling. We can see that their results are getting more and more accurate:
Write a picture here
The advantages and lack of FCN
Compared with the conventional method of stopping image segmentation with CNN, FCN has two distinct advantages: First, it can accept input images of arbitrary size without requiring that all training images and test images have the same size. The second is more efficient due to the prevention of repeated storage and computational convolution problems due to the use of pixel blocks.
At the same time, the defects of FCN are also obvious: First, the results obtained are still not precise enough. Stopping 8 times upsampling is actually much better than 32 times, but the result of upsampling is still vague and warring, and it is not sensitive to the details in the image. The second is to stop the classification of each pixel, there is no sufficient consideration of the relationship between pixels and pixels, neglect the spatial regularization steps used in the usual pixel-based segmentation method, lack of spatial divergence.
FCN expansion
Although FCN is not successful enough, its new thoughts have opened up a new direction of image segmentation. The impact on this category is very ambitious. From March 2015 in arxiv and June in CVPR, publish this blog. At a time of one year, the article has been invoked up to 400 times.
At the root of the FCN, UCLA DeepLab's Liang-Chieh Chen and others used the fully connected conditional random fields after obtaining the pixel classification results to think about the spatial information in the image, which is more accurate and spatially divergent. Sexual results.
Liang-Chieh's article has two major features. One is to neglect the downsampling process and instead use dense convolution kernels to increase the scope of perception. As shown below:
Write a picture here
The second is the use of Fully Connected CRF. The energy function of the CRF includes two items of the data item sliding slip. The data item relates to the probability that each pixel belongs to each category. The smoothing item controls the divergence between the pixel and pixel categories. Traditional CRF smoothing items only consider the relevance of adjacent pixel categories, while Fully Connected CRF considers the category association between two pixels in the image.
The following figure shows the separation of CNN from Fully Connected CRF.
Write a picture here
In practice, CRF or Fully Connected CRF is a post-processing technology for CNN or FCN output. The two steps of pixel classification and spatial regularization are stopped separately. Shuai Zheng et al. expressed Fully Connected CRF as a recurrent neuron network (RNN), placing the CNN and the RNN in a unified framework and able to stop the exercise at the same time in one step. The three steps in the image segmentation: feature extraction, classifier prediction, and spatial reorganization are fully automated. After learning, the results obtained are much better than the FCN-8s and DeepLab methods.
Convolutional neural network (CNN) has achieved great achievements and widespread applications in image classification and image detection since 2012.
The power of CNN lies in its multi-layer construction that automatically learns features and can learn multiple levels of features: shallower convolutional layers have a smaller perceptual domain, and learn some of the features of partial areas; deeper convolutions The layer has a larger perceptual domain and can learn more general features. These general features are less sensitive to the size, position, and orientation of objects, which helps to identify performance improvements.
These general features are very helpful in classifying, and can be used to determine exactly what kind of objects are contained in an image. However, due to the loss of details of some objects, the detailed outline of the object cannot be given, and each pixel is pointed out. It is very difficult to accurately divide the object.
The traditional CNN-based segmentation approach is usually to classify a pixel by using an image block around the pixel as input to the CNN for exercise and prediction. This approach has several drawbacks: First, the cost of storage is large. For example, if the size of an image block to be used for each pixel is 15×15, the required storage space is 225 times that of the original image. The second is inefficient computing. The neighboring pixel blocks are fundamentally iterative, and the convolution is calculated one by one for each pixel block. This calculation also has a large level of repetition. Third, the size of the pixel block limits the size of the sensing area. Usually the size of the pixel block is much smaller than the size of the entire image, and only the features of some parts can be extracted, resulting in the limitation of the performance of the classification.
In response to this problem, Jonathan Long et al. of UC Berkeley proposed Fully Convolutional Networks (FCN) for image segmentation. The network attempts to recover from the general characteristics the category to which each pixel belongs. That is, the classification from the image level further extends to the pixel level classification.
The principle of FCN
The FCN transforms the full convergence layer in the traditional CNN into a convolutional layer. As shown in the figure below, in the traditional CNN structure, the first 5 layers are convolutional layers, the 6th and 7th layers are a one-dimensional vector of length 4096, and the 8th layer is a one-dimensional vector of length 1000. Correspond to the probability of 1000 categories respectively. The FCN represents these three layers as a convolutional layer whose size (number of channels, width, height) is (4096, 1, 1), (4096, 1, 1), (1000, 1, 1), respectively. All layers are convolutional layers and are called full convolutional networks.
Write a picture here
It can be found that after repeated convolutions (and pooling), the resulting image is getting smaller and smaller and the resolution is getting lower (rough images). So how does the FCN get the class of each pixel in the image? In order to restore the original image resolution from this coarse resolution image, FCN uses upsampling. For example, after 5 times of convolution (and pooling), the resolution of the image is reduced by 2, 4, 8, 16, 32 times in turn. With regard to the output image of the last layer, it is necessary to stop upsampling 32 times to obtain the same size as the original image.
This upsampling is done by deconvolution. The output of the 5th layer (32x magnification) is deconvolved to the original image size. The result obtained is still not accurate enough and some details cannot be recovered. So Jonathan deconvolved the output of the fourth layer and the output of the third layer in turn, requiring 16 times and 8 times the upsampling, respectively, and the result is more precise. The following figure shows the process of this convolution and deconvolution upsampling:
Write a picture here
The following figure is a comparison of the results obtained at 32, 16 and 8 times upsampling. We can see that their results are getting more and more accurate:
Write a picture here
The advantages and lack of FCN
Compared with the conventional method of stopping image segmentation with CNN, FCN has two distinct advantages: First, it can accept input images of arbitrary size without requiring that all training images and test images have the same size. The second is more efficient due to the prevention of repeated storage and computational convolution problems due to the use of pixel blocks.
At the same time, the defects of FCN are also obvious: First, the results obtained are still not precise enough. Stopping 8 times upsampling is actually much better than 32 times, but the result of upsampling is still vague and warring, and it is not sensitive to the details in the image. The second is to stop the classification of each pixel, there is no sufficient consideration of the relationship between pixels and pixels, neglect the spatial regularization steps used in the usual pixel-based segmentation method, lack of spatial divergence.
FCN expansion
Although FCN is not successful enough, its new thoughts have opened up a new direction of image segmentation. The impact on this category is very ambitious. From March 2015 in arxiv and June in CVPR, publish this blog. At a time of one year, the article has been invoked up to 400 times.
At the root of the FCN, UCLA DeepLab's Liang-Chieh Chen and others used the fully connected conditional random fields after obtaining the pixel classification results to think about the spatial information in the image, which is more accurate and spatially divergent. Sexual results.
Liang-Chieh's article has two major features. One is to neglect the downsampling process and instead use dense convolution kernels to increase the scope of perception. As shown below:
Write a picture here
The second is the use of Fully Connected CRF. The energy function of the CRF includes two items of the data item sliding slip. The data item relates to the probability that each pixel belongs to each category. The smoothing item controls the divergence between the pixel and pixel categories. Traditional CRF smoothing items only consider the relevance of adjacent pixel categories, while Fully Connected CRF considers the category association between two pixels in the image.
The following figure shows the separation of CNN from Fully Connected CRF.
Write a picture here
In practice, CRF or Fully Connected CRF is a post-processing technology for CNN or FCN output. The two steps of pixel classification and spatial regularization are stopped separately. Shuai Zheng et al. expressed Fully Connected CRF as a recurrent neuron network (RNN), placing the CNN and the RNN in a unified framework and able to stop the exercise at the same time in one step. The three steps in the image segmentation: feature extraction, classifier prediction, and spatial reorganization are fully automated. After learning, the results obtained are much better than the FCN-8s and DeepLab methods.
PREVIOUS：Target detection and positioning in deep
NEXT：用机器学习方法提高图片的清晰度