News classification
Contact us
- Add: No. 9, North Fourth Ring Road, Haidian District, Beijing. It mainly includes face recognition, living detection, ID card recognition, bank card recognition, business card recognition, license plate recognition, OCR recognition, and intelligent recognition technology.
- Tel: 13146317170 廖经理
- Fax:
- Email: 398017534@qq.com
Image Classification and Image Segmentation in Convolutional Neural Networks
Image Classification and Image Segmentation in Convolutional Neural Networks
From image classification to image segmentation
Since 2012, Convolutional Neural Networks (CNN) have achieved great achievements and universal applications in image classification and image detection.
The power of CNN is that its multi-layer structure can automatically learn features, and can learn multiple levels of features: shallower convolutional layer sensing domain is small, learn some features of partial regions; deep convolution The layer has a larger sensing domain and can learn more general features. These general features are less sensitive to the size, position, and orientation of the object, helping to identify performance improvements.
These general features are very helpful for classification, and can well discriminate what kind of objects are contained in an image, but because of the loss of the details of some objects, the detailed contour of the object cannot be well given, and each pixel is pointed out. Which object belongs in detail, so it is very difficult to achieve accurate segmentation.
The traditional CNN-based segmentation approach is usually to classify a pixel by using an image block around the pixel as input to the CNN for exercise and prediction. There are several drawbacks to this approach: First, storage costs are high. For example, if the size of the image block applied to each pixel is 15x15, the required storage space is 225 times that of the original image. Second, the calculation efficiency is low. Adjacent blocks of pixels are fundamentally iterative, and convolution is calculated one by one for each block of pixels, and this calculation also has a large level of iteration. Third, the size of the pixel block limits the size of the perceived area. Usually, the size of a pixel block is much smaller than the size of the entire image, and only some of the features can be extracted, thereby causing a limitation in the performance of the classification.
In response to this problem, UC Berkeley's Jonathan Long et al. proposed Fully Convolutional Networks (FCN) [1] [1] for image segmentation. The network attempts to recover the category to which each pixel belongs from a general feature. That is, the classification from the image level is further extended to the classification at the pixel level.
The principle of FCN
The FCN converts the full connectivity layer in a traditional CNN into a convolutional layer. As shown in the following figure, in the traditional CNN structure, the first 5 layers are convolutional layers, the 6th and 7th layers are respectively a one-dimensional vector with a length of 4096, and the eighth layer is a one-dimensional vector with a length of 1000. Corresponds to the probability of 1000 categories. The FCN expresses these three layers as a convolutional layer, and the size (channel number, width, and height) of the convolution kernel are (4096, 1, 1), (4096, 1, 1), (1000, 1, 1), respectively. All layers are convolutional layers, so they are called full convolutional networks.
Write a picture here
It can be found that after repeated convolution (and pooling), the resulting image is getting smaller and smaller, and the resolution is getting lower and lower (rough image). How does FCN get the category of each pixel in the image? In order to recover from this coarse image with low resolution to the resolution of the original image, FCN uses upsampling. For example, after 5 convolutions (and pooling), the resolution of the image is reduced by 2, 4, 8, 16, 32 times. Regarding the output image of the last layer, it is necessary to stop upsampling 32 times to get the same size as the original image.
This upsampling is done by deconvolution. The output of the 5th layer (32x magnification) is deconvolved to the original image size, and the result is still not accurate enough, and some details cannot be recovered. So Jonathan deconvolved the output of the 4th layer and the output of the 3rd layer in turn, requiring 16x and 8x upsampling respectively, and the result is more precise. The following figure shows the process of upsampling this convolution and deconvolution:
Write a picture here
The figure below is a comparison of the results obtained by upsampling 32 times, 16 times and 8 times. It can be seen that the results obtained by them are more and more accurate:
Write a picture here
The advantages and disadvantages of FCN
Compared with the traditional way of stopping image segmentation with CNN, FCN has two distinct advantages: one is the ability to withstand the size of the input image, without asking for all the exercise image and the test image to have the same size. Second, it is more efficient, because it avoids the problem of repeated storage and computational convolution due to the use of pixel blocks.
At the same time, the defects of FCN are also obvious: First, the results obtained are still not precise enough. Stopping 8x upsampling is a lot better than 32x, but the result of upsampling is still more ambiguous and less sensitive to details in the image. The second is to stop classifying each pixel, without thinking about the relationship between pixels and pixels, neglecting the spatial regularization step used in the usual pixel classification-based segmentation method, lacking spatial divergence.
Since 2012, Convolutional Neural Networks (CNN) have achieved great achievements and universal applications in image classification and image detection.
The power of CNN is that its multi-layer structure can automatically learn features, and can learn multiple levels of features: shallower convolutional layer sensing domain is small, learn some features of partial regions; deep convolution The layer has a larger sensing domain and can learn more general features. These general features are less sensitive to the size, position, and orientation of the object, helping to identify performance improvements.
These general features are very helpful for classification, and can well discriminate what kind of objects are contained in an image, but because of the loss of the details of some objects, the detailed contour of the object cannot be well given, and each pixel is pointed out. Which object belongs in detail, so it is very difficult to achieve accurate segmentation.
The traditional CNN-based segmentation approach is usually to classify a pixel by using an image block around the pixel as input to the CNN for exercise and prediction. There are several drawbacks to this approach: First, storage costs are high. For example, if the size of the image block applied to each pixel is 15x15, the required storage space is 225 times that of the original image. Second, the calculation efficiency is low. Adjacent blocks of pixels are fundamentally iterative, and convolution is calculated one by one for each block of pixels, and this calculation also has a large level of iteration. Third, the size of the pixel block limits the size of the perceived area. Usually, the size of a pixel block is much smaller than the size of the entire image, and only some of the features can be extracted, thereby causing a limitation in the performance of the classification.
In response to this problem, UC Berkeley's Jonathan Long et al. proposed Fully Convolutional Networks (FCN) [1] [1] for image segmentation. The network attempts to recover the category to which each pixel belongs from a general feature. That is, the classification from the image level is further extended to the classification at the pixel level.
The principle of FCN
The FCN converts the full connectivity layer in a traditional CNN into a convolutional layer. As shown in the following figure, in the traditional CNN structure, the first 5 layers are convolutional layers, the 6th and 7th layers are respectively a one-dimensional vector with a length of 4096, and the eighth layer is a one-dimensional vector with a length of 1000. Corresponds to the probability of 1000 categories. The FCN expresses these three layers as a convolutional layer, and the size (channel number, width, and height) of the convolution kernel are (4096, 1, 1), (4096, 1, 1), (1000, 1, 1), respectively. All layers are convolutional layers, so they are called full convolutional networks.
Write a picture here
It can be found that after repeated convolution (and pooling), the resulting image is getting smaller and smaller, and the resolution is getting lower and lower (rough image). How does FCN get the category of each pixel in the image? In order to recover from this coarse image with low resolution to the resolution of the original image, FCN uses upsampling. For example, after 5 convolutions (and pooling), the resolution of the image is reduced by 2, 4, 8, 16, 32 times. Regarding the output image of the last layer, it is necessary to stop upsampling 32 times to get the same size as the original image.
This upsampling is done by deconvolution. The output of the 5th layer (32x magnification) is deconvolved to the original image size, and the result is still not accurate enough, and some details cannot be recovered. So Jonathan deconvolved the output of the 4th layer and the output of the 3rd layer in turn, requiring 16x and 8x upsampling respectively, and the result is more precise. The following figure shows the process of upsampling this convolution and deconvolution:
Write a picture here
The figure below is a comparison of the results obtained by upsampling 32 times, 16 times and 8 times. It can be seen that the results obtained by them are more and more accurate:
Write a picture here
The advantages and disadvantages of FCN
Compared with the traditional way of stopping image segmentation with CNN, FCN has two distinct advantages: one is the ability to withstand the size of the input image, without asking for all the exercise image and the test image to have the same size. Second, it is more efficient, because it avoids the problem of repeated storage and computational convolution due to the use of pixel blocks.
At the same time, the defects of FCN are also obvious: First, the results obtained are still not precise enough. Stopping 8x upsampling is a lot better than 32x, but the result of upsampling is still more ambiguous and less sensitive to details in the image. The second is to stop classifying each pixel, without thinking about the relationship between pixels and pixels, neglecting the spatial regularization step used in the usual pixel classification-based segmentation method, lacking spatial divergence.