Did you notice that Facebook has recently developed an unusual feature: identify friends in your photos. In the past, Facebook allowed you to manually click on friends in photos, enter their names, and then add tags. Now whenever you upload a photo, Facebook will automatically tag everyone for you like magic:



 

This technology is called face recognition. After your friend's face is marked several times, Facebook's algorithm recognizes him. This is an amazing technology - Facebook's face recognition rate is as high as 98%! This is almost the same as human performance.

 

Let's learn how face recognition technology is implemented below! But just identifying your friend's face is too simple. We can maximize this technology to solve a more challenging problem - distinguish between Will Ferrell (famous actor) and Chad Smith (famous rock musician)!



 

 

How to use machine learning to solve complex problems
 

Face recognition consists of a series of related questions:

 

1. First: Look at a photo and find all the faces above

2. Focus on each face, even if the face is turned in a strange direction or when the light is poor.

3. Pick some features from this face to distinguish them from others, such as how big your eyes are, how long your face is, and so on.

4. Finally, compare the features of this face with other faces to finally determine the person's name.

 

As a human being, your brain will do these things automatically. In fact, human beings are too good at recognizing human faces, so that they will try to find faces on everyday objects (as if they were like this, people always like to find shapes on objects, and they feel so adorable). .

 

Computers currently do not have this high level of capabilities. . . So we need to teach them step by step.

 

We need to build a pipeline: We will solve each step of face recognition separately and pass the result of the current step to the next step. In other words, we need to chain several machine learning algorithms.



 

Face recognition - step by step

 

We solve this problem step by step. In each step, we will learn different machine learning algorithms. I will not explain every step of the algorithm, but you can learn the main idea of each algorithm, and how to use OpenFace and dlib to build your own facial recognition system in Python.

 

Step 1: Find all faces

 

The first step in our assembly line is face detection. Obviously we need to mark the face in the picture before we can distinguish the face.

 

If you have used a camera in the last decade, you may have seen face detection in action:



 

Face recognition is a great feature of the camera. When the camera can automatically pick out the face, this will ensure that when the photo is taken, all the faces are in focus. However, we use it for other purposes - to find the areas of the photo we want to transfer in the next step.

 

In early 2000, face detection became mainstream when Paul Viola and Michael Jones invented a face detection technology that could quickly run on cheap cameras. However, now more reliable solutions have emerged. What we are using now is a direction gradient histogram invented in 2005, abbreviated as HOG.

 

In order to recognize the face in the picture, we first need to convert the picture to black and white because we don't need color data when we recognize the face.

 

Then we need to iterate through each pixel in the image in turn. For a single pixel, we also need to look at other elements that directly surround it:

 



 

Our goal is to compare the depth of this pixel with the surrounding pixels. Then we have to draw an arrow to represent the direction of the darkening of the image:



 

 

If you repeat this process for each pixel in the image, each pixel will eventually be replaced by an arrow. These arrows are called gradients. They can show the flow from bright to dark on the image:



 

 

This does not seem to have a clear purpose, but it is actually necessary. If we analyze the pixels directly, the two photos of the same person in different shades will have completely different pixel values. But if you only consider the direction of brightness change, the light and dark images will have the same result. This makes the problem easier to solve!

 

But saving the gradient of each pixel is too detailed, and we end up losing the watermelon in sesame. If we can observe the basic flow of light and darkness from a higher perspective, we can see the basic laws of the image, which will be better than before.

 

To do this, we divide the image into small blocks of 16 x 16 pixels. In each small square, we will calculate how many gradients are in each main direction (how many points up, point to the top right, point to the right, etc.). Then we will use the arrow with the most directivity to replace the small cube.

 

The end result is that we convert the original image into a very simple form of expression that can capture the basic structure of the face in a simple way:



 

The original image is represented as a HOG form to capture the main features of the image, regardless of the brightness of the image.

 

In order to find a face in this HOG image, all we need to do is find the part of our image that looks most similar to some of the known HOG patterns. These HOG patterns are extracted from other facial training data:

 

Using this technique, we can now easily find faces in any picture:

 



 

 

If you want to try it yourself with Python and dlib, this code shows how to generate and view HOG images.

 

Step 2: Different postures and orientations of the face
 

When it's time, we separate the face from the picture. But now, the problem we have to deal with is that for computers, the same face that faces in different directions is two people:



Humans can easily recognize that the two pictures are the same person, but the computer will think that these two pictures are two completely different people.

 

To solve this, we will try to distort each picture so that the eyes and lips are always at the sample place in the image. This will make it easier for us to compare the differences between faces in the next steps.

 

To do this, we will use an algorithm called face landmark estimation. There are many ways to do this, but this time we will use methods invented in 2014 by Vahid Kazemi and Josephine Sullivan.

 

The basic idea is to find landmarks that are commonly found on 68 faces - including the top of the chin, the outer outline of each eye, the inner contour of each eyebrow, and so on. Next we train a machine learning algorithm that allows it to find these 68 specific points on any face:



 

 

We will locate 68 feature points on each face. The author of this picture is Brandon Amos of Carnegie Mellon University, Ph.D., who works at OpenFace.

 

This is the result of locating 68 feature points on the test image:



 

You can also use this technology to implement your own Snapchat real-time 3D face filter!

 

Now that we know where the eyes and mouth are, we rotate, scale and miscut the image so that the eyes and mouth are as close as possible to the center. We will not do any fancy three-dimensional warping, as this will distort the image. We will only use basic image transformations that can keep images relatively parallel, such as rotation and scaling (called affine transformations):

 

Now no matter which side of the human face we face, we can move our eyes and mouth to the middle of the same position. This will make our next step more accurate.

 

If you want to try this step yourself with Python and dlib, there is some code to help you find facial feature points and use these feature points to complete image warping.

 

The third step: code the face
 

Now we have to face the core problem - to accurately identify different faces. This is the interesting part of this matter!

 

The simplest method of face recognition is to compare the unknown face we found in the second step with the face picture we have labeled. When we find that an unknown face looks similar to a previously-labeled face, it must be the same person. This seems perfect, right?

 

In fact this method has a huge problem. For websites like Facebook, which have billions of users and trillions of photos, it's impossible to cycle through and compare each previously-marked face, which is a waste of time. They need to recognize faces in milliseconds instead of hours.

 

The method we need is to extract some basic measurement values from each person's face. Then, we can measure the unknown face in the same way and find the known face that is closest to the measured value. For example, we can measure the size of each ear, the distance between the eyes, the length of the nose, and so on. If you've ever seen a TV show like Crime Scene Investigation, you know what I'm talking about.

 

 

The most reliable way to measure your face
 

OK, so in order to build our known face database, what values should we measure on the face? The size of the ear? The length of the nose? Eye color? What else?

 

It turns out that for us humans, some obvious measurements (such as eye color) are meaningless to the computer. The researchers found that the most accurate method is to let the computer find out the measurement value it wants to collect. Deep learning performs better than humans in finding which parts of the measurements are more important.

 

So, the solution is to train a deep convolutional neural network. However, instead of letting it identify objects in the picture, this time our training is to have it generate 128 measurements for the face.

 

Observe three different facial images every time:

 

1. Load a known person's facial training image

 

2. Load another photo of the same person

 

3. Load another person's photo

 

The algorithm then looks at the measurements it generates for these three images. Then, the neural network is slightly adjusted to ensure that the first and second sheets produce measurements that are close, while the second and third sheets produce slightly different measurements.



 

 

After repeating this step several million times for millions of images of thousands of people, the neural network learned how to reliably generate 128 measurements for each person. For any ten different photos of the same person, it should give roughly the same measurements.

 

Machine learning professionals refer to 128 measurements per face as an embedding. The method of reducing complex raw data (such as pictures) into a sequence that can be generated by a computer has appeared many times in machine learning (especially language translation). The face extraction method we are using was invented by Google researchers in 2015, but there are many similar methods.

 

Encode our facial image
 

This process of outputting face embedding through the training of convolutional neural networks requires a great deal of data and powerful computing power. Even with the expensive Nvidia Telsa graphics card, you need about 24 hours of continuous training to get good accuracy.

 

But once the network training is complete, it can generate measurements for each face, even if it has never seen this face before! So this training can be done only once. Fortunately, the big cattle on OpenFace have already done this, and they released a few networks that they had trained to use. Thanks Brandon Amos for his team!

 

So what we need to do is to process our facial images through their pre-trained network to get 128 measurements. Here are some measurements of our test image:



 

 

So what parts of the face were measured by these 128 numbers? Of course we do not know, but this is not important to us. Our concern is that when we see the same person with two different pictures, our network can get almost the same value.

 

If you want to try this step yourself, OpenFace provides a lua script that can generate embeddings of all the images in a folder and write them to the csv file. Click here to see how it works.

 

Step 4: Find the person's name from the code
 

The last step is actually the simplest step in the whole process. All we have to do is find the person in the database that is closest to the measured value of our test image.

 

You can achieve this goal with any basic machine learning classification algorithm. We do not need too fancy deep learning skills. We will use a simple linear SVM classifier, but there are actually many other classification algorithms available.

 

What we need to do is train a classifier that can take measurements from a new test image and find the closest match. It takes only a few milliseconds for the classifier to run once. The result of the classifier is the name of the person!

 

So let's try our system. First of all, I trained the classifier using the embedding of 20 photos per person by Will Ferrell, Chad Smith and Jimmy Falon:



 

Hmmm...it's these training data!

 

Next, I ran each frame of that video that Will Verrell and Chad Smith imitated on Jimmy Fallon's show on this classifier:

 
 

 

The result was successful! Faces of different angles, even side faces, can be captured!

 

Do it yourself
 

Let us review our steps:

 

1. Use HOG algorithm