#009 How to detect facial landmarks using DLIB and OpenCV

datahacker.rs Other 10.06.2020 | 0

Highlight: In this post you are going to learn how to detect facial landmarks in an image using dlib and OpenCV. This knowledge will be useful, since facial landmarks are the key features in a large number of facial analysis methods and algorithms. Face recognition, face alignment, facial expression recognition, face swapping, drowsiness detection, blink detection, head pose estimation, are the few examples in which facial landmarks play a fundamental role. let us see what facial landmarks are and how can we detect them.

Tutorial Overview:

What-are facial landmarks?
How to detect facial landmarks?

1. What are facial landmarks?

Face Swapping pictures are the extremely popular trend on social media. Snapchat, Cupace, MSQRD are probably the most widely used apps having the face swapping option. In few seconds you can easily swap your face with your friend’s face or with some funny features. However, although face swapping seems to be very simple, it is not an easy task. Now you may wonder “how those apps can perform such advanced face swapping”? Keep reading this post and you will find the answer soon.

To perform face swapping, we cannot just crop one face and replace it with another. What we need to do is to localize the key points that describe the unique location of a facial component in an image (eyes, nose eyebrows, mouth, jawline etc.). To do this, we need to develop a shape prediction method that identifies important facial structures. In our code we are going to implement a method developed by two Swedish Computer Vision researchers Kazemi and Sullivan in 2014, called One Millisecond Face Alignment with an Ensemble of Regression Trees.This detector is built in the dlib library and it detects facial landmarks very quickly and accurately. To better understand this method, have a look at the following image.

In this picture you can see a training set of 68 labeled facial points with specific coordinates that surround certain parts of the face.

JAWLINE POINTS: 1 – 17
RIGHT EYEBROW POINTS: 17 – 22
LEFT EYEBROW POINTS: 22 – 27
NOSE BRIDGE POINTS: 27 – 31
LOWER NOSE POINTS: 31 – 36
RIGHT EYE POINTS: 36 – 42
LEFT EYE POINTS: 42 – 48
LEFT EYE POINTS: 42 – 48
MOUTH INNER POINTS: 61 – 68

For a more detailed explanation of this method you can look at the original research paper by Kazemi and Sullivan here.

2. How to detect facial landmarks?

Detecting the faces in the image

To detect facial landmarks, we are going to use dlib library. This process consists of two steps:

To localize the face on the image or video
To detect the facial landmarks

First, we will detect the face in the input image. Then we will use same method to detect the facial landmarks. It is important to mention that we can use different methods for face detection. But our goal here is to obtain the coordinates of a face bounding box, and coordinates of a key facial points using the same method. Let us have a look in the code below.

First, we need to import the necessary packages. Next, we will load the input image and convert it to a grayscale format.

# Necessary imports
import cv2
import dlib
import numpy as np
from google.colab.patches import cv2_imshow

# Loading the image and converting it to grayscale
img= cv2.imread('Capture 8.PNG')
gray=cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

To detect the face in our image we need to call a frontal face detector dlib.get_frontal_face_detector() from dlib library. This is a pre-trained detector based on Histogram of Oriented Gradients (HOG) features, and a linear classifier in a sliding window detection approach. Then we will create the object called faces in which we will store all detected faces.

# Initialize dlib's face detector
detector = dlib.get_frontal_face_detector()
# Detecting faces in the grayscale image
faces = detector(gray)
print(faces)

Output:

rectangles[[(128, 199) (770, 841)]]

As you can see from the output above, in the object named faces, we have a list of coordinates of the detected faces. In the above example we obtained coordinates for only two points, which indicates that we have only one face in the image. These two points represent our face bounding box (rectangle): first point is at the top left corner, and second is at the bottom right corner of a rectangle.

Now we need to draw the rectangle around the face. We can do this using the following code:.

# Creating a for loop in order to extract
# specific coordinates (x1,x2,y1,y2)
for face in faces:
  x1=face.left()
  y1=face.top()
  x2=face.right()
  y2=face.bottom()
# Drawing a rectangle around the face
  cv2.rectangle(img, (x1,y1), (x2,y2),(0,255,0),3)
cv2_imshow(img)

We created a for loop in order to extract coordinates from the obtained two points. Here \((x_{1},y_{1}) \) are coordinates of the top left point, and \((x_{2},y_{2}) \) are coordinates of a bottom right point. Finally, using the function cv2.rectangle(), we draw a rectangle around the detected face. let us have a look at the output image.

Output:

Detecting facial landmarks

To detect the facial landmarks, we will use the similar method. First, we will load the facial landmark predictor dlib.shape_predictor from dlib library. Additionally, for this shape prediction method, we need to download the file called "shape_predictor_68_face_landmarks.dat". Using following command, you can download and unzip this file directly to your python script.

!wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
!bunzip2 "shape_predictor_68_face_landmarks.dat.bz2"

Then, we will use this predictor to extract the key facial features from the input image. First, we will create a predictor object called landmarks in which will pass two arguments. As a first argument we will pass a gray image from which we want to detect available faces. As the second argument we will specify the area where we are going to predict the facial landmarks. In our case, this area is represented by coordinates of the face that we already detected in the previous step.

p = "shape_predictor_68_face_landmarks.dat"
# Initialize dlib's shape predictor
predictor = dlib.shape_predictor(p)
# Get the shape using the predictor
landmarks=predictor(gray, face)

Now let’s see how we can extract coordinates of the facial landmarks from this object. For Now let us see how we can extract coordinates of the facial landmarks from this object. For example, if we want to extract the nose bridge point with index 31. We can do this in the following way:

# Defining x and y coordinates of a specific point
x=landmarks.part(31).x
y=landmarks.part(31).y
# Drawing a circle
cv2.circle(img, (x, y), 6, (0, 0, 255), -1)
cv2_imshow(img)

We simply defined \(x \) and \(y \) coordinate of one specific point (in our case it is point number 31). After this we draw a circle using these coordinates. You can see this result in the following image:

Output:

If we want to extract different point, we only need to write an index number of that point in our code.

Finally, we extracted all 68 points. Instead detecting them one at the time, we will create a for loop and extract all 68 points.

 for n in range(0,68):
    x=landmarks.part(n).x
    y=landmarks.part(n).y
    cv2.circle(img, (x, y), 4, (0, 0, 255), -1)
cv2_imshow(img)

Output:

We can clearly see that the small red circles are mapped to the specific facial features on the face (eyes, nose, mouth, eyebrows, jawline).

Now let us have a look at one more example. For better visualization we are going to load different input image containing multiple faces.

import cv2
import dlib
import numpy as np
from google.colab.patches import cv2_imshow

img= cv2.imread('GoT.jpg')
gray=cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

!wget http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
!bunzip2 "shape_predictor_68_face_landmarks.dat.bz2"

p = "shape_predictor_68_face_landmarks.dat"
detector = dlib.get_frontal_face_detector()
predictor = dlib.shape_predictor(p)
faces = detector(gray)

for face in faces:
  x1=face.left()
  y1=face.top()
  x2=face.right()
  y2=face.bottom()
  cv2.rectangle(img, (x1,y1), (x2,y2),(0,255,0),3)
  landmarks=predictor(gray, face)
  for n in range(0,68):
    x=landmarks.part(n).x
    y=landmarks.part(n).y
    cv2.circle(img, (x, y), 4, (0, 0, 255), -1)
cv2_imshow(img)

Output:

Summary

In this post we learned what facial landmarks are and how to detect them using dlib In this post we learned what facial landmarks are and how to detect them using dlib library. Now we can apply these facial landmarks detection process in several computer vision techniques. In the next post we will use one of these techniques for face alignment.

References:

[1] Face landmarks detection – Opencv with Python by Sergio Canu

#009 How to detect facial landmarks using DLIB and OpenCV