Project 3 - Face Morphing

Filip Malm-Bägén

Introduction

This project goes though into the techniques of seamlessly transitioning between two images by manipulating the mesh of corresponding points and blending the colors. By performing these operations, it is possible to compute the average face of a group and create caricatures derived from this average face.

Defining Correspondences

Approach

The corresponding points between two images were defined thoguh this tool. I visualized these points on both faces using OpenCV. Then, I calculated the average positions to create a midway shape and applied Delaunay triangulation to these points. This triangulation will guide the smooth transition during the morphing process.

Result

Computing the "Mid-way Face"

To compute the "mid-way face", I follow three steps: (1) computing the average shape, (2) warping both faces to that shape, and (3) averaging the warped images' colors.

Approach

Step 1: Computing the Average Shape

The average shape is calculated by taking the element-wise mean of the keypoints from both images: avg_points = (points_a + points_b) / 2. I thereafter applied Delaunay triangulation using the average points to create triangles for warping.

Step 2: Warping the Faces

For each triangle, we compute an affine transformation matrix between corresponding triangles in both faces and the average shape: A = compute_affine(tri_a, tri_avg). Using inverse warping, we map the pixels from the source image triangles to the average shape. This is applied to both images, creating warped versions warped_a and warped_b.

Step 3: Averaging Colors

Finally, the mid-way face was computed by averaging the warped images: midway_face = (warped_a + warped_b) / 2. This produces a blended face that combines both images' geometry and color.

As seen in the result, there is a slight error in the mid-way face, especially around the hair and shoulders. I think this might be due to the points I selected not being perfectly positioned. If one zooms in, there are artifacts around the face, and a horizontal line across the nose.

Result

The Morph Sequence

Approach

The morphing process involves two main components: shape warping and cross-dissolving. For each frame in the sequence, both images are warped to an intermediate shape defined by a warp_frac parameter, which linearly increases from 0 to 1 over the course of the video. The warping is achieved by applying affine transformations to corresponding triangles from the Delaunay triangulation of the average shape. After warping, the color values of the warped images are cross-dissolved using a weighted average based on dissolve_frac, which also linearly increases from 0 to 1. The final frame is produced by blending the morphed face with the background.

Result

The "Mean face" of a population

Approach

For this task i used the IMM Face Database. The code works by loading the facial images and their landmark points into images and landmarks, normalizing the landmarks to the image dimensions. It computes the average shape average_shape, then uses Delaunay triangulation and affine transformations to morph each image towards this average. Finally, it combines the morphed images to create and display an average male face with a neutral expression.

Result

Here are a few examples of faces in the dataset warped to fit the average face. The result looks somewhat good, but there are still some artifacts around the eyes and mouth.

Part 5. Caricatures: Extrapolating from the mean

Approach

To create the caricature, I started by collecting and averaging the landmarks of female happy faces using load_images_and_landmarks and compute_average_shape. After normalizing these landmarks, I created a caricature of my face by exaggerating the difference between my landmarks and the average using the formula caricature = scale * me + (1 - scale) * avg. Finally, I warped my face to fit these caricatured landmarks with morph_to_average_shape and displayed the result.

Result

Bells and Whistles

I created a morph-video-chain of my friends in Sweden. To to this, I used the tool to define the corresponding points between each image in a sequence, so i first tagged face_a with face_b, then face_b with face_c, and so on. I then used the same approach as in the previous sections to create the morphing sequence. The result is a fun video of my friends morphing into each other. The result is fun, but it is not great. I used images a had laying around, which resulted in some bad morphs. The morphing is not very smooth, and the faces are not perfectly aligned. It would have been much better if I used better images, or aligned the faces better.

It is also clear on which images I've been sloppy with the correspondance points. On these morphings, the faces are not perfectly aligned, and the morphing is not smooth. I should have placed a lot more points, especially around the contour of the face. On some transitions, it just looks like the faces are sliding around.