Procrustes Analysis in MediaPipe Face Mesh - From Screen Coordinates to 3D Face Pose

Introduction

While developing the mediapipe_face_mesh package, I had a chance to dig into MediaPipe’s Face Geometry module.

The Face Mesh model outputs 468 facial landmarks in screen coordinates. But screen coordinates alone are not enough to anchor an AR filter onto a face or compute head orientation. What you need is a pose transform matrix that answers: “where in 3D space, and in what orientation, is the face sitting in front of the camera?”

MediaPipe solves this with a combination of a virtual camera and Procrustes analysis. In this post I walk through that process by following the actual source code.

The code under analysis is MediaPipe’s face_geometry module, and the two key files are geometry_pipeline.cc and procrustes_solver.cc.

The Limits of Face Mesh Output

The landmarks produced by the Face Mesh model come in the following coordinate system.

• x, y : screen coordinates normalized to [0, 1]
• z : relative depth with the center of the head as the origin, using roughly the same scale as x

In other words, z only encodes relative depth within the face — “the nose is closer to the camera than the ears” — and tells you nothing about whether the face is 30cm or 1m away from the camera. There is no absolute position and no unit.

To get into a 3D space with real units, two more things are needed.

1. A reference face model — a standard face with real-world measurements
2. A camera model — the projection rule that connects the screen to 3D space

Canonical Face Model

MediaPipe ships with a standard face mesh called the canonical face model. It consists of 468 vertices defined in a metric 3D space in centimeters, corresponding 1:1 with the Face Mesh landmarks.

The idea goes like this. If we can rotate, scale, and translate this standard face in 3D space so that it overlaps the observed landmarks as closely as possible, then the transform we used is precisely the pose of the face.

\[\text{pose transform} \;=\; \text{canonical face} \rightarrow \text{runtime face alignment transform}\]

“Find the optimal transform that best overlaps two point sets” — that is exactly the problem Procrustes analysis solves.

Procrustes Analysis

The name comes from Greek mythology. Procrustes was a bandit who laid travelers on his bed and made them fit it — stretching them if they were too short, cutting off their limbs if they were too tall. The statistical technique for aligning shapes took his name: deforming a subject to match a reference shape (the bed).

Problem Statement

Given a source point set $a_i$ (canonical landmarks), a target point set $b_i$ (observed landmarks), and per-point weights $w_i$, find the scale $s$, rotation $R$, and translation $t$ that minimize:

\[\underset{s,\,R,\,t}{\arg\min} \; \sum_i w_i \,\bigl\| \, s R a_i + t - b_i \, \bigr\|^2 \quad \text{s.t.} \;\; R^\top R = I,\;\; \det(R) = +1\]

Because $R$ is constrained to be orthogonal, this is called the orthogonal Procrustes problem. It looks like a nonlinear optimization, but remarkably it has a closed-form solution via SVD. MediaPipe’s implementation is the generalized weighted version, and the code comments cite Akca, Generalized Procrustes analysis and its applications in photogrammetry (2003) as the basis.

The Solution

SolveWeightedOrthogonalProblem in procrustes_solver.cc derives the solution in the following order.

1) Weighted centroids and centering

\[\bar{a} = \frac{\sum_i w_i a_i}{\sum_i w_i}, \qquad \bar{b} = \frac{\sum_i w_i b_i}{\sum_i w_i}\]

Both point sets are shifted to their respective weighted centroids, which eliminates the translation $t$ from the problem.

2) Design matrix (cross-covariance)

\[M = \sum_i w_i \,(b_i - \bar{b})(a_i - \bar{a})^\top\]

A $3 \times 3$ matrix capturing the correlation structure between the two point sets.

3) Optimal rotation via SVD

\[M = U \Sigma V^\top, \qquad R = U \begin{bmatrix} 1 & & \\ & 1 & \\ & & \det(UV^\top) \end{bmatrix} V^\top\]

The diagonal matrix in the middle is a guard against reflection. Pure minimization can produce a mirror-image solution with $\det(R) = -1$, which is meaningless for a face, so the sign of the column corresponding to the smallest singular value is flipped to guarantee $\det(R) = +1$.

// Disallow reflection by ensuring that det(`rotation`) = +1 (and not -1)
if (postrotation.determinant() * prerotation.determinant() < 0) {
  postrotation.col(2) *= -1;
}
rotation = postrotation * prerotation;

4) Optimal scale

\[s = \frac{\sum_i w_i \,\bigl\langle R(a_i - \bar{a}),\; b_i - \bar{b} \bigr\rangle}{\sum_i w_i \,\| a_i - \bar{a} \|^2}\]

The ratio obtained by projecting the rotated source onto the target.

5) Optimal translation

\[t = \bar{b} - sR\,\bar{a}\]

The final result is a $4 \times 4$ transform matrix combining all of these.

\[T = \begin{bmatrix} sR & t \\ 0 & 1 \end{bmatrix}\]

Why the Weights Matter

What Procrustes recovers is a rigid transform plus scale. But a face is not a rigid body. Opening your mouth or closing your eyes moves landmarks — that’s an expression change, not a pose change. If all 468 points are treated equally, expressions leak into the pose estimate.

That’s why MediaPipe defines a weight list called procrustes_landmark_basis in the geometry_pipeline_metadata. Only 33 of the 468 landmarks get a non-zero weight — points around the nose, the eye corners, and the face contour, i.e., the points that behave most like a rigid body regardless of expression.

procrustes_landmark_basis { landmark_id: 4   weight: 0.070909 }  # nose tip
procrustes_landmark_basis { landmark_id: 6   weight: 0.032100 }
procrustes_landmark_basis { landmark_id: 33  weight: 0.058724 }  # eye corner
procrustes_landmark_basis { landmark_id: 129 weight: 0.120625 }  # side of nose
...

The largest weights (about 0.12) sit on the points beside the nose (129, 358). The nose barely deforms under expressions — it is the closest thing to a rigid body on the face.

The Virtual Camera and the Camera Matrix

Pinhole Camera Model

To run Procrustes, the observed landmarks must live in the same metric 3D space as the canonical model. Bringing screen coordinates back into 3D (unprojection) requires a camera model.

In the standard pinhole camera model, a 3D point $(X, Y, Z)$ projects onto the image as:

\[\begin{bmatrix} x \\ y \\ 1 \end{bmatrix} \sim K \begin{bmatrix} X \\ Y \\ Z \end{bmatrix} = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} X \\ Y \\ Z \end{bmatrix} \;\;\Longrightarrow\;\; x = f_x \frac{X}{Z} + c_x, \quad y = f_y \frac{Y}{Z} + c_y\]

Here $K$ is the camera intrinsic matrix, $f_x, f_y$ are focal lengths in pixels, and $(c_x, c_y)$ is the principal point. The key insight is that projection is division by depth $Z$. The entire principle of perspective — distant things look smaller — lives in that one division.

MediaPipe’s Virtual Camera

The problem is that ordinary users’ cameras are not calibrated. $K$ is unknown. So instead of demanding real camera parameters, MediaPipe assumes a virtual perspective camera with reasonable defaults.

environment: {
  origin_point_location: TOP_LEFT_CORNER
  perspective_camera: {
    vertical_fov_degrees: 63.0  # 63 degrees
    near: 1.0    # 1cm
    far: 10000.0 # 100m
  }
}

The camera is defined not by an intrinsic matrix but by a vertical field of view (FOV) and near/far clipping planes — an OpenGL-style camera description. The two representations are equivalent and convert as:

\[f_y = \frac{h}{2 \tan(\theta_v / 2)} \quad\Longleftrightarrow\quad \theta_v = 2 \arctan\!\left(\frac{h}{2 f_y}\right)\]

($h$: image height in pixels, $\theta_v$: vertical FOV)

geometry_pipeline.cc computes the view frustum from this camera definition.

const float height_at_near =
    2.f * perspective_camera.near() *
    std::tan(0.5f * kDegreesToRadians *
             perspective_camera.vertical_fov_degrees());

const float width_at_near = frame_width * height_at_near / frame_height;

left   = -0.5f * width_at_near;
right  =  0.5f * width_at_near;
bottom = -0.5f * height_at_near;
top    =  0.5f * height_at_near;

This code computes the physical size of the frustum at the near plane (1cm from the camera). With a 63° FOV, the near-plane height comes out to $2 \times 1 \times \tan(31.5°) \approx 1.23$cm.

Since this is a virtual camera, its FOV may differ from the actual camera’s. In that case the rotation (orientation) stays largely intact, but the absolute distance and scale become inaccurate. If you need the true distance to the face, you should feed an FOV matching your camera’s intrinsics into the environment.

Unprojection

If projection was $x = f\,X/Z$, then unprojection is multiplying the screen coordinate by depth. Given a coordinate $(x, y)$ on the near plane and a depth $Z$:

\[X = x \cdot \frac{Z}{near}, \qquad Y = y \cdot \frac{Z}{near}\]

The code is exactly this.

static void UnprojectXY(const PerspectiveCameraFrustum& pcf,
                        Eigen::Matrix3Xf& landmarks) {
  landmarks.row(0) = landmarks.row(0).cwiseProduct(landmarks.row(2)) / pcf.near;
  landmarks.row(1) = landmarks.row(1).cwiseProduct(landmarks.row(2)) / pcf.near;
}

But here we hit a chicken-and-egg problem.

• Unprojection requires an absolute depth $Z$.
• To know $Z$, you need the scale of the face relative to its real size.
• The scale comes from a Procrustes alignment against the canonical model.
• But Procrustes needs metric coordinates — i.e., unprojected coordinates.

MediaPipe breaks this cycle with an iterative estimation that runs Procrustes twice.

The Full Geometry Pipeline

This is the actual processing order of ScreenToMetricSpaceConverter::Convert.

(1) ProjectXY — screen coordinates onto the near plane

Normalized coordinates in [0, 1] are mapped to physical near-plane coordinates in [left, right] × [bottom, top]. If the origin is at the top-left (TOP_LEFT_CORNER), y is flipped, and z is scaled by the same factor (x_scale). The landmarks now form “a cm-scale drawing of the face painted on the near plane.”

(2) First Procrustes — initial scale estimate

Procrustes is solved between the near-plane landmarks and the canonical model to obtain the first scale $s_1$.

absl::StatusOr<float> EstimateScale(Eigen::Matrix3Xf& landmarks) const {
  Eigen::Matrix4f transform_mat;
  MP_RETURN_IF_ERROR(procrustes_solver_->SolveWeightedOrthogonalProblem(
      canonical_metric_landmarks_, landmarks, landmark_weights_,
      transform_mat));
  return transform_mat.col(0).norm();
}

The top-left $3 \times 3$ of the transform matrix is $sR$, and since the columns of $R$ are unit vectors, the norm of the first column is the scale $s$.

The source comments explain directly why no unprojection happens at this stage. Because z is relative and there is no absolute depth yet, unprojection would be unsafe (“it is not safe to unproject due to the relative nature of the input screen landmark Z coordinate”) — so a rough scale is first estimated from the projected XY-plane coordinates alone.

(3) First unprojection — intermediate landmarks

Using $s_1$, the depths are repositioned and the landmarks unprojected.

static void MoveAndRescaleZ(const PerspectiveCameraFrustum& pcf,
                            float depth_offset, float scale,
                            Eigen::Matrix3Xf& landmarks) {
  landmarks.row(2) =
      (landmarks.array().row(2) - depth_offset + pcf.near) / scale;
}

Subtracting depth_offset (the mean of z) places the face center at the near plane, then everything is divided by the scale. The geometric intuition is similar triangles: if the face projected onto the near plane is $s$ times larger than the canonical face, the real face sits at roughly $near/s$ away. With near at 1cm, if the projected face is 1/30 the size of the standard face, the real face is about 30cm away.

UnprojectXY then unprojects XY, and the z sign is flipped to fix handedness (screen coordinates are left-handed, the metric space is right-handed).

(4) Second Procrustes — scale refinement

Now that the landmarks form a plausible metric shape with perspective removed, the same Procrustes runs once more to obtain a correction scale $s_2$. The first estimate was computed while perspective distortion was still present, so it carries some error; the second pass corrects it.

(5) Final unprojection

\[s_{total} = s_1 \times s_2\]

Applying MoveAndRescaleZ → UnprojectXY → ChangeHandedness to the original screen landmarks with $s_{total}$ yields the final metric landmarks.

(6) Final Procrustes — the pose transform matrix

Solving Procrustes one more time between the canonical model and the final metric landmarks produces exactly what we were after: the face pose transform matrix.

MP_RETURN_IF_ERROR(procrustes_solver_->SolveWeightedOrthogonalProblem(
    canonical_metric_landmarks_, metric_landmarks, landmark_weights_,
    pose_transform_mat));

// Multiply each of the metric landmarks by the inverse pose
// transformation matrix to align the runtime metric face landmarks with
// the canonical metric face landmarks.
metric_landmarks = (pose_transform_mat.inverse() *
                    metric_landmarks.colwise().homogeneous())
                       .topRows(3);

The last line is interesting: the metric landmarks used for output are multiplied by the inverse of the pose. The final output therefore splits into two parts.

• mesh (metric landmarks) : the face shape with the pose removed, aligned to the canonical model’s position. A pure face shape (including expression), independent of pose.
• pose_transform_matrix : the $4 \times 4$ transform that carries that shape to its actual position and orientation in front of the camera.

Because shape and pose are separated, AR rendering can model effects once in canonical space and attach them to the face by multiplying with just the pose matrix.

Summary

The whole pipeline, one line per step:

Step	Operation	Purpose
1	ProjectXY	Normalized coordinates → cm-scale coordinates on the near plane
2	Procrustes ①	Initial scale estimate $s_1$
3	MoveAndRescaleZ + UnprojectXY	First unprojection based on $s_1$
4	Procrustes ②	Scale refinement $s_2$
5	Final unprojection	Metric landmarks fixed with $s_1 s_2$
6	Procrustes ③	Pose transform matrix

The camera matrix (the virtual perspective camera) builds the bridge between the screen and 3D space, and Procrustes analysis aligns the standard face with the observed face inside that 3D space. Extracting an absolute pose from landmarks with only relative depth, on an uncalibrated camera — I find it a remarkably elegant design given the constraints.

For reference, the matrices you receive when enabling the outputFacialTransformationMatrixes option in the MediaPipe Tasks API (FaceLandmarker) are exactly the output of this pipeline.

References

• MediaPipe face_geometry module — geometry_pipeline.cc, procrustes_solver.cc
• D. Akca, Generalized Procrustes analysis and its applications in photogrammetry, 2003
• OpenGL Projection Matrix (songho.ca)