Face Recognition: Turning a Classifier Into an Embedding That Can Learn New Faces
Phones that unlock just by looking at you, office systems that recognize whoever just walked through the door: the trick behind both is rarely "just train a classifier." This post walks through why, and through building my own face recognition system from scratch, all the way to a callable API.
Table of Contents
What Face Recognition Actually Does
Three Stages: Register, Verify, Recognize
Guess First
From Classifier to Embedding
The Build: EfficientNet-B0, MTCNN, FastAPI, Supabase pgvector
Plot Twist: The Bug That Made the Embeddings "Blind"
Verified Results
Limitations
Quiz
Wrapping Up
What Face Recognition Actually Does
Face recognition is a biometric technique for identifying or verifying someone based on facial features. The system analyzes an image or video, extracts distinguishing features like eye spacing, nose shape, and face contour, then compares them against previously stored data.
Applications range from security, like access control and surveillance, to lighter everyday use, like unlocking a phone or auto-tagging photos. Deep learning is what keeps pushing the accuracy of all of this forward year over year.
Three Stages: Register, Verify, Recognize
A face recognition system typically runs through three stages, each serving a different purpose.
| Stage | Flow | Purpose |
|---|---|---|
| Face Register | photo → face detection → cropping → embedder → stored in a vector DB | Enroll a new face |
| Face Verification | photo + claimed identity → same flow → similarity against one stored vector (1:1) | "Is this really who they claim to be?" |
| Face Recognition | photo → same flow → similarity against every stored vector (1:N) | "Whose face is this?" |
Verification is the lighter of the two since it only checks against one claimed identity, which is why it fits device unlocking well. Recognition is heavier because it has to search across every enrolled identity, which is why it shows up in surveillance or mass-identification systems.
Guess First
Before reading on, take a guess: if a classifier was only ever trained to recognize 16 people, and a 17th, completely unseen face shows up, can the system still enroll them as a new identity without retraining anything?
The answer is in the next section.
From Classifier to Embedding
Yes, as long as the classifier is used the right way afterward. The trick is in what happens once training is done:
Train a regular CNN classifier first, where each class represents one person's identity.
Once it's trained, drop the classification head, the final layer that outputs "this is person A, this is person B."
What's left, the convolutional backbone, becomes a feature extractor. Its output is an embedding: a fixed-length numeric representation of the face.
Two embeddings get compared with cosine similarity, not through the classifier anymore.
Because the comparison is based on vector similarity rather than the classifier's fixed set of output labels, the system can enroll anyone at any time, including people who were never part of the original training data. That's what makes the embedding approach so much more practical than a plain classifier for real-world use, where the list of people to recognize keeps growing.
The Build: EfficientNet-B0, MTCNN, FastAPI, Supabase pgvector
I turned the concept above into a callable API rather than leaving it as a notebook demo.
Backbone: EfficientNet-B0, fine-tuned as a 16-identity classifier, then stripped of its classification head and reused as a feature extractor producing a 1280-dimensional embedding.
Face detection & alignment: MTCNN from facenet-pytorch, detecting the face and straightening its orientation before it reaches the embedder.
Vector database: Postgres with the pgvector extension, hosted on Supabase so there's no database server to stand up manually.
Serving: FastAPI with three endpoints,
/face_register/,/face_verification/, and/face_recognition/, tunneled out of Google Colab via ngrok so it's reachable from outside.
Plot Twist: The Bug That Made the Embeddings "Blind"
Right after wiring the training side to the serving side, I went back and double-checked how the backbone was actually being loaded. Turned out there was a bug that would have been genuinely painful to miss.
During training, the backbone gets saved by re-wrapping it into a new nn.Sequential. On the serving side, the model is defined through a custom class with a different attribute structure. The result: the parameter names inside the checkpoint never matched the parameter names the model expected at load time. Since the original code used strict=False, PyTorch never complained. Every mismatched parameter was silently skipped, and the model kept running on random weights instead of the trained ones, with zero errors raised anywhere.
Left unnoticed, the entire verification and recognition system would have kept "working" in the technical sense: no crashes, similarity scores returned, a clean 200 OK on every request. The numbers just wouldn't have meant anything about actual facial similarity.
The fix: load the checkpoint directly into the correct submodule instead of the whole model, and switch strict=False to strict=True. That way, any future mismatch throws a loud, obvious error instead of failing silently again.
Verified Results
Backbone training (EfficientNet-B0, 16-identity classifier):
| Metric | Value |
|---|---|
| Classes | 16 identities |
| Test accuracy | 95% (131 held-out images) |
| Embedding dimensionality | 1280 |
End-to-end API test, using photos that never touched the training set:
| Test | Similarity | Result |
|---|---|---|
| Same person, different photo (verification) | 0.8664 | Verified |
| Same person, same photo (recognition) | 1.0000 | Recognized |
| Different, unregistered person (recognition) | 0.0638 | Not recognized |
The gap between 0.0638 and 0.8664 is the part worth paying attention to. A spread that wide shows the embedding space is actually separating identities, rather than producing similar-looking numbers regardless of who's in the photo, which is exactly the failure mode the bug above would have caused if it had gone unfixed.
Limitations
Training and serving don't preprocess images identically. Training resizes raw photos directly, while serving runs them through MTCNN detection and alignment first. With a reasonably clean dataset where faces are already roughly centered, the effect isn't very visible, but it's worth knowing about.
The 95% figure comes from a single, seeded train/test split for reproducibility, not from cross-validation. It's a reasonable signal, not a rigorous benchmark.
This serving setup is meant for local experimentation, not production hosting. The ngrok tunnel dies the moment the Colab runtime stops.
There's no rate limiting or authentication on the endpoints yet.
Quiz
1. Why is the embedding approach more flexible than a plain classifier for face recognition? Because a new identity can be enrolled just by storing its embedding in the database, without retraining the classifier every time someone new shows up.
2. What's the core difference between Face Verification and Face Recognition? Verification is 1:1, checking similarity against one claimed identity. Recognition is 1:N, searching for the best match across every enrolled identity.
3. Why can strict=False in load_state_dict() be dangerous? Because PyTorch will silently ignore any checkpoint parameters whose names don't match the model, without raising an error. The model can look like it's running fine while the weights it actually loaded are wrong, or even entirely missing.
Wrapping Up
The embedding approach turns face recognition from "guess one of a fixed set of classes" into a system that keeps accepting new identities without ever needing to retrain. That flexibility comes with its own trap, though: the system can keep technically running even when the embeddings it produces mean nothing, unless the parameter names are actually checked all the way down.
Full source code, including both notebooks and the results documentation, is on GitHub: face-recognition-pgvector.
