See yourself,
made up.

The case study

What "See yourself, made up" is actually proving, in the 90 seconds you have.

This is a live AI makeup try-on prototype, built as an AI product management case study - the demo above is real, on-device, and yours to try. Below is the part a demo alone can't show: why this problem, why this approach, what was measured, what's guarded against, and what's honestly still missing.

• The problem: shade uncertainty is the purchase blocker in online beauty - try-on removes the guess.
• What's measured: a fairness bug in the rendering pipeline, found and fixed, with a number attached (see Evaluation).
• What's guarded: thirteen cataloged failure modes, each with a severity and a specific mitigation, not just a working demo (see Failure modes and Guardrails).

01 - The problem

You can't tell how a shade will look on you until it's already on your face.

Online beauty has one purchase blocker that recurs across every retailer: a swatch on a screen doesn't tell you how a shade reads against your own skin.

The two existing fixes are both weak. Self-reported skin-tone quizzes ask people to categorize something they usually can't judge accurately. Static "shade finder" charts assume everyone's screen and lighting render color the same way. Both push the uncertainty back onto the shopper instead of resolving it - and the retailer eats the resulting returns.

This prototype stands in for a retailer's product page or an in-store kiosk: the moment someone is deciding whether to buy, let them see the shade on their actual face, live, before they commit.

02 - Use-case selection

Why try-on, why a rule-based recommender, why on-device - and what would change each call.

Why try-on, not a quiz

A quiz asks someone to self-report something - their undertone, their depth - that most people can't judge accurately about themselves. Try-on replaces a guess with a direct look, using the camera instead of a dropdown.

Why a rule-based recommender on top of an ML tracker, not a trained or generative model

Face tracking is genuinely ML (MediaPipe's Face Landmarker, a 468-point mesh) because that's a hard geometry problem worth a trained model. Shade recommendation, on top of that mesh, is a hand-tuned heuristic - CIELAB hue angle for undertone, luma for depth - deliberately, not by default:

• It's traceable: every recommendation reduces to two numbers a person can inspect, not a black box.
• It needs no labelled training data ("this shade suits this skin") that doesn't exist yet at this project's stage.
• It runs in milliseconds, entirely on-device - which is also what keeps the privacy claim structural rather than a policy promise.

What would change this decision: real recommender-agreement data. If human raters consistently reject the heuristic's picks at a rate a trained model would clearly close, that gap - measured, not assumed - is the evidence that justifies training one.

That test is specified but not yet run (see Evaluation).

Why on-device, not a server-side model

On-device removes per-session inference cost, removes the upload round-trip that would otherwise sit in the critical path of every frame, and means the privacy line in the UI ("on-device only") is true by construction, not a claim that has to be trusted.

03 - How "Style me" reads you

Five sample points, one median, and a hue angle - not a trained model, a transparent one.

Every "Style me" tap takes one fresh reading of your actual skin, live from the camera, and scores the catalog against it. Here's exactly what that reading is - no more, no less.

The reading

The moment you tap "Style me," it samples five fixed points on your face - both cheekbones, both lower cheeks, mid-forehead - from the raw camera frame, never the painted-on makeup.

It takes the median color across those five points, not the average. That's deliberate: one bad patch - a strand of hair, a glare spot, a shadow - can't drag the reading off if the other four agree.

What it's reading for

• Depth - light, medium, or deep - from how bright the sampled skin is.
• Undertone - warm, cool, or neutral - from the hue angle of that color in CIELAB space. The code's own comment is exact about what this is and isn't: "hand-tuned heuristics, depend on camera white balance, not a calibrated model."
• Confidence - if the five sample points disagree with each other past a set threshold, that's flagged as uneven lighting, and every recommendation downstream gets hedged as a "rough guess" instead of stated flatly.

Two guardrails sit in front of all of this: a reading that's too dark, or too washed out by glare, is rejected outright with a toast explaining why - rather than turning bad input into a confident-sounding number.

From a reading to a recommendation

Every shade in the catalog gets scored against your reading: points for a depth match, points for a tone match, partial credit when either side is neutral. Style me applies that scoring to all ten curated looks at once - summing the score across each look's three products - and applies the best-ranked one.

Tap "Another look" and it doesn't re-read your face - it steps to the next look in that same ranked list. One reading, several ranked options to browse, not a fresh guess each time.

04 - Architecture

Nothing but the tracking model itself crosses the network.

The only things that ever cross the network: the MediaPipe model files (loaded from public CDNs) and the page's own static assets. Camera frames, uploaded photos, and the rendered image never leave the browser. Single static page, no build step, no backend.

05 - Evaluation

The fairness bug, measured: a flat 24-32 ΔE band instead of a collapse to 11.

Anyone can say a feature works across skin tones. This is what it looks like to actually check - a defined method, and a result that could have come back bad.

The rendering pipeline blends makeup color onto skin in three passes: hue transfer, pigment coverage, depth. The first version used one multiply pass, which the code's own comment already called out as rejected: "multiply alone only darkens, which mutes bright shades and vanishes on deeper skin."

That was caught by eye - a real but unverifiable finding. This eval puts a number on it.

Method, in plain terms:

• ΔE (Delta E) measures how different two colors look to the eye. Roughly: 1 is barely noticeable, 20+ is obviously different.
• The Monk Skin Tone scale is the 10-point reference Google and others use for fairness testing - MST-1 lightest to MST-10 darkest.

This eval replicates the production color math exactly - the same `color` → `source-over` → `multiply` formulas and alphas `index.html` uses - against all 10 tones and all 16 catalog shades, then measures the ΔE between bare skin and post-makeup color. Synthetically, not on real faces or cameras (see Scope below).

It isn't uniformly strong everywhere. The five weakest (tone, shade) pairs are all neutral eyeshadow on mid-to-deep tones:

That's explainable, not random. Rose Taupe and Soft Grey are low-saturation neutrals by design, and on deeper skin their hue sits close to the skin's own - a "your-skin-but-slightly-different" shade will always read lower-contrast than a saturated one, on any tone.

Scope: this measures the rendering pipeline's color math on synthetic swatches - not tracking accuracy or recommender agreement, which are a separate, real-faces question. Naming that boundary precisely is itself part of the point: an evaluation is only credible if you can say exactly what it does and doesn't cover. Full method and data →

06 - Failure modes

What's been found so far, and how it's handled.

13 entries identified through real-device testing and the fairness eval above.

07 - Guardrails & responsible AI

Specific controls, not "safety filters."

08 - Latency & unit economics

What's actually been measured, and what hasn't.

Cost: on-device inference means no per-session model cost and static hosting only. A server-side alternative would need real vendor quotes to compare honestly - that table isn't filled in here, on purpose, rather than publish a plausible-sounding placeholder with no real numbers behind it.

09 - Metric tree

A north star, guardrails, and quality metrics - wired to real events, not a wishlist.

21 track() events already fire in production - including occasion_select, deeplink_open, copy_look_link, before_after_created/saved/shared, add_to_bag, checkout_demo - each one feeding directly into the tree above.