AI · Research · Product · Summer 2026
DeepTrace.
Deepfake detection research where cross-manipulation generalization is measured, not assumed — plus the inference workspace that makes the model usable.
Role
Inference & Demo Engineer
Team
4 engineers — NEU CS 5330 · with Tavish Hookoom, Zihao Li, Xijia Zeng
Stack
Python · PyTorch · CLIP ViT-B/32 · ONNX Runtime · Gradio · ffmpeg · OpenCV
Year
Summer 2026
01 / The problem
Why this needed building.
Deepfake detectors are trained on face-swap datasets, but the video people actually get fooled by now comes from whole-scene generators like Sora, Kling, and Veo. Our team set out to measure that generalization gap instead of assuming it. Measuring it produced a second problem, the one I owned: a research checkpoint is not a thing anyone can interrogate. Six .pth files and a notebook cannot answer "is this clip real?" for a person who did not write the training loop.
02 / Approach
How I broke it down.
- 01
Built the demo inference path to mirror the finalized evaluation path exactly — same H.264/CRF23/720p normalization, same 1 FPS sampling, same frozen CLIP encoder, same mean aggregation. If the demo and the paper disagreed, that would be a bug, not a caveat.
- 02
Hardened the model seam rather than trusting it. The loader asserts the ONNX contract (exactly one input and one output, named clip_features to logits), validates CLIP returns (batch, 512) and the head returns (batch, 2), and refuses to start if the external .onnx.data weights are missing — each failure raising a sentence a user can act on instead of a stack trace.
- 03
Made every request self-cleaning: a context manager re-encodes and samples inside a temporary directory that is torn down on exit, so no upload outlives its own prediction. The CLIP encoder and ONNX session load once and are cached across requests.
- 04
Designed the interface around interrogation, not a verdict. An adjustable decision threshold, per-frame fake probabilities with flagged frames outlined in the gallery, live latency and throughput counters, and a standing disclaimer that a score is model confidence rather than proof of manipulation.
- 05
Covered the numeric contract with tests — softmax conversion, mean aggregation against the threshold, the unit-interval guard, and the missing-weights failure — then ran a pixel-level QA pass of the built UI against the design reference and logged every patch until it passed.
03 / System
The pipeline, stage by stage.
Scroll to walk through each stage. Each is small on its own; what matters is the composition.
STAGE / 01
Normalize.
Every upload is re-encoded to H.264/CRF23/720p with the aspect ratio preserved by padding and audio stripped — the same normalization the training corpus went through, so compression artifacts cannot leak in as a signal.
STAGE / 02
Sample.
Frames are extracted at 1 FPS. A clip becomes a handful of stills, which is what keeps a whole-video verdict inside an interactive latency budget.
STAGE / 03
Encode.
A frozen CLIP ViT-B/32 encoder maps each frame to a 512-dimensional L2-normalized vector. Freezing the backbone was the team’s deliberate choice: it isolates what the benchmark is measuring from backbone capacity.
STAGE / 04
Classify.
An ONNX MLP head scores every frame. The session is validated against its expected interface on load, and logits become probabilities through a max-shifted softmax so long clips cannot overflow.
STAGE / 05
Aggregate.
Per-frame probabilities are averaged into one clip verdict, and both survive to the interface — the score you see is traceable to the frames that produced it.
04 / Outcomes
What it ended up being good at.
The team quantified the gap: training on AI-generated video alone scores AUC 0.4966 on face-swap — statistically indistinguishable from guessing — a 0.4741 drop from its own 0.9708 in-distribution AUC. A CLIP linear-probe baseline collapsed the same way in both directions (0.4370 and 0.5409).
Joint training recovered both manipulation types at once: AUC 1.0000 on DeepTrace-GV and 0.8210 on Celeb-DF v2, turning "detectors do not transfer" from a claim into a measured, reproducible result.
Under a leave-one-generator-out protocol the joint model held a mean AUC of 0.9745 ± 0.0261 on generators it had never seen (Sora 1.0000, Kling 0.9850, Veo 0.9386 — Veo consistently hardest). We reported it with the caveat it deserves: 24–26 clips per round is exploratory, not statistically robust.
My inference workspace turned all of that into something a non-author can use: upload a clip, get a verdict, move the threshold, and inspect which frames drove it — at 16.5 ms of model inference per clip (303 FPS, 5 frames per clip over 100 clips, T4 GPU for CLIP and CPU for the ONNX head).
DeepTrace-GV shipped as a public benchmark — 246 clips (116 AI-generated across Sora, Kling, and Veo; 130 real from Kinetics-400), 12 metadata fields per clip, stratified 70/15/15 splits, published on Kaggle with a Datasheets-for-Datasets writeup that states its own selection biases.
26 tests across the data pipeline, model, and evaluation code, four of which I wrote to pin the demo path so the numbers on screen stay the numbers in the paper.