Trust and Safety - AI Metric Quality & Expert Review

• Architected model serving platform replacing manual expert review pipelines with ensemble LLMs; designed calibrated threshold selection across rules engines, neural networks, and LLMs to optimize precision-recall tradeoffs under cost constraints

• Drove platform reliability from 90% to 99% job success rate through systematic failure triage, alerting infrastructure, and oncall runbook creation

• Eliminated $1M+/year in operational costs through deep root-cause analysis of processing queue failures, identifying and removing classes of unactionable work items

Trust and Safety Experimentation

• Owned statistical system serving 350+ ML engineers and data scientists across 6,000+ experiments and 8 business verticals; assumed ownership from Staff & Principal engineers

• Led experimentation for a major product safety initiative, unblocking experiments that yielded 10% reduction in a topline safety metric

• Optimized query engine achieving 5× latency improvement for 400+ engineers in critical experiment review cycles

• Grew platform 82% in active users and 6× in experiment volume through new metric capabilities

Trust & Safety Measurement Platform

• Designed probabilistic uncertainty framework: bootstrapped labeling errors across experiments to construct empirical error distributions and derive confidence intervals. Analysis proved zero statistical power in a 5-year measurement initiative, leading to project cancellation and reallocation of 13 engineers (~50 eng-weeks/year saved)

• Owned upsampling system processing 2B+ samples/day, formulated as convex optimization over quality constraints and ML prediction scores to minimize estimation variance under cost targets

• Built real-time proxy metrics integrated into ranking systems: 20% lift in a P1 KPI, 650% increase in operational throughput, enabled $14M efficiency estimation across ranking experiments

• Built offline ML platform reducing model onboarding from 2 weeks to 2 hours and experiment iteration from 30 min to 2 min

• Optimized ML serving via caching/batching redesign: −20% compute, −30% p95 latency ($250k/yr savings)

Led a 10-engineer Better Engineering workstream: $4M/yr infra savings, test coverage 30% → 89%

Trust & Safety Measurement Platform

• Redesigned high-throughput API serving 100M+ QPS with caching architecture, delivering $514k/yr compute savings while maintaining latency SLAs

• Built config-driven metric system enabling zero-code formula updates across real-time and batch pipelines; patterns adopted platform-wide

• Discovered and fixed statistical reporting bug causing 30% underreporting in similarity metrics, and a separate bug causing 300% CPU spike from upstream system changes