Senior Software Engineer, AI Evals
Sentry · San Francisco, California
About this role
ABOUT SENTRY Software runs the world and the pace is faster than ever. Sentry helps developers fix errors and performance issues before users notice, so teams can spend less time firefighting and more time building. Trusted by 200,000+ organizations, Sentry is today’s application monitoring standard and our team is building its AI-native future. ABOUT THE ROLE As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for building the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems. This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale. You’ll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence. IN THIS ROLE YOU WILL - Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems - Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data - Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows - Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria - Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring YOU’LL LOVE THIS JOB IF YOU - Care deeply about correctness, rigor, and measurement in AI systems - Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics - Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team - Thrive in cross-functional environments and enjoy influencing model design through better evaluation QUALIFICATIONS - Minimum 5+ years of professional experience with a Bachelor’s degree in computer sci
Excerpt from the posting Sentry published. Read the full description on their site before applying.
Apply to this role, tailored
Queue Senior Software Engineer, AI Evals at Sentry and I will read the posting, rewrite your resume against it, draft the cover letter, and score the fit before you send anything.
Applications finish as drafts. Nothing is sent until you read it and press send. New accounts start with 500 free credits.
More roles at Sentry
See all- Today
- Today
- Last week
- 2 weeks ago
- 3 weeks ago
- 3 weeks ago
Similar roles elsewhere
See more- Today
Senior Manager, Site Reliability Engineering - Infrastructure Platform
OktaSan Francisco, California
ManagerEngineering - Today
Listed from the job board Sentry publishes. Refolk is not the employer and does not handle their hiring. Applications go to Sentry directly.