RefolkCandidates
JobsBuild my resume

Member of Technical Staff (Data Scientist, Evals)

Perplexity · San Francisco

$200k - $300k/yrRemoteStaffData and MLAI Research & SystemsAI Research & SystemsPosted 5 months ago
Apply with a tailored resumeView the original posting

About this role

Perplexity serves tens of millions of users daily with reliable, high-quality answers grounded in an LLM-first search engine and our specialized data sources. We aim to use the latest models as they are released, but the intelligence frontier is a jagged one, and popular benchmarks do not effectively cover our use cases. In this role, you will build specialized evals to improve answer quality across Perplexity, covering search-based LLM answers and other scenarios popular with our users. RESPONSIBILITIES - Architect and maintain automated evaluation pipelines to assess answer quality across Perplexity's products, ensuring high standards for accuracy and helpfulness - Design evaluation sets and methods specifically to measure the impact of tool calls (particularly web search retrieval) on the final answer's quality - Develop VLM-based solutions to programmatically evaluate how final answers render visually across different platforms and devices - Continuously review public benchmarks and academic evaluations for their applicability to the Perplexity product, adapting and incorporating them into our regular performance measurements - Operate within a small, high-impact team where your evaluation metrics directly shape product changes, collaborating closely with technical leadership to measure and improve Answer Quality QUALIFICATIONS - PhD or MS in a technical field or equivalent experience - 4+ years of experience in data science or machine learning - Strong proficiency in Python and SQL (expected to write production-grade code) - Experience building within a modern cloud data stack, specifically AWS and Databricks - Comfortable with agentic coding workflows and using AI-assisted development tools to iterate faster PREFERRED QUALIFICATIONS - 1+ years of experience working with LLMs at scale, specifically with LLM-as-a-judge setups - Prior experience working on customer-facing web products or consumer apps, with real user traffic at scale - A strong research backgroun

Excerpt from the posting Perplexity published. Read the full description on their site before applying.

Skills this posting mentions

SQLMachine LearningAmazon Web Services

About Perplexity AI

Perplexity AI builds AI native search tools. Perplexity Ask, a new search interface that uses OpenAI GPT 3.5 and Microsoft Bing to directly answer any question you ask.

All 87 openings at Perplexity AI

Apply to this role, tailored

Queue Member of Technical Staff (Data Scientist, Evals) at Perplexity and I will read the posting, rewrite your resume against it, draft the cover letter, and score the fit before you send anything.

Applications finish as drafts. Nothing is sent until you read it and press send. New accounts start with 500 free credits.

Tailor my applicationHow it works

More roles at Perplexity

See all

Similar roles elsewhere

See more

Listed from the job board Perplexity publishes. Refolk is not the employer and does not handle their hiring. Applications go to Perplexity directly.