Refolk
Open nowScience and research

Applied Research Scientist, Safety and Alignment

Fleet AI · San Francisco Bay Area

Location
San Francisco Bay Area
Employment
Full time
Level
Mid level
Posted
28 October

About this role

About Fleet

Fleet studies how environments produce safe intelligence. We believe safety and economic value increasingly reinforce each other: as agents take on more consequential work, their value depends on whether they can be trusted with that responsibility.

As reinforcement learning takes a larger role in post-training, the design of tasks, environments, and feedback becomes increasingly consequential for downstream alignment. RL can reinforce sound judgment, but it can also amplify unintended behaviors. We believe studying and scaling the data and training signals that produce more aligned agents is one of the most impactful ways to contribute to AI safety.

Backed by Sequoia Capital, Menlo Ventures, BCV, and SV Angel.

About the Role

We’re looking for an Applied Research Scientist to work on robust reward design, implement comprehensive telemetry, and design production systems to help ensure Fleet's RL tasks make capable agents more aligned. You’ll work with data and model trajectories to help develop oversight systems to detect misaligned behaviors. You’ll also work directly with safety researchers at frontier labs to help co-design new RL environments and tasks for downstream alignment training and evaluation.

What You’ll Work On

  • Reward Design and Telemetry at Production Scale. Study how task and verifier design affect model-alignment incentives, then improve both.
    • Off-target behaviors from task assumptions and incentives. Investigate how unstated behavioral constraints, unsolvable tasks, and rewards that ignore how success is achieved affect how models behave during evaluation, across many different domains.
    • Telemetry and Verifier Generation Pipelines. Use the insights learned from the above to build 1) the telemetry needed for customers to have complete observability over undesirable model behaviors and 2) the scalable pipelines needed for producing verifiers that translate recorded measurements into qualitative behavioral categories.
    • Production Oversight Systems. Build out rigorous trace-based analysis systems that measure agent misbehavior across all internal Fleet evaluations.
  • Safety evaluations and benchmark analysis. Develop rigorous safety measurements and assess the risks in existing Fleet tasks and external benchmarks.
    • Safety benchmarks. Build held-out and adversarial evaluations to determine the safety profile of existing tasks. Audit Fleet tasks, data, and graders for contamination, poisoning, ambiguity, and shortcuts.
    • External benchmark analysis. Examine widely used benchmarks, datasets, and task designs for incentives that could reinforce unsafe behavior.
    • Specialization in high-impact domains. Get involved in domain-specific safety research with frontier labs, including cybersecurity. Work with experts to identify consequential failure modes and design curricula, tasks, and evaluations aimed at preventing incidents like the Hugging Face breach.
    • What We’re Looking For

      • A willingness to work directly with the data: inspect examples and trajectories, question labels and graders, investigate anomalies, and trace a surprising result to its cause.
      • Strong practical engineering skills - you should be able to build the systems needed to test and scale your ideas.
      • A record of strong empirical experience in alignment, model behavior, model evals, or a closely related field.

      Prior research experience via safety fellowships (e.g. MATS, Astra), a safety organization (e.g. METR, Redwood, Epoch AI), or academia is helpful but not necessary.

      Location

      San Francisco (on-site).

      Compensation

      Salary range: $200,000 - $600,000 USD annually, plus meaningful equity.

      As published by Fleet AI. Applications are handled on their site.

      Skills this posting mentions

      Computer SecurityArtificial IntelligenceResearch

      About Fleet AI

      Gym Environments for Frontier Agents

      All 12 openings at Fleet AI

      One click, then it is written

      Apply to Fleet AI with a resume written for this role.

      Queue Applied Research Scientist, Safety and Alignment and I read the posting, rewrite your resume against it, draft the cover letter, and score the fit. Then you press send, or press one button and I fill in Fleet AI’s form for you.

      1. 01Drop your resume

        A PDF or a LinkedIn URL. About a minute, once.

      2. 02I rank the openings

        Every weekday morning, the live catalog scored against your history. Up to 20 worth your time, not two hundred links.

      3. 03Each one is written up

        Resume rewritten for the posting, a cover letter, a fit score. Press send, or let me fill in the form.

      • 25 sent a week, free
      • No card
      • Nothing sent until you say so

      More roles at Fleet AI

      See all

      Similar roles elsewhere

      See more

      Put this to work

      Paste your career in once. Every application after that is written for you.

      Drop a resume or a LinkedIn URL. I rank the live openings against it, rewrite the resume and write a cover letter for the best of them, and fill in the employer's form when you press the button. You read, you decide what goes out.

      1. 01Drop your resume

        A PDF or a LinkedIn URL. About a minute, once.

      2. 02I rank the openings

        Every weekday morning, the live catalog scored against your history. Up to 20 worth your time, not two hundred links.

      3. 03Each one is written up

        Resume rewritten for the posting, a cover letter, a fit score. Press send, or let me fill in the form.

      • New matches ranked and written before you are up.
      • Every bullet stays inside what your history supports. Nothing invented.
      • Queued, submitted, interviewing, offer: one screen, not a spreadsheet.

      Free, no card. Up to 25 applications sent a week, and nothing is sent until you say so.

      Listed from the job board Fleet AI publishes. Refolk is not the employer and does not handle their hiring. Applications go to Fleet AI directly.

      Hiring at Fleet AI? Post this role to the Refolk talent pool for $299, once. No commission, refunded if I find nobody. Post a role for $299