Refolk
Open nowEngineeringGeneral Research / Engineering

Production Engineer

Recursive · London, United Kingdom

Location
London, United Kingdom
Employment
Full time
Level
Mid level
Posted
20 October

About this role

About the role

Recursive is building AI that improves AI, starting with automated AI research. We’re hiring a Production Engineer to build and operate the systems our researchers and engineers rely on: cloud infrastructure, GPU clusters, deployment tooling, and secure access.

You’ll work across the stack, from Linux hosts and Slurm jobs to Kubernetes services and infrastructure automation. This is a hands-on role with substantial operational ownership. You’ll help teams get unstuck, diagnose failures, and turn recurring problems into reliable tools and processes.

Where you can make an impact

  • Reliable compute: Keep GPU and CPU environments healthy across cloud providers so research workloads can run consistently.

  • Repeatable infrastructure: Make provisioning, configuration, deployment, and access predictable, reviewable, and easy to operate.

  • Researcher productivity: Give teams clear documentation, useful diagnostics, and practical support as their workloads change.

What you’ll do

  • Provision and manage infrastructure across AWS, GCP, and neo cloud providers using infrastructure as code and configuration management.

  • Operate Slurm clusters and container platforms; diagnose GPU, network, storage, scheduling, and job startup and cleanup failures.

  • Build Python and Bash automation and secure CI/CD pipelines, making changes testable, repeatable, and easy to roll back.

  • Manage IAM, secrets, and zero-trust access across providers using tools like Tailscale, keeping permissions deliberate and systems secure.

  • Improve logs, metrics, alerts, and recovery procedures; work with providers to resolve faulty nodes and verify systems before returning them to service.

  • Partner with researchers and engineers to support agentic workloads, understand bottlenecks, and prioritize fixes that remove recurring friction.

What we’re looking for

  • Clear communication, curiosity, and ownership. You can guide researchers through unclear requirements, manage changing priorities, and revise your approach as new evidence appears.

  • Deep Linux systems experience and strong practical troubleshooting across hosts, networking, storage, and GPU environments.

  • You can read unfamiliar automation, reason about failures, and verify changes before running them with elevated privileges.

  • Hands-on infrastructure as code and configuration management experience with tools such as Terraform, Pulumi, Puppet, or Ansible.

  • Practical Kubernetes and container deployment experience, plus a working understanding of CI/CD, IAM policies, and secure access.

  • Comfort using AI coding agents for diagnosis and automation, with the judgment to review generated code, test assumptions, and understand its operational risks.

Especially relevant experience

  • Slurm ecosystem tools such as Slinky, AWS ParallelCluster, Pyxis, and prolog/epilog diagnostics.

  • Container delivery with EKS, ECS, Helm, Kustomize, Argo CD, or GitHub Actions.

  • Observability and hardware management with Prometheus, Grafana, Datadog, or Redfish.

  • Experience operating GPU clusters, including NVIDIA or AMD diagnostics and collective communication tooling such as NCCL or RCCL.

  • Rust programming, SGLang gateways, or Modal applications.

  • Cloud security posture tools such as Wiz, security data lakes, or hands-on support for security and compliance audits.

What success looks like

Researchers can get the compute and access they need, understand failures, and keep work moving. Infrastructure changes are reproducible and reviewed, recurring incidents become less frequent, and recovery is documented and dependable. You leave the platform easier for colleagues to operate and extend.

As published by Recursive. Applications are handled on their site.

Skills this posting mentions

PulumiAnsibleInfrastructure

About Recursive

Recursive self-improving superintelligence to automate knowledge discovery.

All 9 openings at Recursive

One click, then it is written

Apply to Recursive with a resume written for this role.

Queue Production Engineer and I read the posting, rewrite your resume against it, draft the cover letter, and score the fit. Then you press send, or press one button and I fill in Recursive’s form for you.

  1. 01Drop your resume

    A PDF or a LinkedIn URL. About a minute, once.

  2. 02I rank the openings

    Every weekday morning, the live catalog scored against your history. Up to 20 worth your time, not two hundred links.

  3. 03Each one is written up

    Resume rewritten for the posting, a cover letter, a fit score. Press send, or let me fill in the form.

  • 25 sent a week, free
  • No card
  • Nothing sent until you say so

More roles at Recursive

See all

Similar roles elsewhere

See more

Put this to work

Paste your career in once. Every application after that is written for you.

Drop a resume or a LinkedIn URL. I rank the live openings against it, rewrite the resume and write a cover letter for the best of them, and fill in the employer's form when you press the button. You read, you decide what goes out.

  1. 01Drop your resume

    A PDF or a LinkedIn URL. About a minute, once.

  2. 02I rank the openings

    Every weekday morning, the live catalog scored against your history. Up to 20 worth your time, not two hundred links.

  3. 03Each one is written up

    Resume rewritten for the posting, a cover letter, a fit score. Press send, or let me fill in the form.

  • New matches ranked and written before you are up.
  • Every bullet stays inside what your history supports. Nothing invented.
  • Queued, submitted, interviewing, offer: one screen, not a spreadsheet.

Free, no card. Up to 25 applications sent a week, and nothing is sent until you say so.

Listed from the job board Recursive publishes. Refolk is not the employer and does not handle their hiring. Applications go to Recursive directly.

Hiring at Recursive? Post this role to the Refolk talent pool for $299, once. No commission, refunded if I find nobody. Post a role for $299