Senior Staff Network Production Engineer, Operations
Crusoe · San Francisco, California
- Location
- San Francisco, California, USA
- Employment
- Full time
- Level
- Staff
- Posted
- 3 months ago
About this role
Crusoe is on a mission to accelerate the abundance of energy and intelligence. As the only vertically integrated AI infrastructure company built from the ground up, we own and operate each layer of the stack - from electrons to tokens - to power the world's most ambitious AI workloads. When you join Crusoe, you join a team that is building the future, faster.
We're in the midst of the greatest industrial revolution of our time. The demand for AI compute is boundless, and power is a bottleneck. We're solving that - with an energy-first approach that makes AI infrastructure better for the world and faster for the people innovating with AI.
We're looking for problem-solving, opportunity-finding teammates with a sense of urgency, who believe in the scale of our ambition and thrive on a path not fully paved - people who want to grow their careers alongside a team of experts across energy, manufacturing, data center construction, and cloud services.
If you want to do the most meaningful work of your career, help our customers and partners advance their AI strategies, and be part of a high-performing team that believes in each other, come build with us at Crusoe.
About this Role
Crusoe Cloud is seeking a Senior Staff Network Production Engineer to own production reliability across our global network, including edge, backbone, data center fabric, and GPU cluster interconnects. You will drive incident response, root cause analysis, and the operational excellence initiatives that keep our hyperscale AI infrastructure healthy at scale.
This is a senior production ownership role, not architecture, not pre-sales, not purely automation. You will set operational standards, define SLIs and SLOs, mentor Staff and Senior engineers, and serve as the senior escalation point during high-severity events. This is the role that keeps the network up.
What You'll Be Working On
Own Production Reliability: Serve as the senior technical owner for uptime of Crusoe's global edge, backbone, data center, and GPU cluster network, directly affecting the availability of AI workloads running on hundreds of thousands of GPUs.
Lead Incident Response: Own end-to-end response for high-severity network events, including rapid mitigation, stakeholder communication, and postmortem documentation that prevents recurrence.
Drive Root Cause Analysis: Lead RCAs for production incidents, identify systemic issues, author remediation plans, and track them to closure.
Define SLIs and SLOs: Partner with Architecture and Site Reliability to define network reliability metrics and service level objectives, backed by real-time dashboards and alerting.
Set Operational Standards: Author and maintain runbooks, escalation playbooks, and SOPs used by the broader operations team.
Improve Observability: Drive continuous improvement of Crusoe's network monitoring stack including streaming telemetry, SNMP, NetFlow, and tools such as Kentik, Grafana, Prometheus, and ThousandEyes.
Build Operational Automation: Write Python-based auto-remediation tooling that reduces toil and accelerates mean time to resolution for known failure modes.
Mentor and Multiply: Provide technical guidance to Staff and Senior engineers. Drive post-incident learning and build a culture of operational excellence across the team.
What You'll Bring to the Team
12+ years of production network engineering experience with a demonstrated focus on large-scale operations, incident response, and reliability in hyperscale or internet-scale environments.
Observability and Monitoring: Hands-on experience with streaming telemetry, SNMP, NetFlow, sFlow, and tools such as Kentik, Grafana, Prometheus, ThousandEyes, and Arbor.
GPU Cluster and RDMA Networking: Hands-on experience operating RDMA/RoCE (v1 and v2) lossless fabrics for GPU and HPC workloads, including PFC, ECN, and DCQCN tuning. Required at this level.
Demonstrated Technical Leadership: Proven track record owning production reliability at scale, leading RCAs that drove systemic change, and setting operational standards the broader org executes against.
Hyperscale Operational Depth: Comfort operating 10K+ device fleets across multi-region environments with 24/7 on-call responsibility. You have been the senior escalation point during critical network events.
Protocol Fluency: Expert hands-on knowledge of BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, and TCP/IP across production DC fabric environments at scale.
Hardware Platform Depth: Expert knowledge of Arista (EOS), Juniper (Junos), and NVIDIA/Mellanox platforms in leaf-spine CLOS architectures across multi-vendor environments.
Operational Automation: Proficiency in Python for auto-remediation scripts, diagnostic tooling, and operational workflows that reduce toil and accelerate incident resolution.
SLI and SLO Ownership: Experience defining and owning network reliability metrics and service level objectives in partnership with engineering and product leadership.
Education: Bachelor's degree in Computer Science, Electrical Engineering, or a related field, or equivalent practical experience in hyperscale or internet-scale environments.
Benefits:
Competitive compensation
Restricted Stock Units
Paid time off & paid holidays
Comprehensive health, dental & vision insurance
Employer contributions to HSA account
Paid parental leave
Paid life insurance, short-term and long-term disability
Professional development & tuition reimbursement
Mental health & wellness support
Commuter benefits (parking & transit)
Cell phone stipend
401(k) Retirement plan with company match up to 4% of salary
Volunteer time off
Compensation:
Compensation will be paid in the range of $225,000 - $275,000 + Bonus. Restricted Stock Units are included in all offers. Compensation to be determined by the applicant’s education, experience, knowledge, skills, and abilities, as well as internal equity and alignment with market data.
Crusoe is an Equal Opportunity Employer. Employment decisions are made without regard to race, color, religion, disability, genetic information, pregnancy, citizenship, marital status, sex/gender, sexual preference/ orientation, gender identity, age, veteran status, national origin, or any other status protected by law or regulation.
As published by Crusoe. Applications are handled on their site.
Skills this posting mentions
About Crusoe
Crusoe is building the World’s Favorite AI-first Cloud infrastructure company. We’re pioneering vertically integrated, purpose-built AI infrastructure solutions trusted by Fortune 500 companies to power their most advanced AI applications. Crusoe is redefining AI cloud infrastructure, with a mission to align the future of computing with the future of the climate. Our AI platform is recognized as the “gold standard” for reliability and performance. Our data centers are optimized for AI workloads and are powered by clean, renewable energy.
All 371 openings at CrusoeOne click, then it is written
Apply to Crusoe with a resume written for this role.
Queue Senior Staff Network Production Engineer, Operations and I read the posting, rewrite your resume against it, draft the cover letter, and score the fit. Then you press send, or press one button and I fill in Crusoe’s form for you.
01Drop your resume
A PDF or a LinkedIn URL. About a minute, once.
02I rank the openings
Every weekday morning, the live catalog scored against your history. Up to 20 worth your time, not two hundred links.
03Each one is written up
Resume rewritten for the posting, a cover letter, a fit score. Press send, or let me fill in the form.
- 25 sent a week, free
- No card
- Nothing sent until you say so
More roles at Crusoe
See all- 5 weeks ago
- 5 weeks ago
- 5 weeks ago
- 5 weeks ago
- 5 weeks ago
- 5 weeks ago
Similar roles elsewhere
See more- Today
Staff Product Marketing Manager, Developer and Publishers
DiscordSan Francisco Bay Area
$204k - $229k/yrStaffEngineering - Today
- Today
Senior Engineering Manager - Custody
MercurySan Francisco, CARemote
$239k - $299k/yrManagerEngineering - Today
Senior Staff Software Engineer, Developer Productivity
AsanaSan Francisco
$324k - $337k/yrStaffEngineering
Put this to work
Paste your career in once. Every application after that is written for you.
Drop a resume or a LinkedIn URL. I rank the live openings against it, rewrite the resume and write a cover letter for the best of them, and fill in the employer's form when you press the button. You read, you decide what goes out.
01Drop your resume
A PDF or a LinkedIn URL. About a minute, once.
02I rank the openings
Every weekday morning, the live catalog scored against your history. Up to 20 worth your time, not two hundred links.
03Each one is written up
Resume rewritten for the posting, a cover letter, a fit score. Press send, or let me fill in the form.
- New matches ranked and written before you are up.
- Every bullet stays inside what your history supports. Nothing invented.
- Queued, submitted, interviewing, offer: one screen, not a spreadsheet.
500 free credits on sign-up. No card. Nothing is sent until you say so.
Listed from the job board Crusoe publishes. Refolk is not the employer and does not handle their hiring. Applications go to Crusoe directly.