Technical Lead, Data Engineering
Distributed Spectrum · New York City, New York
- Location
- New York City, New York, United States
- Employment
- Full time
- Level
- Mid level
- Posted
- 18 November
About this role
DS creates systems that power the next generation of radio spectrum intelligence. We collect radio data from all over the world, train neural networks to decipher it, and run them on the smallest chips we can. We’re solving a new, technically hard problem where nothing from other fields works out of the box, and along the way, we’ve built our own stack from scratch, including entirely new embedding model architectures, custom GPU kernels, and much more.
Joining DS means owning major parts of a fast-growing AI research organization, joining a collaborative, talent-dense team with decades of experience in probabilistic ML, accelerated computing, embedded systems, and signal theory, and growing your career in the areas that interest you. You’ll fit in if you want to come to work for the problem itself and don’t want to choose between technical rigor, business value, and real-world impact.
We work with high ownership and trust.
The Role
DS collects radio data at a scale few organizations ever see - continuous, high-rate streams from sensors around the globe. We're hiring a Technical Lead, Data Engineering to design the platform that turns that firehose into an asset: a data lakehouse that serves researchers training models, production systems running inference, and agents retrieving context in real time.
You'll own the architecture from ingestion through storage, cataloging, vector search, and access, and you'll design it explicitly for AI/ML workloads, not just analytics.
What You'll Do
Architect and build our data lakehouse: ingestion pipelines, open table formats, partitioning and compaction strategies, cataloging, and governance
Design and operate vector database infrastructure for embedding storage, similarity search, and retrieval at scale - choosing, tuning, and evolving the right systems for our workloads
Build large-scale data management systems: lifecycle and retention, lineage, versioning of datasets for reproducible training, quality monitoring, and cost management across petabyte-class storage
Design data platform capabilities that directly support ML use cases - feature and embedding pipelines, training-set assembly, evaluation datasets, and low-latency retrieval for agents
Establish schema and data-contract standards across teams, and build tooling that lets researchers and engineers self-serve
Own reliability and performance of data pipelines in production, including observability and failure handling
Mentor engineers and lead technical design across the data domain
What We're Looking For
4 - 6+ years of software / data engineering experience, including several years designing and operating large-scale data platforms in production
Deep experience with lakehouse architectures and technologies - e.g., Apache Iceberg, Delta Lake, or Hudi; Parquet; Spark, Flink, Trino, DuckDB, or similar engines
Hands-on experience with vector databases and embedding retrieval (e.g., pgvector, Milvus, Qdrant, Weaviate, Pinecone, LanceDB, or FAISS-based systems), including indexing trade-offs and scaling
Strong experience with AWS or another major cloud provider - object storage, managed data services, compute, and cost optimization at scale
Fluency in Python and SQL; experience with orchestration tools (Airflow, Dagster, Prefect, Step Functions, etc.)
Demonstrated experience designing data architectures specifically for ML / AI workloads: training data pipelines, feature stores, dataset versioning, or retrieval systems
Strong grasp of data modeling, consistency, and the trade-offs between batch and streaming
Must currently hold or be eligible to obtain and maintain a U.S. government security clearance
Nice To Have
Experience with streaming ingestion at high volume (Kafka, Kinesis, Pulsar)
Experience with time-series, geospatial, or signal / sensor data
Familiarity with data governance and security requirements in regulated environments
Experience with Rust, Go, or C++ for performance-critical data paths
Who Thrives at Distributed Spectrum
Fast learners over specific backgrounds - We care more about how quickly you can pick up new skills than where you’ve worked before.
Intellectual honesty - The right answer matters more than being right. You challenge assumptions, test ideas, and pivot when needed.
Adaptability - We’re organized, but sometimes things change quickly. You find a way to make it work and balance short-term deliverables with long-term goals.
Ownership of outcomes - You optimize your own time, focus on what matters to deliver quickly, and cut out inefficiencies.
Not building in a vacuum - You stay connected to the rest of our teams and our customers to make sure all the pieces fit together.
What We Offer
Above-market salary, equity, and benefits package.
Early Series A Equity
Excellent health, dental, and vision coverage
401(k) match - up to 4% of your salary
Flexible PTO
Daily office lunches in NYC
ITAR Requirements
To conform to U.S. Government technology export regulations, including the International Traffic in Arms Regulations (ITAR) you must be a U.S. citizen, lawful permanent resident of the U.S., protected individual as defined by 8 U.S.C. 1324b(a)(3), or eligible to obtain the required authorizations from the U.S. Department of State. Learn more about the ITAR here.
As published by Distributed Spectrum. Applications are handled on their site.
Skills this posting mentions
About Distributed Spectrum
Founded in 2020, Distributed Spectrum is a venture-backed, defense tech company working at the cutting edge of signal processing, machine learning, and embedded systems. We build software and sensors to let anyone understand critical radio signals in any mission.
All 13 openings at Distributed SpectrumOne click, then it is written
Apply to Distributed Spectrum with a resume written for this role.
Queue Technical Lead, Data Engineering and I read the posting, rewrite your resume against it, draft the cover letter, and score the fit. Then you press send, or press one button and I fill in Distributed Spectrum’s form for you.
01Drop your resume
A PDF or a LinkedIn URL. About a minute, once.
02I rank the openings
Every weekday morning, the live catalog scored against your history. Up to 20 worth your time, not two hundred links.
03Each one is written up
Resume rewritten for the posting, a cover letter, a fit score. Press send, or let me fill in the form.
- 25 sent a week, free
- No card
- Nothing sent until you say so
More roles at Distributed Spectrum
See all- 18 November
- 6 October
Senior Software Engineer, Platform Engineering
New York City, New York
$150k - $200k/yrSeniorEngineering - 6 October
- 6 October
- 6 October
- 6 October
Similar roles elsewhere
See more- 19 November
Machine Learning Research Intern (Summer 2027)
Scale AINew York, NY +1
$120k - $150k/yrInternshipData and ML - 16 November
Sr. Machine Learning Engineer, Anonymization
MercorNew York, United States
$180k - $500k/yrSeniorData and ML - 13 November
Software Engineer (AI Data Engineering)
EvolutionIQNew York, NYRemote
$115k - $205k/yrMid levelData and ML - 12 November
Staff Machine Learning Engineer, Ads Foundational Representations
RedditNew York City, NY
$230k - $322k/yrStaffData and ML - 11 November
Put this to work
Paste your career in once. Every application after that is written for you.
Drop a resume or a LinkedIn URL. I rank the live openings against it, rewrite the resume and write a cover letter for the best of them, and fill in the employer's form when you press the button. You read, you decide what goes out.
01Drop your resume
A PDF or a LinkedIn URL. About a minute, once.
02I rank the openings
Every weekday morning, the live catalog scored against your history. Up to 20 worth your time, not two hundred links.
03Each one is written up
Resume rewritten for the posting, a cover letter, a fit score. Press send, or let me fill in the form.
- New matches ranked and written before you are up.
- Every bullet stays inside what your history supports. Nothing invented.
- Queued, submitted, interviewing, offer: one screen, not a spreadsheet.
Free, no card. Up to 25 applications sent a week, and nothing is sent until you say so.
Listed from the job board Distributed Spectrum publishes. Refolk is not the employer and does not handle their hiring. Applications go to Distributed Spectrum directly.
Hiring at Distributed Spectrum? Post this role to the Refolk talent pool for $299, once. No commission, refunded if I find nobody. Post a role for $299