Top Data engineering repositories on GitHub

Apache Superset is a Data Visualization and Data Exploration Platform

• superset
• apache
• apache-superset
• data-visualization
• data-viz
• analytics

Learn how to develop, deploy and iterate on production-grade ML applications.

• machine-learning
• deep-learning
• pytorch
• natural-language-processing
• data-science
• python

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python
• scheduler
• workflow

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. The next cohort starts in January 2026. Join the course here 👇🏼

• data-engineering
• kafka
• spark
• dbt
• docker
• kestra

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

• applied-machine-learning
• production
• applied-data-science
• machine-learning
• data-science
• reinforcement-learning

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

• orchestration
• data-orchestration
• high-availability
• infrastructure-as-code
• automation
• devops

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science
• workflow-engine
• prefect

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering
• java
• python

Turns Data and AI algorithms into production-ready web applications in no time.

• automation
• data-engineering
• data-ops
• data-visualization
• datascience
• developer-tools

Workflow Engine for Kubernetes

• workflow
• kubernetes
• argo
• dag
• knative
• airflow

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science
• workflow-automation
• python

The Data Engineering Cookbook

• data-engineer
• data-engineering
• big-data
• best-practices
• cookbook

Roadmap to becoming a data engineer in 2021

• data-engineer-roadmap
• data-engineering
• cloud
• roadmap

Always know what to expect from your data.

• pipeline-tests
• dataquality
• datacleaning
• datacleaner
• data-science
• data-profiling

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl
• indexing
• python

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

• xonsh
• devops
• iterm2
• data-engineering
• security-automation
• raspberry-pi

Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.

• database
• stream-processing
• rust
• postgresql
• kafka
• materialized-view

🧙 Build, run, and manage data pipelines for integrating and transforming data.

• machine-learning
• artificial-intelligence
• data
• data-engineering
• data-science
• python

Fancy stream processing made operationally mundane

• message-queue
• stream-processing
• streaming-data
• message-bus
• logs
• stream-processor

Open Source Feature Flags, Experimentation, and Product Analytics

• abtesting
• statistics
• abtest
• experimentation
• split-testing
• snowflake

The Open Source Feature Store for AI/ML

• machine-learning
• features
• ml
• big-data
• feature-store
• python

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

• ai-agents
• data-engineering
• document-ai
• generative-ai
• idp
• json-extraction

Business intelligence as code: build fast, interactive data visualizations in SQL and markdown

• analytics
• sql
• business-intelligence
• data-visualization
• dbt
• duckdb

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

• aws
• gcp
• azure
• sql
• data-integration
• elt

High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale

• machine-learning
• python
• data-engineering
• distributed-computing
• rust
• big-data