Top Python Data engineering repositories on GitHub

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

• airflow
• apache
• apache-airflow
• python
• scheduler
• workflow

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

• python
• workflow
• data-engineering
• data-science
• workflow-engine
• prefect

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

• data
• pipeline
• data-analysis
• data-engineering
• java
• python

Turns Data and AI algorithms into production-ready web applications in no time.

• automation
• data-engineering
• data-ops
• data-visualization
• datascience
• developer-tools

An orchestration platform for the development, production, and observation of data assets.

• data-pipelines
• dagster
• workflow
• data-science
• workflow-automation
• python

The Data Engineering Cookbook

• data-engineer
• data-engineering
• big-data
• best-practices
• cookbook

Always know what to expect from your data.

• pipeline-tests
• dataquality
• datacleaning
• datacleaner
• data-science
• data-profiling

🐚 Python-powered shell. Full-featured, cross-platform and AI-friendly.

• xonsh
• devops
• iterm2
• data-engineering
• security-automation
• raspberry-pi

Incremental engine for long horizon agents 🌟 Star if you like it!

• ai
• change-data-capture
• data-indexing
• etl
• indexing
• python

🧙 Build, run, and manage data pipelines for integrating and transforming data.

• machine-learning
• artificial-intelligence
• data
• data-engineering
• data-science
• python

The Open Source Feature Store for AI/ML

• machine-learning
• features
• ml
• big-data
• feature-store
• python

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

• data
• python
• data-engineering
• data-lake
• data-loading
• data-warehouse

DeepAnalyze is the first agentic LLM for autonomous data science. 🎈你的AI数据分析师，自动分析大量数据，一键生成专业分析报告！

• agent
• agentic
• agentic-ai
• chatbot
• data
• data-analysis

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

• python
• aws
• pandas
• apache-arrow
• apache-parquet
• data-engineering

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

• workflow
• machine-learning
• data-science
• data-engineering
• mlops
• papermill

Compare tables within or across databases

• database
• mysql
• postgresql
• snowflake
• rdbms
• trino

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

• dataops
• dataops-platform
• elt
• open-source
• opensource
• data

Data Contracts engine for the modern data stack. https://www.soda.io

• python
• data-engineering
• data-governance
• data-monitoring
• data-observability
• data-profiling

Implementing best practices for PySpark ETL jobs and applications.

• pyspark
• etl-job
• python
• data-engineering
• spark
• data-science

Python Stream Processing

• python
• stream-processing
• rust
• data-engineering
• data-processing
• data-science

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

• data
• data-engineering
• data-engineering-pipeline
• etl-pipeline
• cassandra-database
• postgresql-database

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

• mlops
• python
• data-science
• machine-learning
• data-engineering
• experiment-tracking

More than 2000+ Data engineer interview questions.

• data-engineering
• interview-questions
• interview
• hadoop
• hadoop-hdfs
• spark

Python Streaming DataFrames for Kafka

• kafka
• python
• stream-processing
• data-engineering
• data-science
• machine-learning

Concurrent Python made simple

• asyncio
• concurrency
• python
• threading
• data-pipelines
• data-processing