🔄 Data Transformation & Pipeline Orchestration
- dbt Core (
dbt-labs/dbt-core): The industry standard for the “T” (Transform) in ELT pipelines. It enables data analysts and engineers to write modular SQL select statements, which dbt automatically turns into tables and views inside analytical databases.
👉 dbt Core GitHub Repository [1] - Apache Airflow (
apache/airflow): A highly flexible platform created by Airbnb to programmatically author, schedule, and monitor complex workflows. It is widely used to orchestrate data processing pipelines across entire enterprise data architectures.
👉 Apache Airflow GitHub Repository [1, 2, 3, 4, 5] - Dagster (
dagster-io/dagster): A modern cloud-native orchestrator designed specifically for data development, production, and observation. It focuses on asset-based workflows, making data pipelines easier to test and monitor.
👉 Dagster GitHub Repository [1, 2, 3, 4]