Reference: End-to-End ETL Pipeline

YouTube Channel

Watch the tutorial

Documentation

Apache Airflow Official Documentation (Follow this if you are using Linux or Mac)

---

Installation Instructions

For Windows (PowerShell)

1. Download the Docker Compose YAML file:

   Invoke-WebRequest -Uri "https://airflow.apache.org/docs/apache-airflow/2.9.1/docker-compose.yaml" -OutFile "docker-compose.yaml"

2. Create necessary directories:

   New-Item -ItemType Directory -Force -Path ".\dags", ".\logs", ".\plugins", ".\config"
   "AIRFLOW_UID=50000" | Out-File -Encoding ASCII -FilePath .env

3. Initialize Airflow:

   docker compose up airflow-init

4. Start Airflow:

   docker compose up -d

5. Check logs:

   docker compose logs -f

6. Shut down Airflow:

   docker compose down

---

For Linux/Mac

1. Download the Docker Compose YAML file:

   curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.9.1/docker-compose.yaml'

2. make sure you have docker in your LINUX:

   sudo apt install docker-compose

3. Create necessary directories:

   mkdir -p dags logs plugins config
   echo -e "AIRFLOW_UID=$(id -u)" > .env

4. Initialize Airflow:

   docker-compose up airflow-init

5. Start Airflow:

   docker-compose up -d

6. Check logs:

   docker-compose logs -f

7. On VPS Server you can check what is live to be hosted:

   sudo netstat -tuln

8. Shut down Airflow:

   docker-compose down

Note: You can check if your port is accessible here you get signal