Drop support for all Airflow versions below 2.11 in Helm Chart

chart/README.md

@@ -37,13 +37,14 @@ cluster using the [Helm](https://helm.sh) package manager.
 ## Features
 
 * Supported executors (all Airflow versions): ``LocalExecutor``, ``CeleryExecutor``, ``KubernetesExecutor``
-* Supported executors (Airflow version ``2.X.X``): ``LocalKubernetesExecutor``, ``CeleryKubernetesExecutor``
+* Supported executors (Airflow version ``2.11.X``): ``LocalKubernetesExecutor``, ``CeleryKubernetesExecutor``
+* Supported multiple Executors (``2.11+``)
 * Supported AWS executors with AWS provider version ``8.21.0+``:
    * ``airflow.providers.amazon.aws.executors.batch.AwsBatchExecutor``
    * ``airflow.providers.amazon.aws.executors.ecs.AwsEcsExecutor``
 * Supported Edge executor with edge3 provider version ``1.0.0+``:
    * ``airflow.providers.edge3.executors.EdgeExecutor``
-* Supported Airflow version: ``1.10+``, ``2.0+``, ``3.0+``
+* Supported Airflow version: ``2.11+``, ``3.0+``
 * Supported database backend: ``PostgreSQL``, ``MySQL``
 * Autoscaling for ``CeleryExecutor`` provided by KEDA
 * ``PostgreSQL`` and ``PgBouncer`` with a battle-tested configuration

chart/docs/index.rst

@@ -69,16 +69,16 @@ Features
 --------
 
 * Supported executors (all Airflow versions): ``LocalExecutor``, ``CeleryExecutor``, ``KubernetesExecutor``
-* Supported hybrid static executors (Airflow version ``2.X.X``): ``LocalKubernetesExecutor``, ``CeleryKubernetesExecutor``
-* Supported Hybrid Executors (``2.10+``)
+* Supported hybrid static executors (Airflow version ``2.11.X``): ``LocalKubernetesExecutor``, ``CeleryKubernetesExecutor``
+* Supported multiple Executors (``2.11+``)
 * Supported AWS executors with AWS provider version ``8.21.0+``:
    * ``airflow.providers.amazon.aws.executors.batch.AwsBatchExecutor``
    * ``airflow.providers.amazon.aws.executors.ecs.AwsEcsExecutor``
 * Supported AWS executors with AWS provider version ``9.9.0+``:
    * ``airflow.providers.amazon.aws.executors.aws_lambda.lambda_executor.AwsLambdaExecutor``
 * Supported Edge executor with edge3 provider version ``1.0.0+``:
    * ``airflow.providers.edge3.executors.EdgeExecutor``
-* Supported Airflow version: ``1.10+``, ``2.0+``, ``3.0+``
+* Supported Airflow version: ``2.11+``, ``3.0+``
 * Supported database backend: ``PostgreSQL``, ``MySQL``
 * Autoscaling for ``CeleryExecutor`` provided by KEDA
 * ``PostgreSQL`` and ``PgBouncer`` with a battle-tested configuration

chart/docs/manage-dag-files.rst

@@ -120,12 +120,12 @@ for details.
       # by setting the  dags.persistence.* and dags.gitSync.* values
       # Please refer to values.yaml for details
 
-
 Mounting Dags using git-sync sidecar without persistence
 ........................................................
 
-This option will use an always running Git-Sync sidecar on every scheduler, webserver (if ``airflowVersion < 2.0.0``)
-and worker pods.
+This option will use an always running Git-Sync sidecar on every dag-processor and worker and triggerer pods.
+(In Airflow 2.11 if separate dag-processor is not enabled then Git-Sync sidecar will run on scheduler for Dag parsing as well)
+
 The Git-Sync sidecar containers will sync Dags from a git repository every configured number of
 seconds. If you are using the ``KubernetesExecutor``, Git-sync will run as an init container on your worker pods.
 
@@ -138,8 +138,6 @@ seconds. If you are using the ``KubernetesExecutor``, Git-sync will run as an in
       # by setting the dags.gitSync.* values
       # Refer values.yaml for details
 
-When using ``apache-airflow >= 2.0.0``, :ref:`Dag Serialization <apache-airflow:dag-serialization>` is enabled by default,
-hence Webserver does not need access to Dag files, so ``git-sync`` sidecar is not run on Webserver.
 
 Notes for combining git-sync and persistence
 ............................................

chart/docs/production-guide.rst

@@ -79,9 +79,6 @@ Finally, configure the chart to use the secret you created:
   data:
     metadataSecretName: mydatabase
 
-.. warning::
-  If you use ``CeleryExecutor`` and Airflow version < ``2.4``, keep in mind that ``resultBackendSecretName`` expects a url that starts with ``db+postgresql://``, while ``metadataSecretName`` expects ``postgresql://`` and won't work with ``db+postgresql://``. You'll need to create separate secrets with the correct scheme. For Airflow version >= ``2.4`` it is possible to omit the result backend secret, as Airflow will use ``sql_alchemy_conn`` (specified in ``metadataSecret``) with a db+ scheme prefix by default.
-
 .. _production-guide:pgbouncer:
 
 Metadata DB cleanup
@@ -664,13 +661,11 @@ Here is the full list of secrets that can be disabled and replaced by ``_CMD`` a
 +-------------------------------------------------------+------------------------------------------+--------------------------------------------------+
 | ``<RELEASE_NAME>-webserver-secret-key``               | ``.Values.webserverSecretKeySecretName`` | ``AIRFLOW__WEBSERVER__SECRET_KEY``               |
 +-------------------------------------------------------+------------------------------------------+--------------------------------------------------+
-| ``<RELEASE_NAME>-airflow-result-backend``             | ``.Values.data.resultBackendSecretName`` | | ``AIRFLOW__CELERY__CELERY_RESULT_BACKEND``     |
-|                                                       |                                          | | ``AIRFLOW__CELERY__RESULT_BACKEND``            |
+| ``<RELEASE_NAME>-airflow-result-backend``             | ``.Values.data.resultBackendSecretName`` | ``AIRFLOW__CELERY__RESULT_BACKEND``              |
 +-------------------------------------------------------+------------------------------------------+--------------------------------------------------+
 | ``<RELEASE_NAME>-airflow-broker-url``                 | ``.Values.data.brokerUrlSecretName``     | ``AIRFLOW__CELERY__BROKER_URL``                  |
 +-------------------------------------------------------+------------------------------------------+--------------------------------------------------+
-| ``<RELEASE_NAME>-elasticsearch``                      | ``.Values.elasticsearch.secretName``     | | ``AIRFLOW__ELASTICSEARCH__HOST``               |
-|                                                       |                                          | | ``AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST`` |
+| ``<RELEASE_NAME>-elasticsearch``                      | ``.Values.elasticsearch.secretName``     | ``AIRFLOW__ELASTICSEARCH__HOST``                 |
 +-------------------------------------------------------+------------------------------------------+--------------------------------------------------+
 
 There are also a number of secrets, which names are also determined from the release name, that do not need to

chart/docs/quick-start.rst

@@ -96,7 +96,7 @@ packages, or even custom providers.
    when either the packages you want to install or Airflow is upgraded. Please do not forget about keeping these scripts.
    Also keep in mind, that in cases when you run pure Python tasks, you can use the
    `Python Virtualenv functions <https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator>`_
-   which will dynamically source and install python dependencies during runtime. With Airflow 2.8.0 Virtualenvs can also be cached.
+   which will dynamically source and install python dependencies during runtime. Virtualenvs can also be cached.
 
 The best way to achieve it, is to build your own, custom image.
 

chart/files/pod-template-file.kubernetes-helm-yaml

@@ -53,7 +53,7 @@ spec:
     {{- if .Values.workers.extraInitContainers }}
       {{- tpl (toYaml .Values.workers.extraInitContainers) . | nindent 4 }}
     {{- end }}
-    {{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) (or .Values.workers.kubernetes.kerberosInitContainer.enabled .Values.workers.kerberosInitContainer.enabled) }}
+    {{- if or .Values.workers.kubernetes.kerberosInitContainer.enabled .Values.workers.kerberosInitContainer.enabled }}
     - name: kerberos-init
       image: {{ template "airflow_image" . }}
       imagePullPolicy: {{ .Values.images.airflow.pullPolicy }}

chart/newsfragments/61018.significant.rst

@@ -0,0 +1,4 @@
+Support for old versions of Apache Airflow <2.11 has been dropped.
+
+Minimum supported version of Apache Airflow is now 2.11.0. If you want to deploy an
+old version of Apache Airflow, please use the last released version of the chart 1.19.0.

chart/templates/NOTES.txt

@@ -204,7 +204,7 @@ DEPRECATION WARNING:
 
 {{- end }}
 
-{{- if ne (.Values.workers.args | toJson) (list "bash" "-c" "exec \\\nairflow {{ semverCompare \">=2.0.0\" .Values.airflowVersion | ternary \"celery worker\" \"worker\" }}\n{{- if and .Values.workers.queue (ne .Values.workers.queue \"default\") }}\n{{- \" -q \" }}{{ .Values.workers.queue }}\n{{- end }}" | toJson) }}
+{{- if ne (.Values.workers.args | toJson) (list "bash" "-c" "exec \\\nairflow celery worker\n{{- if and .Values.workers.queue (ne .Values.workers.queue \"default\") }}\n{{- \" -q \" }}{{ .Values.workers.queue }}\n{{- end }}" | toJson) }}
 
  DEPRECATION WARNING:
     `workers.args` has been renamed to `workers.celery.args`.

chart/templates/_helpers.yaml

@@ -62,14 +62,6 @@ If release name contains chart name it will be used as a full name.
   {{- end }}
   - name: AIRFLOW_HOME
     value: {{ .Values.airflowHome }}
-  # For Airflow <2.3, backward compatibility; moved to [database] in 2.3
-  {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__CORE__SQL_ALCHEMY_CONN }}
-  - name: AIRFLOW__CORE__SQL_ALCHEMY_CONN
-    valueFrom:
-      secretKeyRef:
-        name: {{ template "airflow_metadata_secret" . }}
-        key: connection
-  {{- end }}
   {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__DATABASE__SQL_ALCHEMY_CONN }}
   - name: AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
     valueFrom:
@@ -113,15 +105,7 @@ If release name contains chart name it will be used as a full name.
         key: jwt-secret
   {{- end }}
   {{- if or (contains "CeleryExecutor" .Values.executor) (contains "CeleryKubernetesExecutor" .Values.executor) }}
-    {{- if or (semverCompare "<2.4.0" .Values.airflowVersion) (.Values.data.resultBackendSecretName) (.Values.data.resultBackendConnection) }}
-    {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__CELERY__CELERY_RESULT_BACKEND }}
-  # (Airflow 1.10.* variant)
-  - name: AIRFLOW__CELERY__CELERY_RESULT_BACKEND
-    valueFrom:
-      secretKeyRef:
-        name: {{ template "airflow_result_backend_secret" . }}
-        key: connection
-    {{- end }}
+    {{- if or (.Values.data.resultBackendSecretName) (.Values.data.resultBackendConnection) }}
     {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__CELERY__RESULT_BACKEND }}
   - name: AIRFLOW__CELERY__RESULT_BACKEND
     valueFrom:
@@ -139,22 +123,13 @@ If release name contains chart name it will be used as a full name.
     {{- end }}
   {{- end }}
   {{- if .Values.elasticsearch.enabled }}
-  # The elasticsearch variables were updated to the shorter names in v1.10.4
     {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__ELASTICSEARCH__HOST }}
   - name: AIRFLOW__ELASTICSEARCH__HOST
     valueFrom:
       secretKeyRef:
         name: {{ template "elasticsearch_secret" . }}
         key: connection
     {{- end }}
-    {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST }}
-  # This is the older format for these variable names, kept here for backward compatibility
-  - name: AIRFLOW__ELASTICSEARCH__ELASTICSEARCH_HOST
-    valueFrom:
-      secretKeyRef:
-        name: {{ template "elasticsearch_secret" . }}
-        key: connection
-    {{- end }}
   {{- end }}
   {{- if .Values.opensearch.enabled }}
     {{- if .Values.enableBuiltInSecretEnvVars.AIRFLOW__OPENSEARCH__HOST }}
@@ -498,11 +473,7 @@ If release name contains chart name it will be used as a full name.
 {{- end }}
 
 {{- define "celery_executor_namespace" -}}
-  {{- if semverCompare ">=2.7.0" .Values.airflowVersion }}
-    {{- print "airflow.providers.celery.executors.celery_executor.app" -}}
-  {{- else }}
-    {{- print "airflow.executors.celery_executor.app" -}}
-  {{- end }}
+  {{- print "airflow.providers.celery.executors.celery_executor.app" -}}
 {{- end }}
 
 {{- define "pgbouncer_config" -}}
@@ -745,150 +716,44 @@ server_tls_key_file = /etc/pgbouncer/server.key
   {{- include "_serviceAccountName" (merge (dict "key" "databaseCleanup" "nameSuffix" "database-cleanup") .) -}}
 {{- end }}
 
-{{- define "wait-for-migrations-command" -}}
-  {{- if semverCompare ">=2.0.0" .Values.airflowVersion }}
+{{- define "wait-for-migrations-command" }}
   - airflow
   - db
   - check-migrations
   - --migration-wait-timeout={{ .Values.images.migrationsWaitTimeout }}
-  {{- else }}
-  - python
-  - -c
-  - |
-        import airflow
-        import logging
-        import os
-        import time
-
-        from alembic.config import Config
-        from alembic.runtime.migration import MigrationContext
-        from alembic.script import ScriptDirectory
-
-        from airflow import settings
-
-        package_dir = os.path.abspath(os.path.dirname(airflow.__file__))
-        directory = os.path.join(package_dir, 'migrations')
-        config = Config(os.path.join(package_dir, 'alembic.ini'))
-        config.set_main_option('script_location', directory)
-        config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN.replace('%', '%%'))
-        script_ = ScriptDirectory.from_config(config)
-
-        timeout=60
-
-        with settings.engine.connect() as connection:
-            context = MigrationContext.configure(connection)
-            ticker = 0
-            while True:
-                source_heads = set(script_.get_heads())
-
-                db_heads = set(context.get_current_heads())
-                if source_heads == db_heads:
-                    break
-
-                if ticker >= timeout:
-                    raise TimeoutError("There are still unapplied migrations after {} seconds.".format(ticker))
-                ticker += 1
-                time.sleep(1)
-                logging.info('Waiting for migrations... %s second(s)', ticker)
-  {{- end }}
 {{- end }}
 
 {{- define "scheduler_liveness_check_command" }}
-  {{- if semverCompare ">=2.5.0" .Values.airflowVersion }}
   - sh
   - -c
   - |
     CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
     airflow jobs check --job-type SchedulerJob --local
-  {{- else if semverCompare ">=2.1.0" .Values.airflowVersion }}
-  - sh
-  - -c
-  - |
-    CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
-    airflow jobs check --job-type SchedulerJob --hostname $(hostname)
-  {{- else }}
-  - sh
-  - -c
-  - |
-    CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint python -Wignore -c "
-    import os
-    os.environ['AIRFLOW__CORE__LOGGING_LEVEL'] = 'ERROR'
-    os.environ['AIRFLOW__LOGGING__LOGGING_LEVEL'] = 'ERROR'
-    from airflow.jobs.scheduler_job import SchedulerJob
-    from airflow.utils.db import create_session
-    from airflow.utils.net import get_hostname
-    import sys
-    with create_session() as session:
-        job = session.query(SchedulerJob).filter_by(hostname=get_hostname()).order_by(
-            SchedulerJob.latest_heartbeat.desc()).limit(1).first()
-    sys.exit(0 if job.is_alive() else 1)"
-  {{- end }}
 {{- end }}
 
 
 {{- define  "scheduler_startup_check_command" }}
-  {{- if semverCompare ">=2.5.0" .Values.airflowVersion }}
   - sh
   - -c
   - |
     CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
     airflow jobs check --job-type SchedulerJob --local
-  {{- else if semverCompare ">=2.1.0" .Values.airflowVersion }}
-  - sh
-  - -c
-  - |
-    CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
-    airflow jobs check --job-type SchedulerJob --hostname $(hostname)
-  {{- else }}
-  - sh
-  - -c
-  - |
-    CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint python -Wignore -c "
-    import os
-    os.environ['AIRFLOW__CORE__LOGGING_LEVEL'] = 'ERROR'
-    os.environ['AIRFLOW__LOGGING__LOGGING_LEVEL'] = 'ERROR'
-    from airflow.jobs.scheduler_job import SchedulerJob
-    from airflow.utils.db import create_session
-    from airflow.utils.net import get_hostname
-    import sys
-    with create_session() as session:
-        job = session.query(SchedulerJob).filter_by(hostname=get_hostname()).order_by(
-            SchedulerJob.latest_heartbeat.desc()).limit(1).first()
-    sys.exit(0 if job.is_alive() else 1)"
-  {{- end }}
 {{- end }}
 
 {{- define "triggerer_liveness_check_command" }}
-  {{- if semverCompare ">=2.5.0" .Values.airflowVersion }}
   - sh
   - -c
   - |
     CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
     airflow jobs check --job-type TriggererJob --local
-  {{- else }}
-  - sh
-  - -c
-  - |
-    CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
-    airflow jobs check --job-type TriggererJob --hostname $(hostname)
-  {{- end }}
 {{- end }}
 
 {{- define "dag_processor_liveness_check_command" }}
-  {{- $commandArgs := (list) -}}
-  {{- if semverCompare ">=2.5.0" .Values.airflowVersion }}
-    {{- $commandArgs = append $commandArgs "--local" -}}
-    {{- if semverCompare ">=2.5.2" .Values.airflowVersion }}
-      {{- $commandArgs = concat $commandArgs (list "--job-type" "DagProcessorJob") -}}
-    {{- end }}
-  {{- else }}
-    {{- $commandArgs = concat $commandArgs (list "--hostname" "$(hostname)") -}}
-  {{- end }}
   - sh
   - -c
   - |
     CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
-    airflow jobs check {{ join " " $commandArgs }}
+    airflow jobs check --local --job-type DagProcessorJob
 {{- end }}
 
 {{- define "registry_docker_config" }}

chart/templates/check-values.yaml

@@ -21,6 +21,14 @@
 The sole purpose of this YAML file is it to check the values file is consistent for some complex combinations.
 */ -}}
 
+{{- /*
+##############################
+   Version checks
+#############################
+*/ -}}
+  {{- if semverCompare "<2.11.0" .Values.airflowVersion }}
+    {{ required "This chart only supports Apache Airflow version 2.11.0 and above." nil }}
+  {{- end }}
 {{- /*
 ##############################
    Redis related checks

chart/templates/configmaps/configmap.yaml

@@ -69,14 +69,12 @@ data:
   {{- end }}
 
 {{- if or (contains "LocalKubernetesExecutor" $.Values.executor) (contains "KubernetesExecutor" $.Values.executor) (contains "CeleryKubernetesExecutor" $.Values.executor) }}
-{{- if semverCompare ">=1.10.12" .Values.airflowVersion }}
   pod_template_file.yaml: |-
     {{- if .Values.podTemplate }}
       {{- tpl .Values.podTemplate . | nindent 4 }}
     {{- else }}
       {{- tpl (.Files.Get "files/pod-template-file.kubernetes-helm-yaml") . | nindent 4 }}
     {{- end }}
-{{- end }}
 {{- end }}
 
   {{- if .Values.kerberos.enabled }}

chart/templates/dag-processor/dag-processor-deployment.yaml

@@ -20,7 +20,6 @@
 ################################
 ## Airflow Dag Processor Deployment
 #################################
-{{- if semverCompare ">=2.3.0" .Values.airflowVersion }}
 {{- $enabled := .Values.dagProcessor.enabled }}
 {{- if eq $enabled nil}}
   {{ $enabled = ternary true false (semverCompare ">=3.0.0" .Values.airflowVersion) }}
@@ -286,4 +285,3 @@ spec:
           emptyDir: {{- toYaml (default (dict) .Values.logs.emptyDirConfig) | nindent 12 }}
 {{- end }}
 {{- end }}
-{{- end }}

chart/templates/dag-processor/dag-processor-poddisruptionbudget.yaml

@@ -20,7 +20,6 @@
 ################################
 ## Airflow Dag Processor PodDisruptionBudget
 #################################
-{{- if semverCompare ">=2.3.0" .Values.airflowVersion }}
 {{- $enabled := .Values.dagProcessor.enabled }}
 {{- if eq $enabled nil}}
   {{ $enabled = ternary true false (semverCompare ">=3.0.0" .Values.airflowVersion) }}
@@ -47,4 +46,3 @@ spec:
       release: {{ .Release.Name }}
   {{- toYaml .Values.dagProcessor.podDisruptionBudget.config | nindent 2 }}
 {{- end }}
-{{- end }}