Skip to content

Build multi-arch images #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from
Draft

Build multi-arch images #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
12bee01
Build multi-arch images
m1so Jul 7, 2025
0401d49
Skip conda images
m1so Jul 7, 2025
1fcbd67
Remove caching from multi-arch images
m1so Jul 8, 2025
f8ddae0
Increase no output timeout to 60m
m1so Jul 8, 2025
b7f5def
fix: Switch to conda-forge channel
m1so Aug 11, 2025
b014fff
Incorporate PR review suggestions
m1so Aug 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 4 additions & 12 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,8 @@ jobs:
--file=./python/base/Dockerfile.base \
--tag=978928340082.dkr.ecr.us-east-1.amazonaws.com/${REPOSITORY}:${TAG}${TAG_SUFFIX} \
--tag=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX} \
--cache-from type=registry,ref=docker.io/${REPOSITORY}:${TAG}-buildcache \
--cache-from type=registry,ref=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX}-buildcache \
--cache-to type=registry,ref=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX}-buildcache,mode=max,image-manifest=true,oci-mediatypes=true \
--progress plain \
--platform linux/amd64 \
--platform linux/amd64,linux/arm64 \
--provenance=false \
--sbom=false \
--output type=registry,push=true \
Expand All @@ -69,6 +66,7 @@ jobs:
- setup_buildkit_builder
- run:
name: Build and push python image
no_output_timeout: 60m # building Python can take a while, default timeout is 10m
command: |
REPOSITORY="deepnote/python"
TAG="${PYTHON_VERSION}"
Expand All @@ -79,11 +77,8 @@ jobs:
--tag=978928340082.dkr.ecr.us-east-1.amazonaws.com/${REPOSITORY}:${TAG}${TAG_SUFFIX} \
--tag=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX} \
--build-arg CIRCLE_PULL_REQUEST=${CIRCLE_PULL_REQUEST} \
--cache-from type=registry,ref=docker.io/${REPOSITORY}:${TAG}-buildcache \
--cache-from type=registry,ref=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX}-buildcache \
--cache-to type=registry,ref=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX}-buildcache,mode=max,image-manifest=true,oci-mediatypes=true \
--progress plain \
--platform linux/amd64 \
--platform linux/amd64,linux/arm64 \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Multi-arch for python: LGTM. Also align python-conda job.

This job targets both platforms. The python-conda job (Lines 142–147) still builds only linux/amd64 and uses registry cache. Consider aligning it:

-              --cache-from type=registry,ref=docker.io/${REPOSITORY}:${TAG}-buildcache \
-              --cache-from type=registry,ref=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX}-buildcache \
-              --cache-to type=registry,ref=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX}-buildcache,mode=max,image-manifest=true,oci-mediatypes=true \
               --progress plain \
-              --platform linux/amd64 \
+              --platform linux/amd64,linux/arm64 \
               --provenance=false \
               --sbom=false \

If you want consistent cache removal across jobs, drop the cache flags too.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
--platform linux/amd64,linux/arm64 \
--progress plain \
--platform linux/amd64,linux/arm64 \
--provenance=false \
--sbom=false \

--provenance=false \
--sbom=false \
--output type=registry,push=true \
Expand Down Expand Up @@ -113,11 +108,8 @@ jobs:
--tag=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX} \
--build-arg CIRCLE_PULL_REQUEST=${CIRCLE_PULL_REQUEST} \
--build-arg PYTHON_VERSION=${PYTHON_VERSION} \
--cache-from type=registry,ref=docker.io/${REPOSITORY}:${TAG}-buildcache \
--cache-from type=registry,ref=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX}-buildcache \
--cache-to type=registry,ref=docker.io/${REPOSITORY}:${TAG}${TAG_SUFFIX}-buildcache,mode=max,image-manifest=true,oci-mediatypes=true \
--progress plain \
--platform linux/amd64 \
--platform linux/amd64,linux/arm64 \
--provenance=false \
--sbom=false \
--output type=registry,push=true \
Expand Down
13 changes: 12 additions & 1 deletion python/conda/Dockerfile.conda
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,20 @@ ENV CONDA_ALWAYS_YES=true
# but that would require baking in the URLs for
# different Miniconda installer versions into the Dockerfile.
ARG PYTHON_VERSION
RUN wget --quiet "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" -O /tmp/miniconda.sh && \
ARG TARGETARCH
RUN case "$TARGETARCH" in \
amd64) MINICONDA_ARCH="x86_64" ;; \
arm64) MINICONDA_ARCH="aarch64" ;; \
*) echo "Unsupported TARGETARCH: $TARGETARCH" >&2; exit 1 ;; \
esac && \
wget --quiet "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-${MINICONDA_ARCH}.sh" -O /tmp/miniconda.sh && \
Comment on lines +21 to +27
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Pin installer + verify checksum to harden supply chain and ensure reproducibility.

“latest” is mutable and unverified. Recommend pinning a specific installer and checking SHA256.

-ARG TARGETARCH
+ARG TARGETARCH
+# Pin Miniconda installer in CI (set these via build args or here)
+ARG MINICONDA_VERSION    # e.g., py311_24.7.1-0
+ARG MINICONDA_SHA256     # corresponding sha256
 RUN case "$TARGETARCH" in \
       amd64) MINICONDA_ARCH="x86_64" ;; \
       arm64) MINICONDA_ARCH="aarch64" ;; \
       *) echo "Unsupported TARGETARCH: $TARGETARCH" >&2; exit 1 ;; \
     esac && \
-    wget --quiet "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-${MINICONDA_ARCH}.sh" -O /tmp/miniconda.sh && \
+    : "${MINICONDA_VERSION:?Set MINICONDA_VERSION}" && \
+    : "${MINICONDA_SHA256:?Set MINICONDA_SHA256}" && \
+    INSTALLER="Miniconda3-${MINICONDA_VERSION}-Linux-${MINICONDA_ARCH}.sh" && \
+    wget --quiet "https://repo.anaconda.com/miniconda/${INSTALLER}" -O /tmp/miniconda.sh && \
+    echo "${MINICONDA_SHA256}  /tmp/miniconda.sh" | sha256sum -c - && \
     /bin/bash /tmp/miniconda.sh -b -p /opt/conda && \
     rm /tmp/miniconda.sh && \
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
ARG TARGETARCH
RUN case "$TARGETARCH" in \
amd64) MINICONDA_ARCH="x86_64" ;; \
arm64) MINICONDA_ARCH="aarch64" ;; \
*) echo "Unsupported TARGETARCH: $TARGETARCH" >&2; exit 1 ;; \
esac && \
wget --quiet "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-${MINICONDA_ARCH}.sh" -O /tmp/miniconda.sh && \
ARG TARGETARCH
# Pin Miniconda installer in CI (set these via build args or here)
ARG MINICONDA_VERSION # e.g., py311_24.7.1-0
ARG MINICONDA_SHA256 # corresponding sha256
RUN case "$TARGETARCH" in \
amd64) MINICONDA_ARCH="x86_64" ;; \
arm64) MINICONDA_ARCH="aarch64" ;; \
*) echo "Unsupported TARGETARCH: $TARGETARCH" >&2; exit 1 ;; \
esac && \
: "${MINICONDA_VERSION:?Set MINICONDA_VERSION}" && \
: "${MINICONDA_SHA256:?Set MINICONDA_SHA256}" && \
INSTALLER="Miniconda3-${MINICONDA_VERSION}-Linux-${MINICONDA_ARCH}.sh" && \
wget --quiet "https://repo.anaconda.com/miniconda/${INSTALLER}" -O /tmp/miniconda.sh && \
echo "${MINICONDA_SHA256} /tmp/miniconda.sh" | sha256sum -c - && \
/bin/bash /tmp/miniconda.sh -b -p /opt/conda && \
rm /tmp/miniconda.sh && \

🛠️ Refactor suggestion

Good TARGETARCH mapping; add a safe default for local builds.

Buildx injects TARGETARCH, but plain docker build won’t. Default to amd64 to keep local builds working.

-ARG TARGETARCH
-RUN case "$TARGETARCH" in \
+ARG TARGETARCH
+# Default TARGETARCH for local non-buildx builds
+RUN : "${TARGETARCH:=amd64}" && \
+  case "$TARGETARCH" in \
       amd64) MINICONDA_ARCH="x86_64" ;; \
       arm64) MINICONDA_ARCH="aarch64" ;; \
       *) echo "Unsupported TARGETARCH: $TARGETARCH" >&2; exit 1 ;; \
-    esac && \
+  esac && \
   wget --quiet "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-${MINICONDA_ARCH}.sh" -O /tmp/miniconda.sh && \
🤖 Prompt for AI Agents 
In python/conda/Dockerfile.conda around lines 21 to 27, the build assumes
TARGETARCH is provided by buildx which breaks plain docker build locally; set a
safe default by declaring ARG TARGETARCH=amd64 (or add a fallback in the case
statement to treat empty/unknown as amd64) so local builds default to amd64 and
the existing architecture mapping and wget URL resolution continue to work.

/bin/bash /tmp/miniconda.sh -b -p /opt/conda && \
rm /tmp/miniconda.sh && \
# Use community packages from conda-forge instead of Anaconda Inc. default channels
# which require accepting terms of service & using commercial license for orgs with more than 200 employees
/opt/conda/bin/conda config --add channels conda-forge && \
/opt/conda/bin/conda config --remove channels defaults || true && \
/opt/conda/bin/conda config --set channel_priority strict && \
# Install the correct version of python (as the time of writing, anaconda
# installed python 3.11 by default) for parity with our base image
/opt/conda/bin/conda install python=${PYTHON_VERSION} && \
Expand Down