Dockerfile Changes

[tcyrus]

• Use cache mounts to help with rebuilds and image clean-up
• Use ADD --checksum instead of curl/wget + sha256sum
• Use scratch stages + bind mounts to help with caching and image clean-up

[tcyrus]

I'll try to resolve the conflicts soon. Would I be able to get approval for checks after that?

[marcoesters]

Hi, thank you for your PR. I will try to get to reviewing it as soon as possible, I'm just a little swamped.

Could you elaborate on what problem you are trying to solve? Are there performance issues with the builds?

[tcyrus]

I was working on something similar to the miniconda3 Dockerfile and noticed that I could speed up builds with bind mounts, cache mounts, and by separating the file download + checksum into it's own stage.
By using ADD --checksum to download the installer in its own stage, docker can cache the file and the main stage no longer needs curl/wget + sha256sum.

[marcoesters]

Thank you for your patience! I tested your branch on Debian Trixie (the version we use for Miniconda Docker images) using the Miniconda and the Anaconda Amazon Linux images to benchmark.

I really like the scratch image structure and the ADD commands using the checksum. This will allow for better caching behavior for repeated builds.

However, I did not see any substantial speed increase due to the cache mount. I would have expected a speed increase for Miniconda since both my system and the image use Debian 13.3. Could you share what kind of speed increase you have seen?

Also, I disabled the workflow for those package build images for now, since the failures are unrelated to your changes.

[marcoesters]

This change is causing an interesting issue when trying to build it on Debian Trixie. When I build your Miniconda image and then the Anaconda image, the build fails with the error shown below. I have to run docker prune -a to get it to work. It does succeed with the Dockerfile on main though.

 => ERROR [stage-2 2/2] RUN --mount=type=cache,target=/var/cache/yum,sharing=locked     --mount=type=bind,target=/tmp/anaconda.sh,source=/anaconda.sh,from=anaconda_installer     yum install -y         bzi  4.2s
------                                                                                                                                                                                                             
 > [stage-2 2/2] RUN --mount=type=cache,target=/var/cache/yum,sharing=locked     --mount=type=bind,target=/tmp/anaconda.sh,source=/anaconda.sh,from=anaconda_installer     yum install -y         bzip2         ca-certificates         libglib2.0-0         libXext         libSM         libXrender         libXcomposite         libXcursor         libXdamage         libXfixes         libXi         libXinerama         libXrandr         git         mercurial         subversion &&     /bin/bash /tmp/anaconda.sh -b -p /opt/conda &&     ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh &&     echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc &&     echo "conda activate" >> ~/.bashrc &&     find /opt/conda/ -follow -type f -name '*.a' -delete &&     find /opt/conda/ -follow -type f -name '*.js.map' -delete &&     /opt/conda/bin/conda clean -afy:
0.560 Loaded plugins: ovl, priorities
4.025 https://cdn.amazonlinux.com/2/core/2.0/aarch64/9f862452182adbc3431f0dcf54ac57a518f3e987d65b6b61bd3455cc477e12d5/repodata/comps.xml.gz?instance_id=URLError&region=None: [Errno 14] curl#6 - "Could not resolve host: cdn.amazonlinux.com"
4.026 Trying other mirror.
4.027 https://cdn.amazonlinux.com/2/core/2.0/aarch64/9f862452182adbc3431f0dcf54ac57a518f3e987d65b6b61bd3455cc477e12d5/repodata/updateinfo.xml.gz?instance_id=URLError&region=None: [Errno 14] curl#6 - "Could not resolve host: cdn.amazonlinux.com"
4.027 Trying other mirror.
4.039 https://cdn.amazonlinux.com/2/core/2.0/aarch64/9f862452182adbc3431f0dcf54ac57a518f3e987d65b6b61bd3455cc477e12d5/repodata/primary.sqlite.gz?instance_id=URLError&region=None: [Errno 14] curl#6 - "Could not resolve host: cdn.amazonlinux.com"
4.039 Trying other mirror.
4.080 https://cdn.amazonlinux.com/2/core/2.0/aarch64/9f862452182adbc3431f0dcf54ac57a518f3e987d65b6b61bd3455cc477e12d5/repodata/primary.sqlite.gz?instance_id=URLError&region=None: [Errno 14] curl#6 - "Could not resolve host: cdn.amazonlinux.com"
4.080 Trying other mirror.
4.093 https://cdn.amazonlinux.com/2/core/2.0/aarch64/9f862452182adbc3431f0dcf54ac57a518f3e987d65b6b61bd3455cc477e12d5/repodata/primary.sqlite.gz?instance_id=URLError&region=None: [Errno 14] curl#6 - "Could not resolve host: cdn.amazonlinux.com"
4.093 Trying other mirror.
4.094 
4.094 
4.094  One of the configured repositories failed (Amazon Linux 2 core repository),
4.094  and yum doesn't have enough cached data to continue. At this point the only
4.094  safe thing yum can do is fail. There are a few ways to work "fix" this:
4.094 
4.094      1. Contact the upstream for the repository and get them to fix the problem.
4.094 
4.094      2. Reconfigure the baseurl/etc. for the repository, to point to a working
4.094         upstream. This is most often useful if you are using a newer
4.094         distribution release than is supported by the repository (and the
4.094         packages for the previous distribution release still work).
4.094 
4.094      3. Run the command with the repository temporarily disabled
4.094             yum --disablerepo=amzn2-core ...
4.094 
4.094      4. Disable the repository permanently, so yum won't use it by default. Yum
4.094         will then just ignore the repository until you permanently enable it
4.094         again or use --enablerepo for temporary usage:
4.094 
4.094             yum-config-manager --disable amzn2-core
4.094         or
4.094             subscription-manager repos --disable=amzn2-core
4.094 
4.094      5. Configure the failing repository to be skipped, if it is unavailable.
4.094         Note that yum will try to contact the repo. when it runs most commands,
4.094         so will have to try and fail each time (and thus. yum will be be much
4.094         slower). If it is a very temporary problem though, this is often a nice
4.094         compromise:
4.094 
4.094             yum-config-manager --save --setopt=amzn2-core.skip_if_unavailable=true
4.094 
4.094 failure: repodata/primary.sqlite.gz from amzn2-core: [Errno 256] No more mirrors to try.
4.094 https://cdn.amazonlinux.com/2/core/2.0/aarch64/9f862452182adbc3431f0dcf54ac57a518f3e987d65b6b61bd3455cc477e12d5/repodata/primary.sqlite.gz?instance_id=URLError&region=None: [Errno 14] curl#6 - "Could not resolve host: cdn.amazonlinux.com"
------
Dockerfile:16
--------------------
  15 |     # hadolint ignore=DL3033
  16 | >>> RUN --mount=type=cache,target=/var/cache/yum,sharing=locked \
  17 | >>>     --mount=type=bind,target=/tmp/anaconda.sh,source=/anaconda.sh,from=anaconda_installer \
  18 | >>>     yum install -y \
  19 | >>>         bzip2 \
  20 | >>>         ca-certificates \
  21 | >>>         libglib2.0-0 \
  22 | >>>         libXext \
  23 | >>>         libSM \
  24 | >>>         libXrender \
  25 | >>>         libXcomposite \
  26 | >>>         libXcursor \
  27 | >>>         libXdamage \
  28 | >>>         libXfixes \
  29 | >>>         libXi \
  30 | >>>         libXinerama \
  31 | >>>         libXrandr \
  32 | >>>         git \
  33 | >>>         mercurial \
  34 | >>>         subversion && \
  35 | >>>     /bin/bash /tmp/anaconda.sh -b -p /opt/conda && \
  36 | >>>     ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
  37 | >>>     echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
  38 | >>>     echo "conda activate" >> ~/.bashrc && \
  39 | >>>     find /opt/conda/ -follow -type f -name '*.a' -delete && \
  40 | >>>     find /opt/conda/ -follow -type f -name '*.js.map' -delete && \
  41 | >>>     /opt/conda/bin/conda clean -afy

[tcyrus]

I'm not sure what could be causing the issue, but I'll see if I can reproduce and debug it