Skip to content

Conversation

@thatsmydoing
Copy link
Contributor

@thatsmydoing thatsmydoing commented Dec 15, 2025

All kubernetes pods have an infra container (using pause:latest) to setup networking etc. This image is loaded into containerd when kubelet starts. However, due to a misconfiguration the image can get GC-ed and kubelet tries to pull it from Docker Hub but the image does not exist there. This prevents any new pods from being created.

Pinning of the infra image for kubernetes is delegated to the CRI implementation (containerd) since Kubernetes 1.29, and the --pod-infra-container-image flag does nothing and will be fully removed in 1.35.

containerd (config version 2) uses the sandbox_image setting to know what images to pin. However, while it normalizes pause:latest to docker.io/library/pause:latest in the image list, it does not normalize the setting value when checking if the image should be pinned or not. Using the fully qualified name in the setting is enough to make it be correctly pinned after a full containerd restart.

Things done

  • Built on platform:
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • Tested, as applicable:
  • Ran nixpkgs-review on this PR. See nixpkgs-review usage.
  • Tested basic functionality of all binary files, usually in ./result/bin/.
  • Nixpkgs Release Notes
    • Package update: when the change is major or breaking.
  • NixOS Release Notes
    • Module addition: when adding a new NixOS module.
    • Module update: when the change is significant.
  • Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

Add a 👍 reaction to pull requests you find important.

All kubernetes pods have an infra container (using `pause:latest`) to
setup networking etc. This image is loaded into containerd when kubelet
starts. However, due to a misconfiguration the image can get GC-ed and
kubelet tries to pull it from Docker Hub but the image does not exist
there. This prevents any new pods from being created.

Pinning of the infra image for kubernetes is delegated to the CRI
implementation (containerd) since Kubernetes 1.29, and the
`--pod-infra-container-image` flag does nothing and will be fully
removed in 1.35.

containerd (config version 2) uses the `sandbox_image` setting to know
what images to pin. However, while it normalizes `pause:latest` to
`docker.io/library/pause:latest` in the image list, it does not
normalize the setting value when checking if the image should be pinned
or not. Using the fully qualified name in the setting is enough to make
it be correctly pinned after a full containerd restart.
@nixpkgs-ci nixpkgs-ci bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 9.needs: reviewer This PR currently has no reviewers requested and needs attention. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Dec 15, 2025
@Aleksanaa Aleksanaa requested a review from a team December 16, 2025 07:46
Copy link
Contributor

@johanot johanot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this branch vs. nixos-unstable locally and can indeed confirm the behavior. Thanks for fixing this!

@johanot johanot added the backport release-25.11 Backport PR automatically label Dec 16, 2025
@johanot
Copy link
Contributor

johanot commented Dec 16, 2025

@ofborg test kubernetes

@nixpkgs-ci nixpkgs-ci bot added 12.approvals: 1 This PR was reviewed and approved by one person. and removed 9.needs: reviewer This PR currently has no reviewers requested and needs attention. labels Dec 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 12.approvals: 1 This PR was reviewed and approved by one person. backport release-25.11 Backport PR automatically

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants