Refactor image management to use transfer service by ChengyuZhu6 · Pull Request #4583 · containerd/nerdctl

ChengyuZhu6 · 2025-11-05T15:36:40Z

ChengyuZhu6 · 2025-11-25T12:15:41Z

AkihiroSuda · 2025-11-26T00:45:27Z

For limitation 2, there are two potential solutions:

Implement containerd version detection in nerdctl to use different code paths;

Backport to containerd 1.7.x

For limitation 1, there are two potential solutions:

IPFS Registry and Nondistributable Artifacts remain on the legacy resolver-based approach.

containerd transfer API supports custom resolvers

2 in the long term, 1 in the short term

Switch image operations to the transfer API with structured progress reporting and improved TLS/HTTP fallback behavior. Introduce shared helpers for credentials, error classification, progress rendering, and transfer-based import/tag/save flows, updating tests to reflect the new UX. Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>

AkihiroSuda · 2025-11-28T07:05:40Z

+    enable_remote_snapshot_annotations = "true"
+[[plugins."io.containerd.transfer.v1.local".unpack_config]]
+  platform = "linux"
+  snapshotter = "overlayfs"


What happens if this config is not specified?
i.e., what happens with the default config?

By default, it would be

[[plugins."io.containerd.transfer.v1.local".unpack_config]] platform = "linux/amd64"(matches the host's OS/Architecture) snapshotter = "overlayfs" differ = "walking"

AkihiroSuda · 2025-11-28T07:06:56Z

+	"github.com/containerd/containerd/v2/pkg/progress"
+)
+
+// From https://github.com/containerd/containerd/blob/v2.2.0-rc.0/cmd/ctr/commands/image/pull.go#L240-L473


Can't we import the package from containerd/containerd to reduce code clones?

The progress handling code is in cmd/ctr/commands/images, not in pkg/ or core/, so it cannot be imported directly.
And I removed displayName() function that splits names by '-':

func displayName(name string) string { parts := strings.Split(name, "-") for i := range parts { parts[i] = shortenName(parts[i]) } return strings.Join(parts, " ") }

which would break display for image names containing - (e.g., ubuntu-base )

AkihiroSuda · 2025-11-28T07:22:46Z

+}
+
+// SupportsFullTransferService checks if the containerd version fully supports the Transfer service.
+// While containerd 1.7 has Transfer service, full support is only available in 2.0+.


Suggested change

// While containerd 1.7 has Transfer service, full support is only available in 2.0+.

// While containerd 1.7 has Transfer service, full support is only available in 2.0+.

// The following features are missing in containerd 1.7:

// - [ADD MISSING FEATURES]

AkihiroSuda · 2025-11-28T07:23:54Z

+			snapshotter = "fuse-overlayfs"
+		[[plugins."io.containerd.transfer.v1.local".unpack_config]]
+			platform = "linux"
+			snapshotter = "overlayfs"


Not a fan of complicating the config.
What happens if these lines are missing?
ctr doesn't need them AFAICS?

By default, it would be

[[plugins."io.containerd.transfer.v1.local".unpack_config]] platform = "linux/amd64"(matches the host's OS/Architecture) snapshotter = "overlayfs" differ = "walking"

This config is required to specify which snapshotter the transfer service should use when unpacking images. We define multiple unpack targets:

overlayfs for standard OCI images

stargz / soci for lazy-pulling support

Without this, containerd defaults to basic unpacking and won't leverage stargz/soci snapshotters needed for lazy-pulling.

And if we only configure stargz in unpack_config, then all images would be forced to use the stargz snapshotter during transfer/unpacking, even standard OCI images that don't support lazy-pulling.

Can you add that as comment lines, and update the relevant docs?
What about soci, nydus, etc?

Add remote snapshot annotations and transfer unpack config for stargz, soci, and fuse-overlayfs snapshotter plugins. Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>

Add version detection to automatically select Transfer service (2.0+) or legacy resolver methods (< 2.0) for better compatibility. Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>

AkihiroSuda

Thanks

ChengyuZhu6 force-pushed the transfer branch 2 times, most recently from 285c273 to 156a92a Compare November 5, 2025 15:51

AkihiroSuda reviewed Nov 5, 2025

View reviewed changes

Comment thread cmd/nerdctl/image/image_import.go Outdated

AkihiroSuda added this to the v2.3.0 milestone Nov 6, 2025

ChengyuZhu6 force-pushed the transfer branch 21 times, most recently from b5584fe to 18b0448 Compare November 14, 2025 03:54

ChengyuZhu6 force-pushed the transfer branch 5 times, most recently from a49ea0a to f66db0d Compare November 23, 2025 14:43

ChengyuZhu6 force-pushed the transfer branch from d36732a to c5552d0 Compare November 27, 2025 07:01

ChengyuZhu6 marked this pull request as ready for review November 27, 2025 07:19

ChengyuZhu6 force-pushed the transfer branch from c5552d0 to 95e03e0 Compare November 27, 2025 07:20

ChengyuZhu6 requested a review from AkihiroSuda November 27, 2025 07:41

ChengyuZhu6 force-pushed the transfer branch 7 times, most recently from f2e210d to d3b9cab Compare November 27, 2025 10:59

ChengyuZhu6 force-pushed the transfer branch 2 times, most recently from d099f97 to 3341fbd Compare November 28, 2025 01:38

AkihiroSuda reviewed Nov 28, 2025

View reviewed changes

Comment thread pkg/infoutil/infoutil.go

AkihiroSuda requested review from dmcgowan and ktock November 28, 2025 07:07

AkihiroSuda reviewed Nov 28, 2025

View reviewed changes

ChengyuZhu6 force-pushed the transfer branch from 3341fbd to 4187a4d Compare December 3, 2025 03:06

ChengyuZhu6 added 2 commits December 3, 2025 11:08

configure containerd config with remote snapshotter

b4c6c8f

Add remote snapshot annotations and transfer unpack config for stargz, soci, and fuse-overlayfs snapshotter plugins. Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>

Support legacy push/pull for containerd 1.7.x

7d8ab7c

Add version detection to automatically select Transfer service (2.0+) or legacy resolver methods (< 2.0) for better compatibility. Signed-off-by: ChengyuZhu6 <hudson@cyzhu.com>

ChengyuZhu6 force-pushed the transfer branch from 4187a4d to 7d8ab7c Compare December 3, 2025 03:08

AkihiroSuda approved these changes Dec 22, 2025

View reviewed changes

AkihiroSuda merged commit d7b5e3c into containerd:main Dec 22, 2025
76 of 80 checks passed

ningmingxiao mentioned this pull request Jan 26, 2026

Refactor the image management code to use the Transfer service #4573

Closed

Conversation

ChengyuZhu6 commented Nov 5, 2025

Uh oh!

Uh oh!

ChengyuZhu6 commented Nov 25, 2025

Uh oh!

AkihiroSuda commented Nov 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChengyuZhu6 Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AkihiroSuda Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ChengyuZhu6 Nov 28, 2025 •

edited

Loading

AkihiroSuda Nov 28, 2025 •

edited

Loading