feat: Add S3 artifact repository configuration options for parallel transfers and resource limits

[usulkies]

feat: Add S3 artifact repository configuration options for parallel transfers and resource limits

Fixes #12442, #9022, #4014

Motivation

The current S3 artifact repository implementation has performance limitations that cause slow upload/download times:

1. Many small files: Sequential directory uploads/downloads are slow when dealing with hundreds or thousands of files
2. Large single files: Single-threaded transfers for large files (e.g., 50GB) take excessive time
3. No configurability: Users cannot tune parallelism settings for their specific use cases

This feature addresses these issues by adding configurable parallelism options that leverage MinIO's built-in multipart capabilities and implement worker pools for directory operations.

Modifications

1. Added Configuration Options to S3Bucket

Added five new optional fields to the S3Bucket type:

• enableParallelism: Enable/disable parallel operations (default: true)
• parallelism: Number of concurrent workers for parallel operations (default: 10)
• fileCountThreshold: Minimum number of files in a directory to trigger parallel operations (default: 10)
• fileSizeThreshold: Minimum file size to trigger multipart upload/download (default: "64Mi", supports Kubernetes resource quantity strings like "64Mi", "1Gi")
• partSize: Part size for multipart uploads (default: minio default, typically 128MB, supports Kubernetes resource quantity strings)

2. Implementation Details

Parallel Directory Operations

• PutDirectory: When enableParallelism is true and file count exceeds fileCountThreshold, uses a worker pool to upload files concurrently
• GetDirectory: When enableParallelism is true and file count exceeds fileCountThreshold, uses a worker pool to download files concurrently
• Worker pool size is controlled by the parallelism setting

Large File Multipart Operations

• PutFile: For files larger than fileSizeThreshold, configures MinIO's FPutObject with NumThreads and PartSize for parallel multipart uploads
• GetFile: For files larger than fileSizeThreshold, leverages MinIO's automatic multipart download capabilities
• Uses MinIO's built-in multipart upload/download with configurable thread count

3. Configuration Methods

The parallelism configuration can be set in two ways:

Via Artifact Repository (ConfigMap)

apiVersion: v1
kind: ConfigMap
metadata:
  name: artifact-repositories
data:
  default-v1: |
    archiveLogs: true
    s3:
      bucket: my-bucket
      endpoint: minio:9000
      insecure: true
      accessKeySecret:
        name: my-minio-cred
        key: accesskey
      secretKeySecret:
        name: my-minio-cred
        key: secretkey
      enableParallelism: true
      parallelism: 10
      fileCountThreshold: 10
      fileSizeThreshold: 64Mi
      partSize: 128Mi

Via Inline Artifact Properties

outputs:
  artifacts:
    - name: large-file
      path: /tmp/large-file
      s3:
        enableParallelism: true
        parallelism: 5
        fileSizeThreshold: 10Mi
        partSize: 32Mi

4. Code Changes

• pkg/apis/workflow/v1alpha1/workflow_types.go: Added new fields to S3Bucket struct
• workflow/artifacts/s3/s3.go:
  • Added parallelism fields to S3ClientOpts and s3client structs
  • Implemented putDirectoryParallel() and getDirectoryParallel() with worker pools
  • Enhanced PutFile() and GetFile() to configure MinIO multipart options for large files
• workflow/artifacts/artifacts.go: Pass parallelism configuration from artifact repository to S3 client
• Generated files: Updated protobuf, OpenAPI, CRDs, and documentation

Verification

Unit Tests

• Added comprehensive tests in workflow/artifacts/s3/s3_parallel_test.go
• Tested parallel directory upload/download with various file counts
• Tested configuration parsing and defaults
• All existing tests pass

Integration Testing

• Successfully submitted 30+ example workflows from the examples/ directory
• All workflows accepted by API with no validation errors
• Verified backward compatibility (existing workflows continue to work)

Performance Testing

• Tested with workflows containing many files (20+ files) - parallel upload works
• Tested with large single files (100MB+) - multipart upload with parallelism works
• Verified configuration is properly applied from both artifact repository and inline settings

Documentation

• Updated docs/fields.md with new S3Bucket configuration options
• Created feature documentation in .features/pending/s3-parallelism-config.md
• Updated OpenAPI schema and JSON schema
• Updated CRD manifests

Breaking Changes

None. All new fields are optional with sensible defaults, ensuring backward compatibility.

Example Use Cases

1. CI/CD pipelines with many artifacts: Upload hundreds of test result files in parallel
2. Large dataset transfers: Transfer 50GB+ files using multipart uploads with multiple threads
3. High-throughput workflows: Configure higher parallelism for workflows that frequently transfer artifacts

[Joibel]

Just some initial thoughts on this PR. Thanks for working on it.

[antoinetran]

Hi! Would be very interested in this PR, because we have a factor 10 of speed difference, when we save artifact of 35GiB, it takes 2m with awscli and 23m with argo. The options here are great!
Any issue that block merging to main?

[usulkies]

Hi! Would be very interested in this PR, because we have a factor 10 of speed difference, when we save artifact of 35GiB, it takes 2m with awscli and 23m with argo. The options here are great! Any issue that block merging to main?

Well, yes...
This didn't work fully. Need to find some time to try it again.