Skip to content

Conversation

@kvaps
Copy link

@kvaps kvaps commented Oct 20, 2025

The throttling parameters specified in BackupRepository's repositoryConfig were being silently filtered out and not applied to Kopia repositories. This occurred because these parameters were missing from the whitelist in getStorageVariables().

Code:

  • Added throttling options to the valid parameters whitelist in pkg/repository/provider/unified_repo.go:

    • ThrottleUploadBytes - upload bandwidth throttling (bytes/sec)
    • ThrottleDownloadBytes - download bandwidth throttling (bytes/sec)
    • ThrottleReadOPS - read operations throttling (ops/sec)
    • ThrottleWriteOPS - write operations throttling (ops/sec)
    • ThrottleListOPS - list operations throttling (ops/sec)
  • Updated parameter names in pkg/repository/udmrepo/repo_options.go to use "Throttle" prefix for better clarity:

    • readOPSThrottleReadOPS
    • writeOPSThrottleWriteOPS
    • listOPSThrottleListOPS
    • uploadBytesThrottleUploadBytes
    • downloadBytesThrottleDownloadBytes

Documentation:

  • Updated site/content/docs/main/backup-repository-configuration.md with throttling parameters descriptions and examples using the new parameter names

Tests:

  • Added test case in pkg/repository/provider/unified_repo_test.go to verify throttling parameters are properly passed through
  • Updated test expectations to use the new parameter names

Example Usage:

apiVersion: velero.io/v1
kind: BackupRepository
spec:
  repositoryConfig:
    ThrottleUploadBytes: "838860800"    # ~800 MB/s
    ThrottleDownloadBytes: "838860800"  # ~800 MB/s
    ThrottleReadOPS: "100"
    ThrottleWriteOPS: "100"
    ThrottleListOPS: "50"

These parameters are now properly applied to Kopia during file system backups performed by node-agent.

Signed-off-by: Andrei Kvapil [email protected]

@kvaps kvaps force-pushed the throttling branch 2 times, most recently from 469a1ec to ff9e49c Compare October 20, 2025 18:20
@kvaps kvaps marked this pull request as ready for review October 21, 2025 15:50
kaovilai
kaovilai previously approved these changes Oct 22, 2025
@codecov
Copy link

codecov bot commented Oct 22, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.24%. Comparing base (99f12b8) to head (5e643a4).
⚠️ Report is 171 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #9351      +/-   ##
==========================================
+ Coverage   59.67%   60.24%   +0.57%     
==========================================
  Files         383      386       +3     
  Lines       44050    35942    -8108     
==========================================
- Hits        26286    21653    -4633     
+ Misses      16218    12715    -3503     
- Partials     1546     1574      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kvaps kvaps changed the title Fix: Enable throttling parameters in BackupRepository repositoryConfig Fix race condition in data upload controller concurrent limit check Dec 10, 2025
@kvaps kvaps changed the title Fix race condition in data upload controller concurrent limit check Fix: Enable throttling parameters in BackupRepository repositoryConfig Dec 10, 2025
@kvaps kvaps force-pushed the throttling branch 2 times, most recently from ff9e49c to 5e643a4 Compare December 10, 2025 19:12
The `uploadBytes`, `downloadBytes`, and other throttling parameters specified in BackupRepository's `repositoryConfig` were being silently filtered out and not applied to Kopia repositories. This occurred because these parameters were missing from the whitelist in `getStorageVariables()`.

**Code:**
- Added throttling options to the valid parameters whitelist in `pkg/repository/provider/unified_repo.go`:
  - `uploadBytes` - upload bandwidth throttling (bytes/sec)
  - `downloadBytes` - download bandwidth throttling (bytes/sec)
  - `readOPS` - read operations throttling (ops/sec)
  - `writeOPS` - write operations throttling (ops/sec)
  - `listOPS` - list operations throttling (ops/sec)

**Documentation:**
- Updated `site/content/docs/main/backup-repository-configuration.md` with throttling parameters descriptions and examples

**Tests:**
- Added test case in `pkg/repository/provider/unified_repo_test.go` to verify throttling parameters are properly passed through

```yaml
apiVersion: velero.io/v1
kind: BackupRepository
spec:
  repositoryConfig:
    uploadBytes: "838860800"    # ~800 MB/s
    downloadBytes: "838860800"  # ~800 MB/s
```

These parameters are now properly applied to Kopia during file system backups performed by node-agent.

Signed-off-by: Andrei Kvapil <[email protected]>
ThrottleOptionListOps = "listOPS"
ThrottleOptionUploadBytes = "uploadBytes"
ThrottleOptionDownloadBytes = "downloadBytes"
ThrottleOptionReadOps = "ThrottleReadOPS"
Copy link
Contributor

@Lyndon-Li Lyndon-Li Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for misleading of my previous comment. We add throttle for each of the options, but as the convention, we need to make the first word lowercase, like:

    ThrottleOptionReadOps       = "throttleReadOPS"
	ThrottleOptionWriteOps      = "throttleWriteOPS"
	ThrottleOptionListOps       = "throttleListOPS"
	ThrottleOptionUploadBytes   = "throttleUploadBytes"
	ThrottleOptionDownloadBytes = "throttleDownloadBytes"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants