OADP-4640: Downstream only to allow override kopia default algorithms #334

mpryc · 2024-08-02T13:12:25Z

Introduction of downstream only option to override Kopia default:

hashing algorithm
splitting algorithm
encryption algorithm

With introduction of 3 environment variables it is possible to override Kopia algorithms used by Velero:

KOPIA_HASHING_ALGORITHM
KOPIA_SPLITTER_ALGORITHM
KOPIA_ENCRYPTION_ALGORITHM

If the env algorithms are not set or they are not within Kopia SupportedAlgorithms, the default algorithm will be used. This behavior is consistent with current behavior without this change.

Does your change fix a particular issue?

Fixes #OADP-4640

The change introduces new test file to easy rebases. Also the changes to the original file are minimal and split into one function to reduce risk of further rebase headaches.

Testing

$ pkg/repository/udmrepo/kopialib/backend/
$ go test -v .

openshift-ci-robot · 2024-08-02T13:12:30Z

@mpryc: This pull request references OADP-4640 which is a valid jira issue.

Details

In response to this:

Introduction of downstream only option to override Kopia default:

hashing algorithm

splitting algorithm

encryption algorithm

With introduction of 3 environment variables it is possible to override Kopia algorithms used by Velero:

KOPIA_HASHING_ALGORITHM
KOPIA_SPLITTER_ALGORITHM
KOPIA_ENCRYPTION_ALGORITHM

If the env algorithms are not set or they are not within Kopia SupportedAlgorithms, the default algorithm will be used. This behavior is consistent with current behavior without this change.

Does your change fix a particular issue?

Fixes #OADP-4640

The change introduces new test file to easy rebases. Also the changes to the original file are minimal and split into one function to reduce risk of further rebase headaches.

Testing
$ pkg/repository/udmrepo/kopialib/backend/
$ go test -v .

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2024-08-02T13:12:34Z

@mpryc: This pull request references OADP-4640 which is a valid jira issue.

Details

In response to this:

Introduction of downstream only option to override Kopia default:

hashing algorithm

splitting algorithm

encryption algorithm

With introduction of 3 environment variables it is possible to override Kopia algorithms used by Velero:

KOPIA_HASHING_ALGORITHM
KOPIA_SPLITTER_ALGORITHM
KOPIA_ENCRYPTION_ALGORITHM

If the env algorithms are not set or they are not within Kopia SupportedAlgorithms, the default algorithm will be used. This behavior is consistent with current behavior without this change.

Does your change fix a particular issue?

Fixes #OADP-4640

The change introduces new test file to easy rebases. Also the changes to the original file are minimal and split into one function to reduce risk of further rebase headaches.

Testing
$ pkg/repository/udmrepo/kopialib/backend/
$ go test -v .

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

pkg/repository/udmrepo/kopialib/backend/common.go

pkg/repository/udmrepo/kopialib/backend/common_kopia_algorithms_test.go

mateusoliveira43 · 2024-08-08T14:24:00Z

pkg/repository/udmrepo/kopialib/backend/common_kopia_algorithms_test.go

+			},
+		},
+		{
+			name:    "with hash algo in flags, env var ignored",


this test is not doing what is expected, no?

value from env var appears in expected

Yes but I wanted to test that udmrepo does not change the algorhtm note the SHA3-256 is different then SHA3-224 so different ones are set with env and different with udmrepo.

I didn't change this code as I think it's correct.

the test name makes no sense, the environment variable is not being ignored

Introduction of downstream only option to override Kopia default: - hashing algorithm - splitting algorithm - encryption algorithm With introduction of 3 environment variables it is possible to override Kopia algorithms used by Velero: KOPIA_HASHING_ALGORITHM KOPIA_SPLITTER_ALGORITHM KOPIA_ENCRYPTION_ALGORITHM If the env algorithms are not set or they are not within Kopia SupportedAlgorithms, the default algorithm will be used. This behavior is consistent with current behavior without this change. Signed-off-by: Michal Pryc <[email protected]>

openshift-ci · 2024-08-09T20:46:55Z

@mpryc: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

weshayutin · 2024-08-09T20:57:17Z

From https://github.com/openshift/velero
 * [new ref]             refs/pull/334/head -> issue-oadp-4640
Switched to branch 'issue-oadp-4640'
whayutin@thinkdoe:~/OPENSHIFT/git/OADP/velero$ cd pkg/repository/udmrepo/kopialib/backend/
whayutin@thinkdoe:~/OPENSHIFT/git/OADP/velero/pkg/repository/udmrepo/kopialib/backend$ ls
azure          backend.go                       common_test.go       gcs.go       s3.go       utils_test.go
azure.go       common.go                        file_system.go       gcs_test.go  s3_test.go
azure_test.go  common_kopia_algorithms_test.go  file_system_test.go  mocks        utils.go
whayutin@thinkdoe:~/OPENSHIFT/git/OADP/velero/pkg/repository/udmrepo/kopialib/backend$ go test -v .
=== RUN   TestAzureSetup
--- PASS: TestAzureSetup (0.00s)
=== RUN   TestSetupNewRepoAlgorithms
=== RUN   TestSetupNewRepoAlgorithms/with_valid_non-default_hash_algo_from_env
=== RUN   TestSetupNewRepoAlgorithms/with_valid_non-default_encryption_algo_from_env
=== RUN   TestSetupNewRepoAlgorithms/with_valid_non-default_splitter_algo_from_env
=== RUN   TestSetupNewRepoAlgorithms/with_valid_non-default_splitter_and_hashing_algo_from_env,_invalid_encryption_from_env
=== RUN   TestSetupNewRepoAlgorithms/with_unsupported_hash_algo_in_env,_fallback_to_default
=== RUN   TestSetupNewRepoAlgorithms/hash_in_StoreOptionGenHashAlgo_and_env,_env_wins
--- PASS: TestSetupNewRepoAlgorithms (0.00s)
    --- PASS: TestSetupNewRepoAlgorithms/with_valid_non-default_hash_algo_from_env (0.00s)
    --- PASS: TestSetupNewRepoAlgorithms/with_valid_non-default_encryption_algo_from_env (0.00s)
    --- PASS: TestSetupNewRepoAlgorithms/with_valid_non-default_splitter_algo_from_env (0.00s)
    --- PASS: TestSetupNewRepoAlgorithms/with_valid_non-default_splitter_and_hashing_algo_from_env,_invalid_encryption_from_env (0.00s)
    --- PASS: TestSetupNewRepoAlgorithms/with_unsupported_hash_algo_in_env,_fallback_to_default (0.00s)
    --- PASS: TestSetupNewRepoAlgorithms/hash_in_StoreOptionGenHashAlgo_and_env,_env_wins (0.00s)
=== RUN   TestSetupNewRepositoryOptions
=== RUN   TestSetupNewRepositoryOptions/with_hash_algo
=== RUN   TestSetupNewRepositoryOptions/with_encrypt_algo
=== RUN   TestSetupNewRepositoryOptions/with_splitter_algo
=== RUN   TestSetupNewRepositoryOptions/with_retention_algo
--- PASS: TestSetupNewRepositoryOptions (0.00s)
    --- PASS: TestSetupNewRepositoryOptions/with_hash_algo (0.00s)
    --- PASS: TestSetupNewRepositoryOptions/with_encrypt_algo (0.00s)
    --- PASS: TestSetupNewRepositoryOptions/with_splitter_algo (0.00s)
    --- PASS: TestSetupNewRepositoryOptions/with_retention_algo (0.00s)
=== RUN   TestSetupConnectOptions
=== RUN   TestSetupConnectOptions/with_domain
=== RUN   TestSetupConnectOptions/with_username
=== RUN   TestSetupConnectOptions/with_wrong_readonly
=== RUN   TestSetupConnectOptions/with_correct_readonly
=== RUN   TestSetupConnectOptions/with_description
--- PASS: TestSetupConnectOptions (0.00s)
    --- PASS: TestSetupConnectOptions/with_domain (0.00s)
    --- PASS: TestSetupConnectOptions/with_username (0.00s)
    --- PASS: TestSetupConnectOptions/with_wrong_readonly (0.00s)
    --- PASS: TestSetupConnectOptions/with_correct_readonly (0.00s)
    --- PASS: TestSetupConnectOptions/with_description (0.00s)
=== RUN   TestFSSetup
=== RUN   TestFSSetup/must_have_fs_path
=== RUN   TestFSSetup/with_fs_path_only
=== RUN   TestFSSetup/with_prefix
--- PASS: TestFSSetup (0.00s)
    --- PASS: TestFSSetup/must_have_fs_path (0.00s)
    --- PASS: TestFSSetup/with_fs_path_only (0.00s)
    --- PASS: TestFSSetup/with_prefix (0.00s)
=== RUN   TestGcsSetup
=== RUN   TestGcsSetup/must_have_bucket_name
=== RUN   TestGcsSetup/must_have_credential_file
=== RUN   TestGcsSetup/with_prefix
=== RUN   TestGcsSetup/with_wrong_readonly
=== RUN   TestGcsSetup/with_correct_readonly
--- PASS: TestGcsSetup (0.00s)
    --- PASS: TestGcsSetup/must_have_bucket_name (0.00s)
    --- PASS: TestGcsSetup/must_have_credential_file (0.00s)
    --- PASS: TestGcsSetup/with_prefix (0.00s)
    --- PASS: TestGcsSetup/with_wrong_readonly (0.00s)
    --- PASS: TestGcsSetup/with_correct_readonly (0.00s)
=== RUN   TestS3Setup
=== RUN   TestS3Setup/must_have_bucket_name
=== RUN   TestS3Setup/with_bucket_only
=== RUN   TestS3Setup/with_others
=== RUN   TestS3Setup/with_wrong_tls
=== RUN   TestS3Setup/with_correct_tls
=== RUN   TestS3Setup/with_wrong_ca
=== RUN   TestS3Setup/with_correct_ca
--- PASS: TestS3Setup (0.00s)
    --- PASS: TestS3Setup/must_have_bucket_name (0.00s)
    --- PASS: TestS3Setup/with_bucket_only (0.00s)
    --- PASS: TestS3Setup/with_others (0.00s)
    --- PASS: TestS3Setup/with_wrong_tls (0.00s)
    --- PASS: TestS3Setup/with_correct_tls (0.00s)
    --- PASS: TestS3Setup/with_wrong_ca (0.00s)
    --- PASS: TestS3Setup/with_correct_ca (0.00s)
=== RUN   TestOptionalHaveBool
=== RUN   TestOptionalHaveBool/key_not_exist
=== RUN   TestOptionalHaveBool/value_valid
=== RUN   TestOptionalHaveBool/value_invalid
--- PASS: TestOptionalHaveBool (0.00s)
    --- PASS: TestOptionalHaveBool/key_not_exist (0.00s)
    --- PASS: TestOptionalHaveBool/value_valid (0.00s)
    --- PASS: TestOptionalHaveBool/value_invalid (0.00s)
PASS
ok  	github.com/vmware-tanzu/velero/pkg/repository/udmrepo/kopialib/backend	0.008s

weshayutin

/LGTM

shubham-pampattiwar · 2024-08-13T06:26:53Z

pkg/repository/udmrepo/kopialib/backend/common_kopia_algorithms_test.go

+				},
+			},
+		},
+		{


@mpryc IMO we should add another testcase when StoreOptionGenHashAlgos has a valid algo value but the env var has an invalid algo value. Then in such a scenario default would not be used but the one present in StoreOptionGenHashAlgos would be used, right ?

@shubham-pampattiwar right. there are 2 levels of fallback here:

use env var if it's set and value is supported, if not, fall back to...

value in StoreOptionGenHashAlgo, if not fall back to...

default

shubham-pampattiwar · 2024-08-13T15:18:13Z

/approved

openshift-ci · 2024-08-13T15:20:40Z

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: mpryc, shubham-pampattiwar, sseago, weshayutin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

DOWNSTREAM_OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…openshift#334) add missing unit test for kopia hashing algo (openshift#337) Introduction of downstream only option to override Kopia default: - hashing algorithm - splitting algorithm - encryption algorithm With introduction of 3 environment variables it is possible to override Kopia algorithms used by Velero: KOPIA_HASHING_ALGORITHM KOPIA_SPLITTER_ALGORITHM KOPIA_ENCRYPTION_ALGORITHM If the env algorithms are not set or they are not within Kopia SupportedAlgorithms, the default algorithm will be used. This behavior is consistent with current behavior without this change. Signed-off-by: Michal Pryc <[email protected]> Signed-off-by: Shubham Pampattiwar <[email protected]>

…#334) (#338) add missing unit test for kopia hashing algo (#337) Introduction of downstream only option to override Kopia default: - hashing algorithm - splitting algorithm - encryption algorithm With introduction of 3 environment variables it is possible to override Kopia algorithms used by Velero: KOPIA_HASHING_ALGORITHM KOPIA_SPLITTER_ALGORITHM KOPIA_ENCRYPTION_ALGORITHM If the env algorithms are not set or they are not within Kopia SupportedAlgorithms, the default algorithm will be used. This behavior is consistent with current behavior without this change. Signed-off-by: Michal Pryc <[email protected]> Signed-off-by: Shubham Pampattiwar <[email protected]>

…openshift#334) Introduction of downstream only option to override Kopia default: - hashing algorithm - splitting algorithm - encryption algorithm With introduction of 3 environment variables it is possible to override Kopia algorithms used by Velero: KOPIA_HASHING_ALGORITHM KOPIA_SPLITTER_ALGORITHM KOPIA_ENCRYPTION_ALGORITHM If the env algorithms are not set or they are not within Kopia SupportedAlgorithms, the default algorithm will be used. This behavior is consistent with current behavior without this change. Signed-off-by: Michal Pryc <[email protected]>

…#334) (#338) add missing unit test for kopia hashing algo (#337) Introduction of downstream only option to override Kopia default: - hashing algorithm - splitting algorithm - encryption algorithm With introduction of 3 environment variables it is possible to override Kopia algorithms used by Velero: KOPIA_HASHING_ALGORITHM KOPIA_SPLITTER_ALGORITHM KOPIA_ENCRYPTION_ALGORITHM If the env algorithms are not set or they are not within Kopia SupportedAlgorithms, the default algorithm will be used. This behavior is consistent with current behavior without this change. Signed-off-by: Michal Pryc <[email protected]> Signed-off-by: Shubham Pampattiwar <[email protected]>