backend/azure: Remove legacy helper/schema dependency [rebased]

[SarahFrench]

A rebased* version of #34990 :

As part of our efforts to remove the large snapshot of the legacy SDK that we've been using, this replaces the uses of that package with our new lightweight backendbase package, which is a collection of helpers that can substitute for most of the parts of the legacy SDK that backends tend to use.

In the long run we're still planning to move all of the state storage backends out to provider plugins, at which point they'll presumably use the modern plugin framework, but in the meantime the legacy SDK is quite a liability because few people know how to maintain it while hopefully this interim backendbase package is easier to maintain (if needed) due to its relative simplicity.

I don't have any test environment for this backend, so I've only tested this through its unit tests. I'd appreciate if a backend maintainer could, along with reviewing this code, also run the acceptance test suite in case there are regressions that the unit tests aren't able to catch. Thanks!

*= I noticed that the schema had changed since the original PR so my approach was to re-do that work

NOTE: This PR changed metadata_host in the schema from Required to Optional. The legacy SDK allowed the required field to be set using a fallback value, so users never saw required behaviour. Moving off the legacy SDK made the field to begin behaving as required, which isn't correct. Therefore we swapped the schema to mark it as Optional, and this isn't a user-facing change as prior to this PR the field behaved as optional already.

Target Release

1.13.x

Rollback Plan

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

CHANGELOG entry

• This change is user-facing and I added a changelog entry.
• This change is not user-facing.

[SarahFrench]

@hashicorp/terraform-azure, could someone with Azure credentials run the acceptance tests for the azure backend to verify the changes? Thanks!

[SarahFrench]

Interesting - TestBackendConfig fails with The attribute "metadata_host" is required, but no definition was found.

I believe this is because PrepareConfig in the old code would use the empty string default when the field isn't set. This would get around the required behaviour. The new code doesn't seem to accept a default value for a required field like the old code did. Arguably, that's a more correct behaviour.

I think the test needs to be updated here, but I defer to the Azure team.

[SarahFrench]

From a manual test, it seems like metadata_host isn't behaving as a required field despite being marked as required:

It may be that metadata_host needs to be marked as Optional, as that's how it is currently behaving. Otherwise merging this PR would release a breaking change to users.

[jackofallops]

Thanks @SarahFrench - LGTM 👍

[SarahFrench]

Hi @jackofallops - I'm only able to run the TestBackendConfig test and it's currently failing because removing the legacy SDK has forced metadata_host to begin actually behaving like a Required field. We need to decide if it is indeed a Required field and the test should be updated, or whether the field should be Optional (as that's how it's behaving for users today).

[jackofallops]

Hi @jackofallops - I'm only able to run the TestBackendConfig test and it's currently failing because removing the legacy SDK has forced metadata_host to begin actually behaving like a Required field. We need to decide if it is indeed a Required field and the test should be updated, or whether the field should be Optional (as that's how it's behaving for users today).

To the best of my knowledge it should be an optional field, it's an override for non-standard environments, so Optional I believe to be correct there. Not sure how/why it would have been flagged as Required previously.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.