refactor: Change how we pass backend state around when creating a plan file

[SarahFrench]

Background

All commands that use a backend (or state store) to access state will use the Meta struct's backend (and via that, Backend) method to obtain a configured backend for use in subsequent command logic. During that logic, Terraform also checks that the backend configuration is unchanged since the last init. If the configuration and backend state file do not match then that logic will return an error telling users to run init.

Downstream logic within individual commands may want to use data in the backend state file. Instead of re-reading that data from disk the data is currently stored in the Meta, in-memory. This happens in backendFromConfig, which is the method where Terraform compares backend state file contents to the configuration. Once backendFromConfig has returned we're either in an error state (in which case the in-memory copy of the backend state file isn't used) or we know that the backend state file is up-to-date and can be used as a source of truth.

Here is how the data is stored in-memory on the Meta:

terraform/internal/command/meta_backend.go

Lines 728 to 735 in c9d0240

	// Upon return, we want to set the state we're using in-memory so that
	// we can access it for commands.
	m.backendState = nil
	defer func() {
	if s := sMgr.State(); s != nil && !s.Backend.Empty() {
	m.backendState = s.Backend
	}
	}()

Alternative approaches

Re-read the backend state file from disk when needed

Instead of storing the data in-memory we could just have the downstream logic read the backend state file again. This is implemented in 8b9be31.

This approach isn't ideal because although this data is only needed by the plan command, ALL commands that use Operations will run the altered logic. This means every plan/apply/query/refresh/state show/console command would have an extra step of reading from a file on disk, despite that data only being required for plan operations. This has an unnecessary impact on performance.

Store the backend state file contents in the Meta, not just subsets of the data

Originally the Meta would store only the description of the Backend kept in the backend state file. After #37956 the Meta has two mutually exclusive fields to store either backend or state store config state.

The changes in 676b2a6 propose storing the entire backend state file in the Meta. This means interpretation of the data, including how to handle the lack of backend state, is the responsibility of the logic that's specific to the plan command. Only the plan command will need special logic interpreting the lack of backend state file data as an indication of using the implicit local backend.

By the time the plan command's own specific logic is executed we know that Terraform has determined that the working directory has been initialised and the configuration is unchanged. Therefore the plan command can confidently interpret the lack of a backend state file as meaning an implicit local backend is in use.

In this approach we avoid re-reading from the backend state file from disk, so there isn't any impact due to increased I/O.

Target Release

1.15.x

Rollback Plan

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

CHANGELOG entry

• This change is user-facing and I added a changelog entry.
• This change is not user-facing.

[github-actions]

The equivalence tests failed. Please investigate here.

[SarahFrench]

@radeksimko - Tagging you here as we can potentially discuss these changes in relation to this thread #37956 (comment)

[SarahFrench]

After discussion we agreed that a better solution would be to tackle the behemoth that is the Meta struct. It'd be nicer to have all backend-related logic scoped inside a struct that isn't the Meta but is still embedded within the different Command structs in the code base.

Due to that, and how this PR is "rearranging a mess into a sightlier prettier mess" 😆 , I'm closing this PR.