-
Notifications
You must be signed in to change notification settings - Fork 2.2k
ADAP-814: Refactor prep for MV updates #8459
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 10 commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
f0b07d0
first draft of adding in table - materialized view swap
mikealfare 24d9c42
Merge branch 'main' into feature/materialized-views/adap-814
mikealfare 7372440
tests pass
mikealfare d20f7dc
table/view/materialized view can all replace each other
mikealfare 28e1612
changie
mikealfare 4483f6c
update renameable relations to a config
mikealfare 03a15e7
revert to just reformatting changes
mikealfare d755c51
revert to just reformatting changes
mikealfare d898469
revert to just reformatting changes
mikealfare 46e8aef
revert to just reformatting changes
mikealfare cefc7d4
add logging back to get_create_materialized_view_as_sql
mikealfare 49d0237
changie
mikealfare File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
44 changes: 0 additions & 44 deletions
44
core/dbt/include/global_project/macros/adapters/drop_relation.sql
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 5 additions & 0 deletions
5
...alized_view/replace_materialized_view.sql → .../relations/materialized_view/_replace.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
5 changes: 3 additions & 2 deletions
5
...ialized_view/create_materialized_view.sql → ...rs/relations/materialized_view/create.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,9 +1,10 @@ | ||
| {% macro get_create_materialized_view_as_sql(relation, sql) -%} | ||
| {{- log('Applying CREATE to: ' ~ relation) -}} | ||
| {{- adapter.dispatch('get_create_materialized_view_as_sql', 'dbt')(relation, sql) -}} | ||
| {%- endmacro %} | ||
|
|
||
|
|
||
| {% macro default__get_create_materialized_view_as_sql(relation, sql) -%} | ||
| {{ exceptions.raise_compiler_error("Materialized views have not been implemented for this adapter.") }} | ||
| {{ exceptions.raise_compiler_error( | ||
| "`get_create_materialized_view_as_sql` has not been implemented for this adapter." | ||
| ) }} | ||
| {% endmacro %} | ||
13 changes: 13 additions & 0 deletions
13
core/dbt/include/global_project/macros/adapters/relations/materialized_view/drop.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| {# /* | ||
| This was already implemented. Instead of creating a new macro that aligns with the standard, | ||
| this was reused and the default was maintained. This gets called by `drop_relation`, which | ||
| actually executes the drop, and `get_drop_sql`, which returns the template. | ||
| */ #} | ||
|
|
||
| {% macro drop_materialized_view(relation) -%} | ||
| {{ return(adapter.dispatch('drop_materialized_view', 'dbt')(relation)) }} | ||
| {%- endmacro %} | ||
|
|
||
| {% macro default__drop_materialized_view(relation) -%} | ||
| drop materialized view if exists {{ relation }} cascade | ||
| {%- endmacro %} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
13 changes: 13 additions & 0 deletions
13
core/dbt/include/global_project/macros/adapters/relations/table/drop.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| {# /* | ||
| This was already implemented. Instead of creating a new macro that aligns with the standard, | ||
| this was reused and the default was maintained. This gets called by `drop_relation`, which | ||
| actually executes the drop, and `get_drop_sql`, which returns the template. | ||
| */ #} | ||
|
|
||
| {% macro drop_table(relation) -%} | ||
| {{ return(adapter.dispatch('drop_table', 'dbt')(relation)) }} | ||
| {%- endmacro %} | ||
|
|
||
| {% macro default__drop_table(relation) -%} | ||
| drop table if exists {{ relation }} cascade | ||
| {%- endmacro %} |
13 changes: 13 additions & 0 deletions
13
core/dbt/include/global_project/macros/adapters/relations/view/drop.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| {# /* | ||
| This was already implemented. Instead of creating a new macro that aligns with the standard, | ||
| this was reused and the default was maintained. This gets called by `drop_relation`, which | ||
| actually executes the drop, and `get_drop_sql`, which returns the template. | ||
| */ #} | ||
|
|
||
| {% macro drop_view(relation) -%} | ||
| {{ return(adapter.dispatch('drop_view', 'dbt')(relation)) }} | ||
| {%- endmacro %} | ||
|
|
||
| {% macro default__drop_view(relation) -%} | ||
| drop view if exists {{ relation }} cascade | ||
| {%- endmacro %} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| {% macro drop_relation(relation) -%} | ||
| {{ return(adapter.dispatch('drop_relation', 'dbt')(relation)) }} | ||
| {% endmacro %} | ||
|
|
||
| {% macro default__drop_relation(relation) -%} | ||
| {% call statement('drop_relation', auto_begin=False) -%} | ||
| {%- if relation.is_table -%} | ||
| {{- drop_table(relation) -}} | ||
| {%- elif relation.is_view -%} | ||
| {{- drop_view(relation) -}} | ||
| {%- elif relation.is_materialized_view -%} | ||
| {{- drop_materialized_view(relation) -}} | ||
| {%- else -%} | ||
| drop {{ relation.type }} if exists {{ relation }} cascade | ||
| {%- endif -%} | ||
| {%- endcall %} | ||
| {% endmacro %} |
10 changes: 10 additions & 0 deletions
10
core/dbt/include/global_project/macros/relations/rename.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| {% macro rename_relation(from_relation, to_relation) -%} | ||
| {{ return(adapter.dispatch('rename_relation', 'dbt')(from_relation, to_relation)) }} | ||
| {% endmacro %} | ||
|
|
||
| {% macro default__rename_relation(from_relation, to_relation) -%} | ||
| {% set target_name = adapter.quote_as_configured(to_relation.identifier, 'identifier') %} | ||
| {% call statement('rename_relation') -%} | ||
| alter table {{ from_relation }} rename to {{ target_name }} | ||
| {%- endcall %} | ||
| {% endmacro %} |
79 changes: 0 additions & 79 deletions
79
plugins/postgres/dbt/include/postgres/macros/materializations/materialized_view.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,84 +1,5 @@ | ||
| {% macro postgres__get_alter_materialized_view_as_sql( | ||
| relation, | ||
| configuration_changes, | ||
| sql, | ||
| existing_relation, | ||
| backup_relation, | ||
| intermediate_relation | ||
| ) %} | ||
|
|
||
| -- apply a full refresh immediately if needed | ||
| {% if configuration_changes.requires_full_refresh %} | ||
|
|
||
| {{ get_replace_materialized_view_as_sql(relation, sql, existing_relation, backup_relation, intermediate_relation) }} | ||
|
|
||
| -- otherwise apply individual changes as needed | ||
| {% else %} | ||
|
|
||
| {{ postgres__update_indexes_on_materialized_view(relation, configuration_changes.indexes) }} | ||
|
|
||
| {%- endif -%} | ||
|
|
||
| {% endmacro %} | ||
|
|
||
|
|
||
| {% macro postgres__get_create_materialized_view_as_sql(relation, sql) %} | ||
| create materialized view if not exists {{ relation }} as {{ sql }}; | ||
|
|
||
| {% for _index_dict in config.get('indexes', []) -%} | ||
| {{- get_create_index_sql(relation, _index_dict) -}} | ||
| {%- endfor -%} | ||
|
|
||
| {% endmacro %} | ||
|
|
||
|
|
||
| {% macro postgres__get_replace_materialized_view_as_sql(relation, sql, existing_relation, backup_relation, intermediate_relation) %} | ||
| {{- get_create_materialized_view_as_sql(intermediate_relation, sql) -}} | ||
|
|
||
| {% if existing_relation is not none %} | ||
| alter materialized view {{ existing_relation }} rename to {{ backup_relation.include(database=False, schema=False) }}; | ||
| {% endif %} | ||
|
|
||
| alter materialized view {{ intermediate_relation }} rename to {{ relation.include(database=False, schema=False) }}; | ||
|
|
||
| {% endmacro %} | ||
|
|
||
|
|
||
| {% macro postgres__get_materialized_view_configuration_changes(existing_relation, new_config) %} | ||
| {% set _existing_materialized_view = postgres__describe_materialized_view(existing_relation) %} | ||
| {% set _configuration_changes = existing_relation.get_materialized_view_config_change_collection(_existing_materialized_view, new_config) %} | ||
| {% do return(_configuration_changes) %} | ||
| {% endmacro %} | ||
|
|
||
|
|
||
| {% macro postgres__refresh_materialized_view(relation) %} | ||
| refresh materialized view {{ relation }} | ||
| {% endmacro %} | ||
|
|
||
|
|
||
| {%- macro postgres__update_indexes_on_materialized_view(relation, index_changes) -%} | ||
| {{- log("Applying UPDATE INDEXES to: " ~ relation) -}} | ||
|
|
||
| {%- for _index_change in index_changes -%} | ||
| {%- set _index = _index_change.context -%} | ||
|
|
||
| {%- if _index_change.action == "drop" -%} | ||
|
|
||
| {{ postgres__get_drop_index_sql(relation, _index.name) }}; | ||
|
|
||
| {%- elif _index_change.action == "create" -%} | ||
|
|
||
| {{ postgres__get_create_index_sql(relation, _index.as_node_config) }} | ||
|
|
||
| {%- endif -%} | ||
|
|
||
| {%- endfor -%} | ||
|
|
||
| {%- endmacro -%} | ||
|
|
||
|
|
||
| {% macro postgres__describe_materialized_view(relation) %} | ||
| -- for now just get the indexes, we don't need the name or the query yet | ||
| {% set _indexes = run_query(get_show_indexes_sql(relation)) %} | ||
| {% do return({'indexes': _indexes}) %} | ||
| {% endmacro %} |
16 changes: 16 additions & 0 deletions
16
plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/_replace.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| {# /* | ||
| This only exists for backwards compatibility for 1.6.0. In later versions, the general `get_replace_sql` | ||
| macro is called as replace is inherently not limited to a single relation (it takes in two relations). | ||
| */ #} | ||
|
|
||
|
|
||
| {% macro postgres__get_replace_materialized_view_as_sql(relation, sql, existing_relation, backup_relation, intermediate_relation) %} | ||
| {{- get_create_materialized_view_as_sql(intermediate_relation, sql) -}} | ||
|
|
||
| {% if existing_relation is not none %} | ||
| alter materialized view {{ existing_relation }} rename to {{ backup_relation.include(database=False, schema=False) }}; | ||
| {% endif %} | ||
|
|
||
| alter materialized view {{ intermediate_relation }} rename to {{ relation.include(database=False, schema=False) }}; | ||
|
|
||
| {% endmacro %} |
43 changes: 43 additions & 0 deletions
43
plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/alter.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| {% macro postgres__get_alter_materialized_view_as_sql( | ||
| relation, | ||
| configuration_changes, | ||
| sql, | ||
| existing_relation, | ||
| backup_relation, | ||
| intermediate_relation | ||
| ) %} | ||
|
|
||
| -- apply a full refresh immediately if needed | ||
| {% if configuration_changes.requires_full_refresh %} | ||
|
|
||
| {{ get_replace_materialized_view_as_sql(relation, sql, existing_relation, backup_relation, intermediate_relation) }} | ||
|
|
||
| -- otherwise apply individual changes as needed | ||
| {% else %} | ||
|
|
||
| {{ postgres__update_indexes_on_materialized_view(relation, configuration_changes.indexes) }} | ||
|
|
||
| {%- endif -%} | ||
|
|
||
| {% endmacro %} | ||
|
|
||
|
|
||
| {%- macro postgres__update_indexes_on_materialized_view(relation, index_changes) -%} | ||
| {{- log("Applying UPDATE INDEXES to: " ~ relation) -}} | ||
|
|
||
| {%- for _index_change in index_changes -%} | ||
| {%- set _index = _index_change.context -%} | ||
|
|
||
| {%- if _index_change.action == "drop" -%} | ||
|
|
||
| {{ postgres__get_drop_index_sql(relation, _index.name) }}; | ||
|
|
||
| {%- elif _index_change.action == "create" -%} | ||
|
|
||
| {{ postgres__get_create_index_sql(relation, _index.as_node_config) }} | ||
|
|
||
| {%- endif -%} | ||
|
|
||
| {%- endfor -%} | ||
|
|
||
| {%- endmacro -%} |
8 changes: 8 additions & 0 deletions
8
plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/create.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| {% macro postgres__get_create_materialized_view_as_sql(relation, sql) %} | ||
| create materialized view if not exists {{ relation }} as {{ sql }}; | ||
|
|
||
| {% for _index_dict in config.get('indexes', []) -%} | ||
| {{- get_create_index_sql(relation, _index_dict) -}} | ||
| {%- endfor -%} | ||
|
|
||
| {% endmacro %} |
5 changes: 5 additions & 0 deletions
5
plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/describe.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| {% macro postgres__describe_materialized_view(relation) %} | ||
| -- for now just get the indexes, we don't need the name or the query yet | ||
| {% set _indexes = run_query(get_show_indexes_sql(relation)) %} | ||
| {% do return({'indexes': _indexes}) %} | ||
| {% endmacro %} |
3 changes: 3 additions & 0 deletions
3
plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/refresh.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| {% macro postgres__refresh_materialized_view(relation) %} | ||
| refresh materialized view {{ relation }} | ||
| {% endmacro %} |
3 changes: 3 additions & 0 deletions
3
plugins/postgres/dbt/include/postgres/macros/relations/materialized_view/rename.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| {% macro postgres__get_rename_materialized_view_sql(relation, new_name) %} | ||
| alter materialized view {{ relation }} rename to {{ new_name }} | ||
| {% endmacro %} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.