Skip to content

Ct 1919/get column schema from query macro core #6981

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
MichelleArk wants to merge 119 commits into from
Closed

Ct 1919/get column schema from query macro core #6981

MichelleArk wants to merge 119 commits into from

Conversation

@MichelleArk
Copy link
Contributor

@MichelleArk MichelleArk commented Feb 15, 2023 •
edited
Loading

resolves #6751

Description

Checklist

❯ dbt run --select test_schema
02:53:06  Running with dbt=1.5.0-a1
02:53:06  Found 6 models, 20 tests, 0 snapshots, 1 analysis, 503 macros, 0 operations, 3 seed files, 0 sources, 1 exposure, 1 metric
02:53:06  
02:53:07  Concurrency: 1 threads (target='dev')
02:53:07  
02:53:07  1 of 1 START sql table model test_arky.test_schematest_v2 ...................... [RUN]
02:53:07  1 of 1 ERROR creating sql table model test_arky.test_schematest_v2 ............. [ERROR in 0.04s]
02:53:07  
02:53:07  Finished running 1 table model in 0 hours 0 minutes and 0.46 seconds (0.46s).
02:53:07  
02:53:07  Completed with 1 error and 0 warnings:
02:53:07  
02:53:07  Compilation Error in model test_schema (models/test_schema.sql)
02:53:07    Please ensure the name, data_type, order, and number of columns in your `yml` file match the columns in your SQL file.
02:53:07    Schema File Columns: 
02:53:07      a INT, b TEXT
02:53:07    
02:53:07    SQL File Columns: 
02:53:07      a INT, c TEXT 
02:53:07    
02:53:07    > in macro assert_columns_equivalent (macros/materializations/models/table/columns_spec_ddl.sql)
02:53:07    > called by model test_schema (models/test_schema.sql)
02:53:07  
02:53:07  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

@sungchun12
start off the blueprints
e879cdd
@sungchun12
Merge branch 'main' of https://github.com/dbt-labs/dbt into feature/d…
bd1afc7 
…bt-constraints
@sungchun12
Merge branch 'main' of https://github.com/dbt-labs/dbt into feature/d…
4b2a881 
…bt-constraints
@sungchun12
test commit
e8b52e0
@sungchun12
working snowflake env
01a07d3
@sungchun12
update manifest expectation
3b31a15
@sungchun12
add error handling
1a1f46a
@sungchun12
clean up language
ebaa54c
@sungchun12
constraints validator
fd7a47a
@sungchun12
cleaner example
7e3a9be
@sungchun12
better terminal output
6df02da
@sungchun12
add python error handling
bc3c5bc
@sungchun12
Merge branch 'main' of https://github.com/dbt-labs/dbt into dbt-const…
4b901fd 
…raints
@sungchun12
add to manifest schema
e29b571
@sungchun12
add to schema config
7d085dc
@sungchun12
clean up comments
e6559d4
@sungchun12
backwards compatible nodeconfig
ca89141
@sungchun12
remove comments
2c4a4cf
@sungchun12
clean up more comments
1975c6b
@sungchun12
add changelog
f088a03
@sungchun12
clarify error message
7421caa
@sungchun12
constraints list type
a7395bb
@sungchun12
fix grammar
5d06524
@sungchun12
add global macros
380bd96
@sungchun12
clearer compile error
e6e490d
@sungchun12
remove comments
9c498ef
@sungchun12
fix tests in this file
bed1fec
@sungchun12
conditional compile errors
8c466b0
@sungchun12
add conditional check in ddl
547ad9e
@sungchun12
Merge branch 'main' of https://github.com/dbt-labs/dbt into dbt-const…
52bd35b 
…raints
sungchun12 and others added 21 commits January 17, 2023 10:01
@sungchun12
Merge branch 'main' of https://github.com/dbt-labs/dbt into dbt-const…
a501c29 
…raints
@sungchun12 @dave-connors-3
Update core/dbt/include/global_project/macros/materializations/models…
59b0298 
…/table/columns_spec_ddl.sql


For whitespace control in the dbt.log file!

Co-authored-by: dave-connors-3 <[email protected]>
@sungchun12
Revert "Update core/dbt/include/global_project/macros/materialization…
e46066b 
…s/models/table/columns_spec_ddl.sql
"

This reverts commit 59b0298.
@sungchun12
update for pre-commit hooks
e80b8cd
@sungchun12
update for black formatter
257eacd
@sungchun12
update for black formatter on all files
391bd3b
@jtcohen6
Merge remote-tracking branch 'origin/main' into dbt-constraints
0441417
@jtcohen6
Refactor functional tests
e0bcb25
@jtcohen6
Fixup formatting
dcf7062
@jtcohen6
Dave feedback
903a2cb
@jtcohen6
another one - dave
c40ee92
@jtcohen6
the hits keep coming
111683a
@dave-connors-3
adjust whitespace
f8b16bc
@sungchun12
Merge branch 'main' of https://github.com/dbt-labs/dbt into dbt-const…
97f0c6b 
…raints
@jtcohen6
Light touchup
2a33baf
@jtcohen6
Merge remote-tracking branch 'origin/main' into dbt-constraints
6806a7c
@jtcohen6
Add more flexibility for spark
1256e7b
@jtcohen6
Nearly there for spark
0304dbf
@jtcohen6
Merge main
b5b1699
@jtcohen6 @MichelleArk
Experiment: column types from empty query
1b0c213
@MichelleArk
use get_empty_schema_sql to get column schema from schema yml
64470ca
@cla-bot cla-bot bot added the cla:yes label Feb 15, 2023
@MichelleArk MichelleArk changed the base branch from main to dbt-constraints February 15, 2023 02:55
@github-actions
Copy link
Contributor

github-actions bot commented Feb 15, 2023

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

MichelleArk
MichelleArk commented Feb 15, 2023
View reviewed changes
core/dbt/adapters/sql/connections.py
)

@classmethod
def get_column_schema_from_cursor(cls, cursor: Any) -> List[Tuple[str, str]]:
Copy link
Contributor Author

@MichelleArk MichelleArk Feb 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: change to -> List[Tuple[str, Any]]

MichelleArk
MichelleArk commented Feb 15, 2023
View reviewed changes
core/dbt/adapters/sql/connections.py
# sql = self._add_query_comment(sql)
# # auto_begin is ignored on bigquery, and only included for consistency
# query_job, iterator = self.raw_execute(sql)
# columns = [(field.name, field.field_type) for field in resp.iterator]
Copy link
Contributor Author

@MichelleArk MichelleArk Feb 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/resp.iterator/iterator.schema

Base automatically changed from dbt-constraints to main February 15, 2023 15:50
@MichelleArk
Copy link
Contributor Author

MichelleArk commented Feb 15, 2023

Closing in favor of #6986

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

cla:yes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[CT-1919] Create get_column_schema_from_query macro

6 participants

@MichelleArk @sungchun12 @FishtownBuildBot @jtcohen6 @dave-connors-3