[CT-1585] [Feature] New top level commands: interactive preview

[ChenyuLInx]

Describe the feature

See #6358 for more background of what interactive means.

This one is for supporting interactive preview function as a native function for dbt cli. Example of how this happens in dbt-rpc and dbt-server(link to actual code)

AC:
See last comment

[jtcohen6]

A lot of community interest in this one! :) #5418, #6087, #6090

Mostly the same comments as in #6358 (comment). I do think a totally new command for this one makes sense. It would work much the same as interactive compile, with the added step of actually executing the compiled query.

$ dbt preview --code <base64encoded>

From the CLI perspective, this should feel much the same as:

-- macros/preview.sql
{% macro preview(code) %}
  {% set result = run_query(code) %}
  {% do result.print_table() %}
{% endmacro %}

$ dbt run-operation preview --args '{"code": "select 1 as id, True as is_odd union all select 2, False"}'
| id | is_odd |
| -- | ------ |
|  1 |   True |
|  2 |  False |

formatting + passing the result

In the existing implementation, execute_result returns an agate.Table, and then we turn it into a simpler ResultTable Python object. My inclination here is that, on the CLI, we'd send the result to stdout, with the format depending on the --log-format:

• Text format: agate.Table.print_table() (GitHub-flavored Markdown), though capturing the output rather than sending it straight to sys.stdout.
  • Here's a fun one: did you know there's a dbt seed --show? The way we implemented that (a long time ago) is pretty funny :)
• JSON format: either agate.Table.print_json(), or our own JSON serialization of the ResultTable object
• Binary format: ...hm... let's ignore for now

For programmatic consumers (dbt-server), should we return the ResultTable directly as a Python object? Include a serialized representation in a structured log event? Is there a way we could proto-tize that result object, to maximize interoperability? (Is it time for us to finally roll our own "dataframe" helper for result sets, and replace agate?)

limiting result set

What are the arguments we want to support for this? # of lines?

Hm. I'm not 100% sure that this should be in dbt-core, but it does feel like it could be a place to enact a configurable limit to the result set. I'd be curious to hear what folks like @bossnunta @isaac-taylor @davidharting think.

The default should either be:

• something reasonable (--limit 500)
• nothing at all; dbt doesn't try to limit the result set unless passed an explicit configuration

Options on how to implement:

• Modify SQL! Add limit 500 to the end of queries — or detect if they contain their own limit clause, in which case, don't add a limit clause.
  • Of course, this makes sense for SQL, but not for other languages! We could add as an optional arg to the statement macro, and attempt some conditional logic based on language. Python dataframes do support methods like .take(<size>) or .head(<size>), and we could try passing that into submit_python_job (which would also need a refactor to support "interactive" queries, never mind how slow it would be on some platforms).
• Add a new argument to adapter.execute: adapter.execute(<code>, fetch=True, limit=500). Then, in each database adapter, we plumb this through to connections.execute. Many cursors support fetchmany(size=<int>) (e.g. psycopg2, snowflake-connector-python), and there's a natural spot for it here.
  • This feels like the surest way to go, and least error-prone
  • But if a user really did want >500 rows, we'd need to either do SQL parsing to detect that their query includes limit 1000 (as above), or provide another means to pass in that information, e.g. in the dbt Cloud IDE.

Let's keep thinking through this part. I think it can be out of scope for the initial first-pass implementation.

[isaac-taylor]

Hm. I'm not 100% sure that this should be in dbt-core, but it does feel like it could be a place to enact a configurable limit to the result set. I'd be curious to hear what folks like @bossnunta @isaac-taylor @davidharting think.

In my head, the question comes down to 'should we modify user code directly?' And if so (the IDE does today), where in the stack should that happen.

My guess is Core is the right place long term. That will make extending interactive preview/compile support to Python (if that's even doable) a bit easier, as we won't have multiple components needing to understand user code. I'm thinking the IDE shouldn't know about anything related to dbt code parsing itself.

[jtcohen6]

@isaac-taylor Agreed. We can discuss this more offline — what's your + @bossnunta's appetite to have a "Limit [500]" checkbox in the IDE again? That's probably the right UX, if what we're doing is passing a --limit [500] argument into the top-level dbt-core command (programmatically via dbt-server).

[isaac-taylor]

@isaac-taylor Agreed. We can discuss this more offline — what's your + @bossnunta's appetite to have a "Limit [500]" checkbox in the IDE again? That's probably the right UX, if what we're doing is passing a --limit [500] argument into the top-level dbt-core command (programmatically via dbt-server).

Seems like just a missing feature from 1.1 that we had in 1.0. Very doable from a technical perspective. It's about UX and prioritization I think Let me bring that one up to Nate to see if that's on his radar.

[bossnunta]

@isaac-taylor Agreed. We can discuss this more offline — what's your + @bossnunta's appetite to have a "Limit [500]" checkbox in the IDE again? That's probably the right UX, if what we're doing is passing a --limit [500] argument into the top-level dbt-core command (programmatically via dbt-server).

I think having a version of "limit [500]" as a UX (and/or centralized admin configuration) is a very good approach here. I don't believe we had that in IDE v1.0 either. I shared notes on this in Slack. (What we do in both 1.0 and 1.1 is the default to 500 rows that is overridden when you have a limit clause).

However, from product perspective, the IDE shouldn't really be doing sql parsing to deal with this in my opinion. Ideally we could do this with dbt-server (or maybe Core given that it's about parsing?)