[CT-2949] [Feature] Graph operator for selecting parents / children, but not specified models

[jaklan]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Hi, I would like to figure out a local development flow utilising the new dbt feature - dbt clone. To mimic the behaviour of --defer, I need to run the clone command only for upstream dependencies of modified models to create views in my personal schemas in dev database (we overwrite the generate_schema_name macro for that).

To achieve it today, I have to run:

dbt clone --select +model_a +model_b --exclude model_a model_b --state path/to/artifacts

or

dbt clone --select +state:modified --exclude state:modified --state path/to/artifacts

which is pretty inconvenient. It would be much easier if we introduce a new graph operator not to include specified models, so we could run e.g.:

dbt clone --select ++model_a ++model_b --state path/to/artifacts

or

dbt clone --select ++state:modified --state path/to/artifacts

Who will this benefit?

dbt users using dbt clone (but not only)

[dbeatty10]

I would like to figure out a local development flow utilising the new dbt feature - dbt clone.

Hey @jaklan, it's great to see that you're exploring the dbt clone command! Thanks for sharing your thoughts on it and suggesting ways to make it even better 👑.

I see that the following command is a bit longer and includes an extra command-line option compared to what you suggested:

dbt clone --select +state:modified --exclude state:modified --state path/to/artifacts

However, I don't think it's too complicated. We're very interested in hearing more about any difficulties you might have with it and how it affects your development process. Let us know!

[jaklan]

Hi @dbeatty10, that's all about the user experience. Recently new selectors were introduced to make state selection MECE, that's a similar issue - to apply MECE principle to model selection as well. And the moment it's not fulfilled, because there's no way to select a subset of models not including the specified models without using exclude - dbt clone is just a sample use-case when it could be useful.

[jaklan]

Also regarding dbt clone - if we ask our users to use dbt clone as the default approach for local development, they would have to remember to use exclude every time they introduce a new model to avoid errors like:

17:05:25  Completed with 1 error and 0 warnings:
17:05:25
17:05:25  Database Error in model model (models/model.sql)
17:05:25    Could not find parent table for alias "database.schema.model".
17:05:25    compiled Code at target/run/project/models/model.sql

So as I mentioned above - it's not about an unresolvable issue, but simply about improving the UX.

[graciegoheen]

Hey @jaklan - totally understand why it's frustrating to remember to use exclude every time. Have you looked at all at adding a YAML selector to capture this logic? That way, you could define and name the selector that captures --select +state:modified --exclude state:modified which would simplify the command developers need to use on the command line.

[jaklan]

Hi @graciegoheen - yes, that would work for that concrete example, but wouldn't work when you want to specify the model you currently work on directly. The new operator (like proposed ++) would simply cover all such use-cases, with no need to think about any workarounds.