[CT-1984] [Regression] In v1.4, seeds with hooks calling macros raise an AttributeError

[steveh-101]

Is this a regression in a recent version of dbt-core?

• I believe this is a regression in dbt-core functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

After trying to upgrade both dbt-utils, dbt-core and dbt-snowflake to 1.0.0, 1.4.1 and 1.4.0 respectively, my project no-longer compiles.

Downgrading just the dbt parts (core and snowflake) and it compiles.

Expected/Previous Behavior

I expect that after following migration guides etc. that the project will compile.

It compiles as normal if I downgrade dbt to previous version (1.3.2)

Steps To Reproduce

This is not easily reproducible. It did not affect a new project, just a current one. Something in 1.4.1 is different in the fact that downgrading dbt, the project does compile.

Relevant log output

dbt compile  
18:42:38  Running with dbt=1.4.1
18:42:54  Encountered an error:
'SeedNode' object has no attribute 'depends_on'
18:42:54  Traceback (most recent call last):
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/main.py", line 135, in main
    results, succeeded = handle_and_check(args)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/main.py", line 198, in handle_and_check
    task, res = run_from_args(parsed)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/main.py", line 245, in run_from_args
    results = task.run()
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/task/runnable.py", line 454, in run
    self._runtime_initialize()
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/task/runnable.py", line 165, in _runtime_initialize
    super()._runtime_initialize()
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/task/runnable.py", line 94, in _runtime_initialize
    self.load_manifest()
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/task/runnable.py", line 81, in load_manifest
    self.manifest = ManifestLoader.get_full_manifest(self.config)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/parser/manifest.py", line 203, in get_full_manifest
    manifest = loader.load()
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/parser/manifest.py", line 339, in load
    self.parse_project(
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/parser/manifest.py", line 467, in parse_project
    parser.parse_file(block)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/parser/base.py", line 425, in parse_file
    self.parse_node(file_block)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/parser/base.py", line 386, in parse_node
    self.render_update(node, config)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/parser/base.py", line 363, in render_update
    self.update_parsed_node_config(node, config, context=context)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/parser/base.py", line 336, in update_parsed_node_config
    get_rendered(hook.sql, context, parsed_node, capture_macros=True)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/clients/jinja.py", line 590, in get_rendered
    return render_template(template, ctx, node)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/clients/jinja.py", line 545, in render_template
    return template.render(ctx)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/jinja2/sandbox.py", line 393, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/clients/jinja.py", line 328, in __call__
    with self.track_call():
  File "/Users/stephen.herron/.pyenv/versions/3.10.7/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/Users/stephen.herron/.local/share/virtualenvs/snowflake_dbt-aabVNRJU/lib/python3.10/site-packages/dbt/clients/jinja.py", line 319, in track_call
    self.node.depends_on.add_macro(unique_id)
AttributeError: 'SeedNode' object has no attribute 'depends_on'

Environment

• OS: MacOS
• Python: 3.10.7
• dbt-core (working version): 1.3.2
• dbt-core (regression version): 1.4.1

Which database adapter are you using with dbt?

snowflake

Additional Context

This must be something weird with my project. Out of interest I created a new project in the same virtual env and that worked.

I will start trying to copy over things to see what is causing the breakage, starting with seeds as the error mentioned SeedNode

[steveh-101]

Think I narrowed it down..

There are seeds in my project that have a post-hook that calls a macro.

e.g.

seeds:
  quote_columns: false
   my_project:
     reporting:
       +schema: reporting
       +post-hook:
         - "{{ apply_masking_policy('models') }}"

Works ok on previous? Need to try a simpler macro..

I tried changing the post-hook to some generic sql, i.e. "select 1 as test" and the project compiled.

edit: does not work with a my_silly_macro() either.

[dbeatty10]

Thanks for opening @RudeB0y !

Agreed that you narrowed it down, and I was able to reproduce the same error using dbt-core==1.4.1 and dbt-snowflake==1.40

Here's what I did to reproduce:

dbt_project.yaml

name: "my_dbt_project"
version: "1.0.0"
config-version: 2
profile: "my_profile"

seeds:
  +post-hook:
    - "{{ my_silly_macro() }}"

seeds/dummy_seed.csv

id
1

macros/my_silly_macro.sql

{% macro my_silly_macro() %}
select 1 as test
{% endmacro %}

Then run:

dbt seed -s dummy_seed

Then I got the same error as you:

AttributeError: 'SeedNode' object has no attribute 'depends_on'

[jeremyyeo]

Does this remove the depends_on attr from seeds?

dbt-core/core/dbt/contracts/util.py

Line 253 in 52be5ff

"depends_on",

[jtcohen6]

In v1.4, we removed dependency-related attributes from seeds, since they can't actually use ref/source/metric, or call macros directly, in their definition (CSV). That included depends_on, refs, sources, and metrics.

However, it's true that seeds can have pre- and post-hooks. Those hooks can call macros! It's important we track those dependencies for dbt seed --select state:modified.macros (docs). That's what we're trying to do here, and it's where the error is surfacing:

dbt-core/core/dbt/clients/jinja.py

Lines 308 to 324 in b2ea2b8

	# This adds the macro's unique id to the node's 'depends_on'
	@contextmanager
	def track_call(self):
	# This is only called from __call__
	if self.stack is None:
	yield
	else:
	unique_id = self.macro.unique_id
	depth = self.stack.depth
	# only mark depth=0 as a dependency, when creating this dependency we don't pass in stack
	if depth == 0 and self.node:
	self.node.depends_on.add_macro(unique_id)
	self.stack.push(unique_id)
	try:
	yield
	finally:
	self.stack.pop(unique_id)

---

Can those macros also call ref/source/metric???

At first glance, yes. In practice, I sure hope not.

If I try this in 1.3.latest, adapting @dbeatty10's reproduction case above, it's true that it ""works"":

{% macro my_silly_macro() %}
select * from {{ ref('my_model') }}
{% endmacro %}

I run dbt seed, it succeeds:

[0m11:36:42.191361 [debug] [Thread-1  ]: On seed.test.seed: 
          insert into "jerco"."dbt_jcohen"."seed" ("id") values
          (%s)
      ...
�[0m11:36:42.192621 [debug] [Thread-1  ]: SQL status: INSERT 0 1 in 0.0 seconds
�[0m11:36:42.197663 [debug] [Thread-1  ]: Writing runtime SQL for node "seed.test.seed"
�[0m11:36:42.212489 [debug] [Thread-1  ]: Using postgres connection "seed.test.seed"
�[0m11:36:42.212629 [debug] [Thread-1  ]: On seed.test.seed: /* {"app": "dbt", "dbt_version": "1.3.2", "profile_name": "garage-postgres", "target_name": "dev", "node_id": "seed.test.seed"} */

        select * from "jerco"."dbt_jcohen"."my_model"

In manifest.json:

...
    "seed.test.seed": {
      "resource_type": "seed",
      "depends_on": {
        "macros": [
          "macro.test.my_silly_macro"
        ],
        "nodes": [
          "model.test.my_model"
        ]
      },
...

I hate this. I'm willing to call it an unsupported & undocumented anti-pattern.

Proposed resolution

• Add depends_on back to seeds, but only MacroDependsOn (the same that macros have)
• Raise an explicit error if a seed tries to take a dependency on another resource, via its pre/post hook

I can open a PR for this, with the (bad) code I wrote while trying to understand the codepaths in play—but I definitely want to get @gshank's blessing before moving forward with this approach.