[CT-1853] [Bug] When a dependency package is missing the dbt_project.yml, dbt deps in the main project fails with a non-descript error

[Mathyoub]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Main dbt project A calls dependency package B.
Dependency B is not in the package hub, but rather in a customer hosted repo retrieved by git.

When the job runs dbt deps, it fails with a strange error message:

[2022-11-08 00:22:21.759993] ERROR: dbt_cloud: Runtime Error
  no dbt_project.yml found at expected path /tmp/dbt-downloads-eh3_yove/c1592655e2ed8fe15c54c562a0934786/dbt_project.yml

Except that Project A actually has a dbt_project.yml and otherwise runs fine in the IDE.

Further research discovers that Package B is actually missing the dbt_project.yml, and that’s what the error message is actually referring too: a missing .yml in a dependency, not the main project.

Fixing this missing .yml fixes the problem.

However, this error message makes it look more like a system failure (/tmp path) than a user-inflicted error.

Expected Behavior

dbt deps fails, but with an error message stating which dependency (in this case a package URL) has the problem.

Steps To Reproduce

See current behavior

Relevant log output

Here’s the traceback:

Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1396, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 154, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 191, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 238, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 131, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/base.py", line 86, in fetch_metadata
    self._cached_metadata = self._fetch_metadata(project, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/git.py", line 103, in _fetch_metadata
    loaded = Project.from_project_root(path, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/config/project.py", line 652, in from_project_root
    partial = cls.partial_load(project_root, verify_version=verify_version)
  File "/usr/local/lib/python3.8/dist-packages/dbt/config/project.py", line 639, in partial_load
    return PartialProject.from_project_root(
  File "/usr/local/lib/python3.8/dist-packages/dbt/config/project.py", line 482, in from_project_root
    project_dict = _raw_project_from(project_root)
  File "/usr/local/lib/python3.8/dist-packages/dbt/config/project.py", line 166, in _raw_project_from
    raise DbtProjectError(
dbt.exceptions.DbtProjectError: Runtime Error
  no dbt_project.yml found at expected path /tmp/dbt-downloads-eh3_yove/c1592655e2ed8fe15c54c562a0934786/dbt_project.yml

Environment

- OS:
- Python:
- dbt:

Which database adapter are you using with dbt?

No response

Additional Context

Workaround: Examine all called deps for a missing dbt_project.yml

[dbeatty10]

Thanks for opening @Mathyoub !

Initial assessment

After an initial reading, it doesn't look like dbt is behaving erroneously, so this actually doesn't feel like a bug to me.

However, I can see how the error message might not have enough specificity in some cases. e.g., it might not be clear to the end user that one of the packages listed within packages.yml (or one of its transitive dependencies!) is missing dbt_project.yml.

Is there a simple solution?

Unfortunately, I don't see a simple way of just updating the error message to say something like "no dbt_project.yml found for package XYZ".

This is because of a few things:

• the variety of ways that packages can be specified (not all have an obvious name to use, like local packages)
• the location of the current error lacks the context of knowing about the package that it's related to (it only knows the path)
• the earliest place that has the context is pretty far up the chain

Options moving forward

1. One option is to make no changes. While not trivial to troubleshoot currently, it also doesn't feel impossible.
2. Another option is doing a rather invasive refactor in order to inject more specific context of the offending "package" into the error message. This feels overkill in this case.
3. A final option is updating the current generic message to contain just a bit more guidance of what the user can do to troubleshoot this effectively on their own. I think we should go this route!

Current error message:

dbt.exceptions.DbtProjectError: Runtime Error
  no dbt_project.yml found at expected path /tmp/dbt-downloads-eh3_yove/c1592655e2ed8fe15c54c562a0934786/dbt_project.yml

Proposed error message:

dbt.exceptions.DbtProjectError: Runtime Error
  no dbt_project.yml found at expected path /tmp/dbt-downloads-eh3_yove/c1592655e2ed8fe15c54c562a0934786/dbt_project.yml
  verify that each entry within `packages.yml` (and each their transitive dependencies) contains a file named dbt_project.yml

Can you think of any better ways to phrase this error message in a similarly generic fashion @Mathyoub ?

[Mathyoub]

@dbeatty10 I think that suggested generic error message makes sense! With that being said though I think 1 or 3 are fine options because this seems relatively low priority. Although if 3 is as easy as it seems then maybe we just go ahead and update it?

[dbeatty10]

if 3 is as easy as it seems then maybe we just go ahead and update it?

I put up a PR to update it here! It's marked as "ready for review", but we can still theoretically make updates to the error message if we want.