Skip to content

load_via_dbt_ls() requires dbt is installed in base airflow environment #670

Closed
Closed
load_via_dbt_ls() requires dbt is installed in base airflow environment#670
Labels
area:dependenciesRelated to dependencies, like Python packages, library versions, etcarea:executionRelated to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etcdbt:listPrimarily related to dbt list command or functionalityexecution:virtualenvRelated to Virtualenv execution environmentparsing:dbt_lsIssues, questions, or features related to dbt_ls parsingpriority:highHigh priority issues are blocking or critical issues without a workaround and large impact

Description

@dylanharper-qz
dylanharper-qz
opened on Nov 13, 2023

Apologies if I'm misunderstanding anything here, we're big fans of Cosmos at Quizlet.

In version 1.2.2, we are seeing this error when using LoadMode.DBT_LS:

Broken DAG: [/usr/local/airflow/dags/my_dbt_hourly.py] Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/cosmos/dbt/graph.py", line 184, in load
    load_method[method]()
  File "/usr/local/lib/python3.11/site-packages/cosmos/dbt/graph.py", line 233, in load_via_dbt_ls
    raise CosmosLoadDbtException(f"Unable to find the dbt executable: {self.dbt_cmd}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to find the dbt executable: dbt

If we add dbt to our default airflow environment, the dag works as expected.

It is required to install dbt outside of the venv described in the docs?

This code seems to be checking for dbt in the airflow environment:

astronomer-cosmos/cosmos/dbt/graph.py

Lines 232 to 233 in 6e41471

if not shutil.which(self.dbt_cmd):
raise CosmosLoadDbtException(f"Unable to find the dbt executable: {self.dbt_cmd}")

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:dependenciesRelated to dependencies, like Python packages, library versions, etcarea:executionRelated to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etcdbt:listPrimarily related to dbt list command or functionalityexecution:virtualenvRelated to Virtualenv execution environmentparsing:dbt_lsIssues, questions, or features related to dbt_ls parsingpriority:highHigh priority issues are blocking or critical issues without a workaround and large impact

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions