Is this a new bug in dbt v2.x compared to the latest version of dbt 1.x?
Current Behavior
A failed dbt run-operation writes target/run_results.json with one error result per model in the project. Every message is exactly Compilation Error, elapsed_time is 0, and the models were not compiled or run. The real error is only on stderr. args.which is run-operation.
A successful run-operation writes a single result for the macro.
Expected Behavior
run_results.json for run-operation should contain the operation result and the real error message. It should not report project models as compilation failures.
dbt run already does the right thing: a compile failure records only the selected node, with the actual message.
Steps To Reproduce
dbt 2.0.6. DuckDB. Complete script:
rm -rf /tmp/repro && mkdir -p /tmp/repro/{models,macros} && cd /tmp/repro
export DBT_PROFILES_DIR=$PWD
cat > profiles.yml << 'EOF'
repro:
target: dev
outputs:
dev:
type: duckdb
path: repro.duckdb
schema: main
EOF
cat > dbt_project.yml << 'EOF'
name: repro
version: 1.0.0
profile: repro
model-paths: [models]
macro-paths: [macros]
EOF
printf 'select 1 as id\n' > models/good.sql
printf 'select 2 as id\n' > models/also_good.sql
cat > macros/ops.sql << 'EOF'
{% macro ok() %}
{% do log('ok-macro', info=True) %}
{% endmacro %}
{% macro boom_compile() %}
{% do exceptions.raise_compiler_error('intentional compile boom') %}
{% endmacro %}
{% macro boom_sql() %}
{% do run_query('select * from this_table_does_not_exist_repro_zz') %}
{% endmacro %}
EOF
show() {
python3 -c 'import json; r=json.load(open("target/run_results.json")); print(r["args"]["which"], r["elapsed_time"], [(x["unique_id"], x["status"], x["message"]) for x in r["results"]])'
}
dbt --version
rm -rf target && dbt run-operation ok --target dev; echo "exit=$?"; show
rm -rf target && dbt run-operation boom_compile --target dev; echo "exit=$?"; show
rm -rf target && dbt run-operation boom_sql --target dev; echo "exit=$?"; showboom_compile and boom_sql produce the same artifact. The macro name and the actual error text are not in it.
Relevant log output
Finished 'run-operation' with 1 error for target 'dev' [1.2s]
exit=1
run-operation 0.0 [('model.repro.also_good', 'error', 'Compilation Error'), ('model.repro.good', 'error', 'Compilation Error')]
`ok` prints `run-operation 0.271 [('ok', 'success', 'Succeeded')]`.
Stderr for `boom_sql` is `DbDriverFailed (dbt1308): Catalog Error: Table with name this_table_does_not_exist_repro_zz does not exist!` That string is not in `run_results.json`.Environment
- OS: macOS
- dbt: 2.0.6 (Fusion)
- adapter: duckdb
Which database adapter are you using?
duckdb
Is this a discrepancy vs. dbt 1.x?
Additional Context
On failure, dbt stamps every selected node with the message Compilation Error so dbt retry can rerun them. For run-operation the selected set is the whole project, so a macro failure is recorded as a compilation error on every model. Reproduced on 2.0.5 and 2.0.6.
Is this a new bug in dbt v2.x compared to the latest version of dbt 1.x?
Current Behavior
A failed
dbt run-operationwritestarget/run_results.jsonwith oneerrorresult per model in the project. Every message is exactlyCompilation Error,elapsed_timeis0, and the models were not compiled or run. The real error is only on stderr.args.whichisrun-operation.A successful
run-operationwrites a single result for the macro.Expected Behavior
run_results.jsonforrun-operationshould contain the operation result and the real error message. It should not report project models as compilation failures.dbt runalready does the right thing: a compile failure records only the selected node, with the actual message.Steps To Reproduce
dbt 2.0.6. DuckDB. Complete script:
boom_compileandboom_sqlproduce the same artifact. The macro name and the actual error text are not in it.Relevant log output
Environment
Which database adapter are you using?
duckdb
Is this a discrepancy vs. dbt 1.x?
Additional Context
On failure, dbt stamps every selected node with the message
Compilation Errorsodbt retrycan rerun them. Forrun-operationthe selected set is the whole project, so a macro failure is recorded as a compilation error on every model. Reproduced on 2.0.5 and 2.0.6.