the majority of operational problems that arise when moving to a distributed architecture are ultimately grounded in two areas: networking and observability.

track

In the case of dbt, rather than executing dbt models as black boxes, the framework can accumulate information about dbt runs over time, making that information available for longitudinal views and connecting it back to individual runs of your pipelines with no marginal setup effort.Engineers can monitor dbt model execution in the same tool they use to monitor other technologies on the data platform.Observability boils down to a couple of simple capabilities that we think everyone operating with data needs. With Dagster, individual dbt users can self-serve questions like “Which dbt models are at risk of becoming bottlenecks in our reporting pipeline?” And engineers can monitor dbt model execution in the same tool they use to monitor other technologies on the data platform.

Support for observability is built into the system. Whether the nodes in your orchestration graph (what Dagster calls solids) run dbt models or reports in Mode, instigate Spark jobs or perform pure Python computations, they yield a stream of structured metadata back to the framework as they execute. This metadata makes deep integrations with other technologies possible.

"The wider world is mostly characterized by wicked learning environments, where you can’t see information. It’s hidden from us. Feedback is delayed and sometimes inaccurate.