dbt Labs Unveils AI Innovations and New Features to Improve Collaboration and Multi-platform Analytics

One dbt unlocks the potential of the Analytics Development Lifecycle

dbt Labs, the pioneer in analytics engineering, announced new features and enhancements to dbt Cloud, cementing the platform as the data control plane for enterprise analytics. Announced during the keynote of dbt Labs’ annual conference, Coalesce 2024, the new innovations support users across various stages of the analytics development lifecycle by providing unprecedented cross-platform flexibility, empowering more people to contribute to the analytics workflow while accelerating speed and productivity, and improving organizational trust in data. This includes the launch of dbt Copilot, an AI engine embedded across dbt Cloud to bolster productivity while improving data quality.

Also Read: SAP Supercharges Copilot Joule with Collaborative Capabilities to Ignite Enterprise AI Revolution

Unifying analytics workflows with One dbt

A central theme to Coalesce 2024 is One dbt, a commitment to create a single unified dbt experience, regardless of infrastructure, data platform or cloud an organization is using.

Through One dbt, capabilities will be unified for data transformation, observability, orchestration, cataloging, and semantics, regardless of data platform or cloud an organization chooses to use – providing a universal view of what’s happening within the data estate, accelerating data delivery, tuning data quality, and optimizing compute costs.

According to dbt Labs’ 2024 State of Analytics Engineering report, decade-long challenges like building data transformations and improving access to compute have largely been solved, but major hurdles related to data quality, data literacy, and data ownership continue to plague the industry. By embracing the Analytics Development Lifecycle (ADLC), an integrated, mature analytics workflow, organizations can now benefit from a standardized, scalable way to overcome these issues and move faster with trusted data.

“The data industry has made real progress towards maturity over the past decade,” said Tristan Handy, founder and CEO of dbt Labs. “But real problems persist. Siloed data. Lack of trust. Too much ‘duct tape’ in our operational systems. Our announcements from this week go a long way toward fixing these gaps: One dbt experience that is cross-platform, multi-persona, trusted, and infused with AI. All facilitating a single mature workflow: the Analytics Development Lifecycle.”

Delivering enhanced flexibility, collaboration, and trust with the dbt Cloud data control plane

dbt Cloud is a data control plane that supports users across every stage of the ADLC—regardless of their title, technical aptitude, chosen data platform, or where they build and consume data— and further accelerates the workflow with AI. dbt Cloud centralizes metadata and makes it actionable across the ADLC workflow.

A suite of new features and capabilities across the dbt Cloud data control plane, introduced at Coalesce, have been designed to scale adoption across a more diverse set of data practitioners, make data development more accessible, streamlined, and governed, and build and automate high quality data pipelines. These include:

• dbt Copilot, the AI engine in dbt Cloud that helps users accelerate their analytics workflows. dbt Copilot is designed to automate tasks that previously required repetitive manual work, significantly improving productivity, data quality, and stakeholder trust. Today, this includes the ability to auto-generate tests, documentation, and semantic models (all in beta), an AI-chatbot that allows business stakeholders to ask natural language questions of their data (in beta as part of the dbt native app in Snowflake), and the ability to bring your own OpenAI API key (GA). In the coming months, dbt Copilot will extend to help automate model code generation.
• Cross-platform dbt Mesh will build on dbt Mesh’s existing support for cross-project references and allow for cross-platform references using the Iceberg table format as the underlying transport layer. This lets users eliminate silos while maintaining data governance, even in increasingly complex, multi-platform environments. Using cross-platform references in dbt Mesh, data teams will be able to centrally define and maintain data governance standards, see end-to-end lineage across various data platforms, and easily find, reference, and re-use existing data assets instead of rebuilding.
• Support for Apache Iceberg enables users to create tables in the Iceberg format and benefit from Iceberg’s first-class performance and portability. Apache Iceberg support makes cross-platform dbt Mesh possible. Snowflake support is currently in beta, and Athena, Spark, Databricks, Starburst/Trino, and Dremio are GA.
• A new visual editing experience, currently in beta, is a low-code, drag-and-drop environment for building and exploring dbt models, designed to democratize the ADLC to more types of users. Just like everything else in dbt, under-the-hood these visual models compile down to SQL and new code must be version controlled before being deployed into production. This new development interface gives downstream users (who already have the most business context) the ability to accessibly—and safely—author analytics code. Users who are more familiar with SQL can opt to use the visual editing experience to check their work and explore a visual representation of their models.
• Advanced CI, now GA, allows users to compare code changes as part of the CI process to catch unexpected behavior before new code is merged into production. This improves code quality and helps organizations optimize compute spend by only materializing correct models. Users can see a summary of their changes in their Git pull request and dive into modified, added, and removed rows and columns within dbt Cloud.
• Data health tiles, now GA, can be embedded into any downstream app so data consumers can have real-time context into critical trust signals like data freshness and data quality directly in the tools where they work.
• Auto-exposures with Tableau, now in Preview, automatically incorporates Tableau dashboards into dbt lineage. This allows data practitioners to optimize, expedite, and automate the orchestration of end-to-end pipelines—from source to dashboard. Business users can be confident that they always have the freshest data powering their decisions.
• An upcoming Power BI integration for the dbt Semantic Layer will enable business users who have standardized on the Microsoft ecosystem to query and analyze consistent metrics.cloud
• Teradata and Athena are supported adapters. dbt Cloud now integrates with Teradata (in Preview) and AWS Athena (GA), enabling more organizations and teams to collaborate on data workflows.

Also Read: Taking Advantage of Gen AI With Next-level Automation

These new features and others are enabling dbt customers like Roche to unify their data, standardize and accelerate their workflows, and maximize their investment in dbt.

“dbt Core jump-started our data platform’s growth, and dbt Cloud allowed us to spread it across the globe,” said Yannick Misteli, Head of Engineering, Global Product Strategy at Roche. “Today, we are able to power our platform in 70 countries and run over 15,000 models and 40,000 tests every day. We can support our core and country teams with the workflows that best suit them and promote code to production in two week cycles instead of the previous quarter or semester-long cycles.”

[To share your insights with us as part of editorial or sponsored content, please write to psen@itechseries.com]