Why DBT Optimisation Hits a Ceiling (And How SQL Mesh Breaks Through)

Published on August 27, 2025

DBT+Snowflake teams are hitting a frustrating ceiling: despite months of optimisation work, costs keep climbing whilst engineering velocity stagnates. The problem isn’t your team’s competence – it’s architectural limitations that no amount of tuning can solve.

During our recent LinkedIn Live discussion “Tuning Won’t Cut Costs in Half – Time for a Technology Shift to SQL Mesh?”, practitioners shared hard-won insights about when DBT optimisation reaches its limits and SQL Mesh migration becomes the strategic path forward. The conversation revealed something counterintuitive: the teams achieving 50-70% cost reductions aren’t necessarily better at optimisation – they’ve migrated to fundamentally more efficient architecture.

You’ll discover in this session recap:

  • Why DBT’s full-rebuild architecture creates unavoidable cost scaling problems
  • How SQL Mesh’s incremental processing changes the economics of data transformation
  • What makes SQL Mesh migration complex and how to eliminate traditional migration risks
  • When teams should plan the transition and what results to expect

This article captures key insights from our LinkedIn Live featuring Joseph Lane from Avios (IAG Loyalty), Shane Gibson from AgileData.io, and Aaron Phethean from Matatika, where they discussed the practical realities of SQL Mesh migration at scale.

Meet the Experts

Joseph Lane – Technical Lead, Avios (IAG Loyalty)

Joseph led a comprehensive migration from DBT to SQL Mesh at Avios, the loyalty currency behind British Airways and other major brands. His background spans both startups and large-scale systems, giving him perspective on SQL Mesh’s benefits at different scales. At Avios, he’s responsible for data infrastructure that supports a business processing over £1 billion annually whilst growing at 20% year-on-year. His migration approach focused on applying software engineering discipline throughout the SQL Mesh transition.

Shane Gibson – Co-Founder, AgileData.io

Shane brings 35 years of experience witnessing technology cycles across the data industry, including the rise of DBT and emergence of SQL Mesh alternatives. As author of “An Agile Data Guide” and coach to data teams worldwide, he specialises in helping organisations navigate technology migrations without falling into common transformation traps. His work focuses on balancing innovation with sustainable migration practices, helping teams avoid the chaos that often accompanies major platform transitions.

Aaron Phethean – Founder, Matatika

Aaron guides teams through complex SQL Mesh migrations whilst eliminating traditional migration risks through Matatika’s Mirror Mode approach. His background includes building systems used by over 1.2 billion people, giving him deep experience with infrastructure that must scale reliably. At Matatika, his approach centres on making SQL Mesh migration feel simple through careful planning and parallel validation techniques, allowing teams to validate SQL Mesh performance before committing to the switch.

When DBT+Snowflake Optimisation Hits Its Ceiling

There’s a common assumption that rising costs and slowing velocity mean you need better DBT optimisation. But our discussion with these practitioners revealed something different: sometimes the problem isn’t inefficiency – it’s DBT’s fundamental architecture that no amount of tuning can solve.

Teams feel trapped when finance demands cost cuts whilst fearing the complexity of SQL Mesh migration. Engineering capacity burns on DBT-specific workarounds whilst leadership loses confidence in the current data infrastructure approach.

Joseph pointed out how DBT’s limitations compound at scale: “You start to hit constraints where DBT scaling becomes expensive, and that translates to developers blocking each other, burning Snowflake credits, just by the way DBT works. Or you end up writing custom code around DBT to make it behave efficiently.”

The pressure mounts when traditional DBT optimisation delivers diminishing returns while data volumes grow exponentially. That’s when SQL Mesh migration shifts from interesting to essential.

 

1. Why SQL Mesh’s State-Aware Processing Changes Everything

The insight: DBT’s stateless architecture wastes compute on unchanged data. SQL Mesh’s state-aware processing dramatically reduces costs by only transforming what’s actually different.

Joseph explained the fundamental difference that drives SQL Mesh migration decisions: “DBT is unaware of the state of anything in your data warehouse. SQL Mesh understands the state of objects it’s altering, so it can run much smaller queries. All of a sudden you’re processing a lot less data.”

This architectural advantage eliminates DBT’s expensive pattern of full rebuilds. Instead of DBT’s compute costs scaling with total data volume, SQL Mesh scales costs with actual change volume – typically a fraction of total data.

The productivity gains extend beyond compute savings. Joseph noted about their SQL Mesh migration: “You’re not double-running everything like in DBT environments. When you have thousands of models and 30-40-50 people committing daily, you’re effectively halving your workload.”

Expected outcome: Teams migrating from DBT to SQL Mesh typically see 50-70% cost reductions because SQL Mesh processes only changed data, not DBT’s expensive full rebuilds.

 

2. Break Free from DBT’s Tool Proliferation Problem

The insight: DBT’s democratisation without governance creates expensive technical debt that SQL Mesh’s structure prevents.

Shane identified a costly DBT pattern: “When DBT democratises without governance, we get 5000 models and massive waste. That’s expensive technical debt disguised as progress that SQL Mesh’s approach naturally prevents.”

This happens when DBT’s ease of use encourages rapid model creation without consolidation. Teams end up maintaining thousands of overlapping DBT transformations with unclear ownership and mounting maintenance costs.

SQL Mesh migration offers structural advantages here. Shane’s solution focuses on disciplined patterns: “SQL Mesh forces you to think about consolidation upfront. The migration process itself creates opportunities to clean up DBT’s accumulated chaos.”

How to approach SQL Mesh migration cleanup:

  • Use SQL Mesh migration as opportunity to decommission redundant DBT logic
  • Consolidate overlapping DBT transformations into efficient SQL Mesh patterns

Expected outcome: SQL Mesh migration typically eliminates 20-30% of redundant processing whilst establishing sustainable patterns that DBT struggles to enforce.

 

3. Apply Engineering Discipline Through SQL Mesh Migration

The insight: SQL Mesh migration forces software engineering practices that DBT’s flexibility often undermines.

Joseph’s approach during Avios’s SQL Mesh migration: “SQL Mesh fits software engineering workflows much better than DBT. The migration forced us to implement proper testing, code review, and deployment practices that DBT made optional.”

DBT’s analyst-friendly approach often leads to throwaway scripts rather than maintainable code. SQL Mesh migration creates natural checkpoints for implementing engineering discipline.

Joseph emphasised the team transition: “SQL Mesh is better suited for teams of engineers doing data rather than analysts doing engineering. The migration includes upskilling that strengthens the entire team.”

How to structure SQL Mesh migration for engineering discipline:

  • Require unit testing for all SQL Mesh transformations during migration
  • Use SQL Mesh migration to establish version control and deployment pipelines
  • Build data quality validation into SQL Mesh development workflows

Expected outcome: SQL Mesh migration delivers higher reliability and reduced maintenance overhead compared to DBT’s often ad-hoc development practices.

 

4. Plan SQL Mesh Migration Before DBT Renewal Pressure

The insight: The worst time to evaluate SQL Mesh migration is under DBT renewal pressure with escalating costs.

Aaron emphasised this critical timing issue: “Most teams wait until DBT pain is unbearable. The SQL Mesh migration project might have happened earlier, but teams don’t have time to explain how bad DBT has become.”

Crisis-driven SQL Mesh migration decisions rarely optimise for long-term success. Under DBT renewal pressure, teams make reactive choices rather than strategic SQL Mesh transitions.

The solution requires treating SQL Mesh migration evaluation as ongoing strategic work. Teams that maintain optionality can negotiate better DBT terms whilst planning measured SQL Mesh transitions.

How to prepare for strategic SQL Mesh migration:

  • Track DBT renewal dates 12-18 months in advance
  • Test SQL Mesh quietly using Mirror Mode validation
  • Build SQL Mesh migration business cases before forced decisions

Expected outcome: Teams gain negotiating leverage with DBT vendors whilst making strategic SQL Mesh migration decisions rather than reactive ones under pressure.

 

5. Make the Business Case for SQL Mesh Migration

The insight: Focus SQL Mesh migration business cases on productivity gains and cost impact, not technical features.

Joseph’s successful approach at Avios: “The SQL Mesh migration case was show, not tell. Here’s what SQL Mesh delivers compared to DBT processes. Here are the migration options. This is what we recommend.”

Business stakeholders don’t care about SQL Mesh’s technical elegance compared to DBT – they care about outcomes. The most compelling SQL Mesh migration cases translate architectural improvements into business language: faster delivery, reduced engineering overhead, predictable costs.

Shane provided tactical guidance: “Copy the format of your last approved business case. Don’t make the SQL Mesh migration proposal look completely different because you’ll confuse stakeholders.”

How to structure SQL Mesh migration business cases:

  • Demonstrate concrete productivity improvements over current DBT setup
  • Show SQL Mesh cost savings from reduced compute and maintenance overhead
  • Quantify current DBT pain points in terms of engineering time lost

Expected outcome: Leadership sees clear ROI from SQL Mesh migration rather than technical complexity arguments about DBT limitations.

Making SQL Mesh Migration Happen

Start with impact analysis rather than immediate SQL Mesh migration planning. Identify your three highest-cost DBT processes and map them to business value delivered. If you can’t explain why something costs £10k monthly in business terms, it’s a prime candidate for SQL Mesh migration benefits.

Sequence SQL Mesh migration for maximum impact with minimum disruption. Begin with DBT audit and cleanup – eliminate unused models, consolidate duplicates, establish monitoring. These changes create cleaner baseline metrics for measuring SQL Mesh migration success.

Key success metrics to track throughout SQL Mesh migration:

  • Cost per unit of business value delivered (not just absolute DBT vs SQL Mesh costs)
  • Engineering time spent on maintenance vs new feature development
  • Infrastructure costs as percentage of total data team budget

The goal is sustainable efficiency through SQL Mesh that scales with business growth rather than hitting DBT’s architectural walls.

Real-World Results: Avios SQL Mesh Migration

Joseph shared specific outcomes from Avios’s SQL Mesh migration, providing concrete evidence of what’s possible when teams move beyond DBT’s limitations.

“Engineers really like SQL Mesh compared to DBT. We’re growing rapidly. Data is delivering significant value across the business. We are decommissioning legacy DBT systems and moving everything into SQL Mesh processing patterns.”

The SQL Mesh migration wasn’t just technical – it required comprehensive organisational restructuring. They moved from DBT’s centralised bottlenecks to domain-based ownership, where teams own specific business areas rather than sharing DBT infrastructure.

The results compound over time. What started as SQL Mesh migration became an organisational capability that supports Avios’s 20% annual growth. Teams can onboard new partners and create new opportunities because SQL Mesh infrastructure scales efficiently rather than hitting DBT’s cost walls.

Joseph emphasised the holistic approach that made SQL Mesh migration successful: “We didn’t treat SQL Mesh as a silver bullet replacing DBT. We aligned business needs with SQL Mesh’s technical advantages for our situation.”

The key lesson: successful SQL Mesh migration requires changing how teams work together, not just replacing DBT technology. SQL Mesh enables better outcomes, but organisational design determines whether those capabilities translate into business results.

From DBT Optimisation Limits to SQL Mesh Strategic Control

Our discussion revealed a clear progression: optimise DBT where you can, but recognise when architectural constraints require SQL Mesh migration. Most teams exhaust DBT improvements before considering that their transformation platform itself creates the bottleneck.

What successful teams do differently: they start with impact analysis, build SQL Mesh migration cases early, and validate the new approach before committing to change. They treat SQL Mesh migration as business strategy, not just technical replacement of DBT.

The goal isn’t change for change’s sake – it’s sustainable growth through SQL Mesh that scales with business demands rather than hitting artificial limits imposed by DBT’s architecture.

Teams that master SQL Mesh migration gain competitive advantage through infrastructure that enables rather than constrains business opportunities.

Book Your ETL Escape Audit

Ready to evaluate whether DBT optimisation has hit its limits and SQL Mesh migration makes strategic sense? Our conversation with these experts confirmed what we see daily: DBT optimisation works until architectural constraints become the bottleneck, and SQL Mesh migration requires proper planning.

We’ll help you assess your current DBT situation, identify where optimisation has reached its ceiling, and show you how SQL Mesh’s incremental processing could impact your costs and roadmap. If migration makes sense, we’ll demonstrate how Mirror Mode validation eliminates traditional SQL Mesh migration risks.

You’ll get concrete benchmarks of your current DBT constraints and visibility into realistic improvements from SQL Mesh migration that you can present to leadership with confidence.

Book Your ETL Escape Audit →

Further Resources

#Blog

Data Leaders Digest

Stay up to date with the latest news and insights for data leaders.