The Data Leaders Digest – Seventh Edition

The best insights come from real conversations with real leaders making tough calls, solving messy problems, and figuring out how to scale in the real world.

Our mission is simple: turn these conversations into actionable guidance you can implement immediately. No fluff, no abstract strategies. Just practical takeaways from people doing the work.

---

This Month’s Focus – When DBT Optimisation Hits Its Ceiling

Our August LinkedIn Live session tackled a challenge that’s quietly frustrating data teams everywhere: “Tuning Won’t Cut Costs in Half – Time for a Technology Shift to SQL Mesh?”

Over 200 data leaders joined the conversation, building on insights from our previous sessions on ETL cost management and migration strategy. What emerged was a clear pattern: teams are discovering that optimisation has architectural limits no amount of tuning can solve.

The discussion featured three expert practitioners sharing real-world experience:

• Joseph Lane from Avios (IAG Loyalty), who led a comprehensive migration from DBT to SQL Mesh whilst supporting over £1 billion in annual transactions
• ✨ Shane Gibson from AgileData.io, bringing 35 years of perspective on technology cycles and what makes transformation succeed
• Aaron Phethean from Matatika, sharing how teams eliminate traditional migration risks

The conversation builds on the deeper, one-on-one discussions we’re having through the Data Matas podcast, where we explore these challenges in long-form conversations with data leaders from companies like Co-op, MoonPay, and Monzo. Together, they give us both the strategic context and tactical implementation detail needed to make better decisions.

Watch the full LinkedIn Live session featuring Joseph, Shane, and me discussing SQL Mesh migration in detail

---

When Optimisation Hits Its Ceiling

The conversation revealed something that cuts through industry noise: teams achieving 50-70% cost reductions aren’t necessarily better at optimisation. They’ve recognised when architectural constraints require different solutions.

Four Signals Your DBT Optimisation Has Hit Its Ceiling

1. Your Team Spends More Time on Workarounds Than Features

Joseph pointed out the core problem: “You start writing custom code around DBT to make it behave efficiently, or you end up with developers blocking each other, burning Snowflake credits, just by the way DBT works.”

Watch for these signs:

• Engineering time consumed by DBT-specific configuration issues
• Increasing complexity in managing development environments
• Full rebuilds becoming cost-prohibitive as data volumes scale

Do this week: Calculate hours spent on DBT maintenance versus new feature development. If it’s approaching 1:1, you’ve hit architectural limits.

2. Costs Scale with Data Volume, Not Business Value

DBT’s stateless architecture rebuilds everything from scratch. SQL Mesh’s state-aware processing changes this fundamentally.

Joseph explained it clearly: “SQL Mesh understands the state of objects it’s altering, so it runs much smaller queries. All of a sudden you’re processing a lot less data.”

Measure these ratios:

• Cost per unit of business value (not just absolute spend)
• Percentage of compute dedicated to unchanged data processing
• Incremental versus full refresh operations

Do this week: Audit your three highest-cost DBT processes. If you can’t explain why they cost £10k monthly in business terms, they need architectural review.

3. Tool Proliferation Creates Expensive Chaos

Shane identified a costly pattern: “When DBT democratises without governance, we get 5000 models and massive waste. That’s expensive technical debt disguised as progress.”

This happens when DBT’s ease of use encourages rapid model creation without consolidation. Teams end up maintaining thousands of overlapping transformations with unclear ownership.

Look for these problems:

• Models that exist “just in case” but aren’t actively used
• Duplicate logic across different team repositories
• Tests that run but don’t prevent actual issues

Do this week: Identify models created in the last six months that haven’t been queried. Start your cleanup there.

4. Business Cases Focus on Tools, Not Outcomes

Successful migrations happen when business need drives technology choice, not the reverse. Joseph’s approach worked: “Show, don’t tell. Here’s what SQL Mesh delivers compared to current processes. Here are the options. This is what we recommend.”

Reframe your approach:

• Lead with productivity gains and cost impact
• Translate architectural improvements into business language
• Use your organisation’s established business case format

Do this week: Draft a one-page impact statement linking current DBT constraints to specific business friction points.

---

The Strategic Migration Framework

Our panel discussion revealed how to approach the optimise-versus-migrate decision:

Start with impact analysis. Don’t jump straight to SQL Mesh evaluation. Map your current pain points to business value and engineering capacity.

Sequence for maximum impact. Begin with DBT cleanup. Eliminate unused models, consolidate duplicates, establish monitoring. This creates cleaner baseline metrics for measuring migration success.

Plan before crisis hits. The worst time to evaluate alternatives is under renewal pressure. Strategic teams build migration cases early and test options before pain becomes unbearable.

Test before committing. Successful migrations require proving performance with real workloads, not promises.

---

Real Results: What Avios Achieved Through SQL Mesh Migration

Joseph shared specific outcomes: “Engineers really like SQL Mesh compared to DBT. We’re growing rapidly. Data is delivering significant value across the business. We’re decommissioning legacy systems and moving everything into efficient processing patterns.”

Technical results:

• Eliminated infrastructure overhead of multiple development schemas
• Reduced compute costs through incremental processing
• Streamlined governance without complex role hierarchies

Organisational impact:

• Domain teams can move at business pace without central bottlenecks
• 20% annual growth supported by scalable data infrastructure
• Transformation enabled organisational capability, not just cost reduction

The transformation wasn’t just technical. They restructured from centralised bottlenecks to domain-based ownership. The results compound over time. What started as SQL Mesh migration became an organisational capability that supports Avios’s 20% annual growth.

The key lesson: successful migration requires changing how teams work together, not just replacing DBT technology. SQL Mesh enables better outcomes, but organisational design determines whether those capabilities translate into business results.

---

A Different Approach to Migration Risk

Traditional migrations create the problems teams fear: months of disruption, double billing, uncertain outcomes. Most teams stay trapped with expensive, inefficient systems because switching feels too risky.

We built Mirror Mode differently. Rather than forcing disruptive transitions, Mirror Mode runs new ETL systems in parallel with existing ones, validating performance before any cutover.

Teams get proof, not promises. They validate cost savings, performance improvements, and team productivity gains with their actual data before committing to change.

The difference between betting on vendor claims and making decisions based on demonstrated results.

---

Your Next Step

If your team is hitting optimisation limits, whether with DBT, other transformation tools, or broader ETL infrastructure, the path forward starts with understanding your current constraints clearly.

Are your costs scaling with data volume rather than business value? Is your team spending more time on maintenance than features? Are renewal pressures creating decision urgency?

We’ll help you assess where optimisation reaches its ceiling and identify the most strategic path forward. If migration makes sense, we’ll show you how to validate alternatives without traditional risks.

Book Your ETL Escape Audit

You’ll get concrete benchmarks of your current setup and clear visibility into improvements that you can present to leadership with confidence.

---

Further Resources

• Watch the full LinkedIn Live session featuring Joseph, Shane, and me discussing SQL Mesh migration in detail
• Download the ETL Escape Plan — our strategic framework for evaluating migration options before pressure hits
• Connect with our panellists: Joseph Lane, Shane Gibson, Aaron Phethean

Found this edition useful? Pass it on!

Help us reach more data leaders who need these insights:

• Forward this email to a colleague who’d benefit
• Join our growing community of 300+ data leaders getting monthly insights

Follow Matatika on LinkedIn | Subscribe for Insights | Visit Our Website