From Tool Mastery to Systems Design: How Data Engineers Actually Grow

Published on May 22, 2025

Most data professionals hit a ceiling when they’ve mastered the tools but haven’t learnt to think in systems. This transition from writing queries to designing architectures, separates analytics engineers from those who burn out maintaining fragile pipelines.

 

The Hidden Ceiling in Data Engineering Careers

For many data professionals, the early years of their career are defined by mastering tools. SQL, dbt, Looker, Power BI – learn them, use them, ship dashboards. But what happens when the business needs more than just a dashboard? When systems have to scale, models must evolve, and the team starts to feel the weight of their own architecture?

In a time when AI hype dominates boardroom agendas and self-serve data is seen as the endgame, the role of the analytics engineer is quietly shifting. And with it, the skills required to be effective.

In this episode of the Data Matas podcast, Oleg Agapov, Senior Analytics Engineer at Hiive, shared a refreshingly clear-headed take on what it really means to grow from junior to senior in this field. If you lead a data team, build analytics platforms, or are trying to grow your own career, here’s what you’ll take away:

  • Why mastering tools is only step one in your growth journey
  • How to recognise when it’s time to move from building queries to designing systems
  • What data modelling maturity actually looks like in practice
  • How to use AI as a partner – without outsourcing your thinking
  • The critical role discovery plays in sustainable engineering
  • How to mentor junior engineers with clarity and context

 

Meet Oleg Agapov

Oleg Agapov is a Senior Analytics Engineer at Hiive, a fintech startup that’s modernising the secondary private stock market. With over 15 years in the industry, Oleg’s journey from data analyst to systems thinker wasn’t scripted by formal training but shaped by solving real problems in fast-paced environments.

His work sits at the intersection of engineering rigour and business practicality. He’s responsible for building scalable, self-serve analytics platforms while mentoring junior engineers on how to go beyond tools.

“When you start, it’s all about writing the queries and learning the syntax. But that doesn’t scale. As you grow, your job becomes thinking about systems, not code.” – Oleg Agapov

He brings this mindset into everything – from mentoring emerging talent to challenging assumptions about how analytics teams should work.

 

Why Scaling Analytics Starts with Rethinking Roles

Most organisations still treat analytics engineering as an advanced data analyst role. The assumption is: once you’re fluent in SQL and dbt, you’re ready to lead projects. But that underestimates the shift in thinking required to become a systems designer.

At Hiive, Oleg faced the challenge of building self-serve analytics across a complex, three-sided marketplace. The business logic wasn’t just difficult – it was in constant flux.

“We had to refactor our models entirely in the first six months. You can’t just yolo your SQL into dbt forever. It doesn’t hold.”

The real risk? Engineers spend years perfecting syntax and speed, only to burn out when faced with changing business needs. The fix isn’t more dashboards. It’s teaching engineers to think like architects.

 

How to Grow Beyond the Tools and Deliver Systems That Scale

Key Takeaways to implement

1. Design systems, not queries

Analytics engineers must grow beyond writing isolated queries to designing resilient systems. This means thinking in components and interfaces, not individual tables. Each model should be reusable, well-documented, and designed with future change in mind.

To implement:

  • Refactor legacy dbt models quarterly
  • Define naming conventions and interface contracts
  • Document ownership and intended use cases for models

2. Use AI to review your work, not replace it

Rather than relying on AI to generate business logic, use it to challenge your assumptions and spot issues you missed. AI can function as a second pair of eyes, not a hands-free developer.

To implement:

  • Use tools like Cursor to generate tests and run quality reviews
  • Avoid the temptation to create models without your own understanding
  • Keep a log of AI assumptions to prevent downstream errors

3. Align sync schedules with real decision-making

Not all data needs to be synced in real-time. Oleg’s team reduced cost and instability by syncing based on usage patterns—not assumptions.

To implement:

  • Identify which decisions each data sync supports
  • Shift low-use datasets to daily or weekly syncs
  • Use access logs to validate actual business usage

4. Prioritise discovery before delivery

Engineers are often pushed to deliver before fully understanding the problem. Oleg’s team builds in dedicated discovery time to align stakeholders and avoid building the wrong solution fast.

To implement:

  • Block out discovery sprints for all new initiatives
  • Ask clarifying questions before jumping into delivery (e.g., “What decision will this dashboard support?”)
  • Share diagrams and draft models to build shared understanding early

5. Mentor with clarity, not complexity

Junior engineers need frameworks—not hand-holding. Oleg uses whiteboarding, clear structure, and candid feedback to accelerate growth.

To implement:

  • Run weekly mentoring sessions focused on real project examples
  • Encourage juniors to lead reviews and present their models
  • Provide repeatable templates for discovery, modelling, and review

6. Build trust before introducing AI

Trust is the foundation for any AI-driven analytics. If users don’t trust the data, no tool or assistant will fix that.

To implement:

  • Ensure auditability in all metric calculations
  • Let AI run in shadow mode before showing results to users
  • Make trust a deliverable, not an afterthought

 

Making the Shift to Scalable Engineering

To embed these practices, Oleg recommends sequencing your efforts over six to eight weeks:

  • Weeks 1-2 – Review model structure and sync schedules for opportunities to simplify.
  • Weeks 3-4 – Introduce peer reviews for all new dbt models. Focus on naming, clarity, and reuse.
  • Weeks 5-6 – Run a “stop-doing” workshop to identify pointless syncs and AI overreach.

Track these metrics:

  • % of data models reused across projects
  • Pipeline failure rates before/after sync optimisation
  • Review turnaround time using AI as a reviewer

Tools to consider:

  • dbt for modelling and documentation
  • Metabase or Lightdash for tracking usage
  • Claude, ChatGPT or Cursor for AI review prompts

 

Inside Hiive: Real Results from Refactoring

In just six months, Oleg and his team transformed Hiive’s analytics foundation.

By reducing full syncs to delta loads and auditing sync frequency, they cut compute costs and eliminated several unstable jobs. More importantly, they reduced noise for engineers and improved trust in BI outputs.

“We’re still early, but now our models align with how the business thinks. That’s the real win.”

 

Your Next Move

If you’re stuck maintaining fragile models, firefighting sync failures, or feeling like AI is creating more questions than answers – it’s time to shift your mindset.

Start thinking like a system designer:

  • Build modular, intentioned models
  • Use AI to sharpen, not shortcut
  • Sync only what supports a decision
  • Mentor with clarity, not complexity

Want more? Listen to the full conversation with Oleg Agapov on the Data Matas podcast to hear how his journey can inform yours.

 

Frequently Asked Questions

How does the transition from query writer to systems thinker impact ETL pipelines?

When engineers evolve to systems thinking, ETL pipelines become more resilient and adaptable to business changes. Instead of patching together queries that break under pressure, systems-oriented engineers build modular pipelines with clear interfaces, comprehensive testing, and thoughtful incremental loads that reduce processing costs while improving reliability.

What skills should I prioritise to grow from a junior to senior data engineer?

Beyond technical skills, focus on understanding the business domain deeply, improving your ability to translate requirements into system designs, and developing communication skills to align stakeholders. Senior engineers spend more time on discovery and architecture than coding, ensuring solutions solve the right problems and can evolve as the business changes.

How can data teams optimise their ETL costs without sacrificing performance?

Following Oleg’s approach, audit your sync schedules against actual business usage patterns. Many teams sync data hourly by default when daily or weekly would suffice. Implement incremental loading where possible, and regularly review access logs to identify which data actually drives decisions versus what’s synced out of habit.

Is AI replacing the need for skilled data engineers?

No – AI is changing the role rather than replacing it. As Oleg demonstrates, successful teams use AI as a review partner and acceleration tool, not a replacement for human judgement. The most valuable skills now involve knowing how to frame problems for AI assistance while maintaining critical oversight of business logic and data integrity.

 

Dig Deeper

Looking to optimise your ETL systems without disruption? Learn how Matatika’s Mirror Mode can help you validate new approaches with zero risk 

#Blog

Data Leaders Digest

Stay up to date with the latest news and insights for data leaders.