Building Data Platforms That Actually Solve Business Problems

1. The Golden Rule: Analyst Before Engineer

When a business decides it needs to be more “data-driven,” the immediate instinct is often to hire a Data Engineer to start building. Teddy argues this is premature.

Teddy’s first advice to any stakeholder is: hire a Data Analyst first.

The analyst acts as the essential translator, bridging the gap between business needs (e.g., “We need to sell more pergolas”) and the technical requirements (“We need an ingestion pipeline that joins sales and inventory data”). By defining the what and the why first, you ensure the engineer doesn’t build the wrong thing faster.

2. Diagram Before You Code

Teddy stresses the importance of visualization as the first phase of any project, especially when inheriting a legacy “mess.”

He never starts with a line of code. Instead, he uses diagrams to modelize and visualize the proposed architecture, starting with a wide view (Source A to Dashboard B) and then zooming into specific components like the data lake or transformation layers.

“A diagram helps you be sure of what you’re doing and helps your client understand your intention.”

This step forces all parties to align on the logic and prevents costly rework down the line.

3. The Unexpected Art of Finding Clients

Moving away from the typical online hustle, Teddy shares that his best clients often come from the real world. He advocates for attending meetups and simply being open to conversation.

He shares a story about literally finding a client who needed an SQL database while renting a bike. The lesson: don’t neglect real-life interaction. Being present and communicative in your local community is often more effective than endlessly optimizing your LinkedIn profile.

4. Solve the “Richard” Problem

Many companies run on legacy systems—often a sophisticated, macro-heavy Excel or Google Sheet built years ago by a single person (“Richard”). These systems are often too late and too painful to modify once the company grows.

Teddy notes that the conversation starts when these systems become inconsistent or too slow. The key to moving past this is establishing clear benchmarks and outcomes. Success isn’t just about replacement; it’s about providing a measurable benefit like reduced latency or increased frequency of data delivery.

5. Documentation is Mandatory (But You Don’t Have to Write It)

Documentation is notoriously boring and often neglected. Teddy, however, deems it mandatory, especially for a consultant or a future colleague who might inherit the system.

His game-changing advice for the modern team: Use LLMs.

“Documentation is mandatory. The fact that you’re writing it, not really. Give it to your LLM.”

Using generative AI to write or summarize documentation based on code and historical issues is the most effective way to maintain a living, up-to-date knowledge base, freeing engineers to focus on building.

Putting it all together

The most efficient data platforms are not the most complex; they are the most intentional.

Start by defining the business need with an analyst. Plan the solution with a diagram. Build it simply. And document every decision not just for yourself, but for your future colleague. This intentional approach minimizes cost, maximizes trust, and makes data a true business asset.

Your Next Move

• Take one of your current technical projects and try to explain its necessity and outcome to a stakeholder without using any technical jargon.
• Model the architecture of your next pipeline using a simple diagram before writing a single line of code.
• Delegate the first draft of your next code README to an LLM.

Dig Deeper 🎧

Listen: Building Data Platforms That Actually Solve Business Problems
📺 Watch: https://youtu.be/TEBKIV2Jiq8
👤 Connect with Teddy Bernays on LinkedIn
🌐 More episodes: Matatika.com/podcasts

#Blog #DataEngineering #Freelance #DataStrategy #AI #Diagrams

#Blog #analytics #Cloud Cost Optimisation #Data Engineering #Data Infrastructure #ETL Tools