AI Didn't Eat Data Engineering. It Split It in Two.

Dear Reader…

The first thunderstorm of the summer has just broken over the City of London, cracking a week of brutal June heat. Sheeting rain washes down the café window near Liverpool Street, smearing the light into long amber streaks. At a corner table, two men are having a conversation that, if you lean in, is being had in ten thousand other rooms around the world.

One of them is Christof. Twenty-odd years in data, a senior architect who has weathered every hype cycle since the first warehouses. Across from him is James, three years into a career that until recently felt like a rocket. Sharp, fast, the kind of engineer who can make a pipeline sing. He has recently been let go.

James is describing the interviews, the ones where he felt like a genius talking through his DAGs and his Spark optimisations, right up until someone asked how he would approach a global metadata lineage strategy, or agentic data stewardship, and watched his face go blank. He has the skills. He can out-code most of his old team. And he has walked straight into a glass ceiling he did not know was there.

Christof turns his espresso a quarter-turn on its saucer. "You build elegant pipes, James," he says quietly. "You just have no idea what is flowing through them, who owns it, or what happens when it breaks the trust of a million customers."

We are only the fly on the wall here. But I have been listening to versions of this conversation for three years, and I have come to think it is the most important one the profession is having.

The wrong verb

For two years, the question that kept arriving in my inbox, in different costumes, was some form of "is AI coming for my job?" We ran it more than once. The honest answer, I have slowly decided, is that the question had the wrong verb. Generative AI did not eat the data engineer's job. It split it in two.

It took the half that is mechanical, the plumbing and the syntax, the boilerplate transformation and the glue code, and it started doing that half itself, quickly and cheaply and at three in the morning without complaint. In the same motion it raised the price of the other half, the part that was never really about code at all. The judgment about what the data means. Who is accountable for it. What breaks, and who it hurts, when it is wrong.

James is not bad at his job, and that is the cruel part. He is very good at the half that is now being automated. The market spent fifteen years paying handsomely for exactly the skills he built, and then, in roughly eighteen months, quietly changed what it was buying and forgot to send the memo to those it affected most.

Two generations, two blind spots

Here is what makes the café conversation more than a story about one unlucky engineer. Christof and James are not just two people. They are two generations, and each learned the job with a hole in it shaped precisely like the other.

Christof's generation learned architecture the slow and expensive way. Through warehouses that took years to build and sometimes failed anyway. Through migrations that went wrong in front of the whole company. Through decades of living with the consequences of their own decisions. They understand governance in their bones, because they were in the room when the absence of it cost real money and real trust. What many of them are shaky on is the new tooling, the cloud-native, dbt-shaped, agent-assisted stack that James breathes without thinking about it.

James's generation came up the other way. Bootcamps and cloud consoles and a stack that abstracts away most of the pain Christof learned from. They can stand up in an afternoon what used to take a quarter. What they were never taught, because the curriculum did not cover it and the market did not demand it, is the "why". Why governance sits at the centre of every functioning platform rather than at the edge of it as paperwork. Why an architecture decision made casually on a Tuesday quietly determines whether an AI programme succeeds or stalls two years down the line.

For fifteen years none of this mattered, because the market rewarded James's half and largely ignored Christof's. The gap was invisible because it was never tested. Generative AI tested it. It closed the arbitrage almost overnight, and suddenly the thing the older generation knew and the younger generation skipped is the very thing that decides who keeps a seat at the table.

The uncomfortable truth, and I say this as someone who loves the craft, is that our universities and bootcamps have been training a generation with enormous care for the half of the job a model can now do. It is the gap I have ended up writing a whole book around, because it deserves more room than a newsletter can give it.

Three years from this chair

I have had an unusual vantage point on all of this. For three years I have watched the profession's mood change one week at a time, through the strange keyhole of what people open, what they click, and what they take the trouble to reply to.

In 2024, the tools were the story. Readers wanted to know how to build things, how to model, how to move data from here to there without it rotting on the way. In 2025 it became a parade of model releases and retrieval patterns and platform wars, everyone trying to keep their footing while the ground moved under them. By 2026 the centre of gravity had shifted again, to agents, to cost, to trust, and something changed in the questions themselves. They stopped being "how do I build X" and started being, more quietly and more often, "what should I become".

The pieces that consistently landed were never really about a tool. The ones that travelled furthest were about the career underneath the tool. The jobs pieces. The governance pieces. The unglamorous architecture comparisons that were secretly about power and lock-in and who gets to decide. For three years the audience was asking Christof's question without quite saying it aloud, and it took me an embarrassingly long time to notice that Christof's question was the one I had been answering all along.

What actually appreciates

So if the mechanical half of the job is falling in value and the architectural half is rising, the only sensible response is to work out, precisely, what appreciates, and to move towards it on purpose rather than by accident.

It is judgment, mostly. The ability to look at a system and know not merely whether it runs but whether it can be trusted. Governance in the real sense rather than the compliance sense: knowing what the data is, where it came from, who owns it, and what promises it is quietly making to people who will never meet you. Systems thinking in place of syntax. The discipline to put an autonomous agent inside sensible boundaries and hold it accountable, rather than being quietly replaced by one because you never learned to supervise it. And, above any single skill, the ability to translate, to stand between the business and the platform and make each one legible to the other.

You can draw the map, and it is not a mystery. It runs from foundation, understanding the framework that actually governs enterprise data, through architecture and design, where you learn to build systems that scale rather than merely work, through the operational lifecycle that stops velocity from eating reliability, to the hardest country of all, trust and ethics, where privacy law and responsible AI live and where the real dragons are. That is the road from mechanic to architect. James can walk it. Most of his generation can. Nobody ever told them it was there, or that it was the only road that still pays.

The Ultimate Guide for Usage-Based Pricing for SaaS and AI

Implementing usage-based pricing successfully requires more than just a pricing strategy.

Download this guide for practical advice and best practices when considering usage-based pricing. 

👉 Get your guide

The horizon

Back in the café, the coffee has gone cold and the storm is softening to a drizzle. Christof's line about elegant pipes and data swamps is landing on James differently now, not as an insult but as an invitation.

Because here is the reframe worth leaving you with, in the middle of your summer or your winter, wherever this finds you. The glass ceiling James walked into is not a ceiling at all. It is the floor of the next room, and the strange thing is that Christof is already standing in that room, quietly wishing he had some of what James takes for granted. The veteran knows the why and fumbles the new tools. The newcomer owns the tools and was never shown the why. Neither of them is the finished article. Put them at the same table and they very nearly are.

That is the next step, and it runs in both directions. James needs the architecture and the governance his training skipped, the half of the job that now decides everything and that no model will do for him. Christof needs the fluency James breathes without thinking, in the agent-shaped, cloud-native stack that is quietly rewriting his own trade. The mechanic has to become an architect. The architect has to keep pace with the machines. And the most useful thing either of them can do this year is stay at that table a little longer and teach each other.

Because the conversation I have been the fly on the wall for, across three years and a few hundred issues, was never really an argument between two generations. It was two halves of one job, finally being introduced. The profession is not splitting into winners and losers. It is being asked to mature in both directions at once.

The storm has passed. James and Christof are still talking, and somewhere in the middle of that conversation is the whole future of the work. The only question left on the table is no longer about one of them being ready. It is about the two of them being ready to meet in the middle.

I think they are. I think you are too.

That’s a wrap for this week

Happy Engineering Data Pro’s