How the first half of 2025 rewrote the rules of data engineering

Dear Reader…

As we gather around our metaphorical fireplaces this December, it's worth reflecting on a year that shook the foundations of data engineering like no other. Much like the traditional carol's accumulating gifts, 2025 delivered a cascade of AI developments, each building upon the last until the entire landscape had transformed beyond recognition.

If 2024 was the year of breathless experimentation with generative text, 2025 was when the rubber met the road. The theoretical promises of artificial intelligence collided violently with the physical constraints of enterprise infrastructure, forcing a complete rethink of how we build, govern, and deploy data systems. What follows is the first half of our journey through the year's defining moments.

The First Day of AI Christmas: The Efficiency Revolution

The year opened with a bang. On 28 January, DeepSeek R1 landed like a meteor strike on the AI landscape, shattering the prevailing orthodoxy that state-of-the-art performance required billion-dollar budgets and nuclear-scale energy footprints. Industry observers compared it to the Hadoop revolution of 2006, when scarcity became commodity overnight.

The numbers were staggering. While traditional frontier models commanded development costs exceeding $100 million, DeepSeek R1 was built for approximately $5.6 million. Training infrastructure requirements plummeted from clusters of over 10,000 GPUs to a mere 512. Perhaps most critically, the hardware needed to run these models shifted from exclusive, supply-constrained A100/H100 clusters to consumer-grade GPUs.

For data engineering teams, this "budget democratisation" reversed the polarity of AI deployment. Throughout 2023 and 2024, teams had been shackled to centralised, API-dependent architectures where data travelled to distant model providers, incurring latency, egress costs, and sovereignty risks. DeepSeek R1 enabled migration towards decentralised, on-premise, and hybrid deployments. Small teams, previously priced out of the generative AI market, suddenly possessed the capability to deploy state-of-the-art reasoning models within their own secure perimeters.

The shift demanded new skills. Engineers pivoted from managing massive cloud expenditures to optimising local inference, building hardware-aware pipelines with strict memory budgets, and architecting "air-gapped" intelligence layers where sensitive data could be processed without ever traversing the public internet. The era of brute force was over. The age of the "lean stack" had begun.

Second Day: Research at the Speed of Thought

As February arrived, so did a deluge of faster, better models from global players. But the most transformative productivity shift wasn't a new model architecture. It was a functional capability: AI-powered deep research.

OpenAI's "Deep Research" announcement, quickly adopted by Perplexity and Google's Gemini 2.0, represented a fundamental leap from search to synthesis. Traditional search engines returned lists of links requiring manual traversal. Early LLMs, while conversational, hallucinated freely and lacked citation density. Deep research agents bridged this gap, planning multi-step research trajectories, analysing heterogeneous sources, identifying conflicting narratives, and generating structured, cited reports.

OpenAI's Sam Altman quantified the economic implications: research accounts for roughly 5% of all economically valuable activity performed using the internet. By compressing weeks of information sifting into minutes, deep research tools offered efficiency gains measured in the trillions of dollars.

For data engineers, the impact was immediate. The traditional bottleneck of "understanding the data domain" was brutally compressed. These agents could hand off structured findings to downstream tools. A data engineer could task an agent to "identify all major GDPR fines in 2024, extract violation type and penalty amount, and format as CSV," effectively automating the ETL process for unstructured web data. Data catalogues, often barren and outdated, began to be populated by agents that could autonomously research the lineage and definition of obscure database columns.

However, this capability introduced new risks. The "black box" nature of synthesis required engineers to build verification loops, systems that would spot-check citations against source text. The engineer's burden shifted from finding data to verifying synthesis, a theme that would culminate in a crisis later in the year.

The Future of Shopping? AI + Actual Humans.

AI has changed how consumers shop, but people still drive decisions. Levanta’s research shows affiliate and creator content continues to influence conversions, plus it now shapes the product recommendations AI delivers. Affiliate marketing isn’t being replaced by AI, it’s being amplified.

Download the full report

The Third Day: The Retrieval Reckoning

By March, a painful realisation had settled over the data engineering community: DIY Retrieval-Augmented Generation (RAG) was a trap. The "RAG 1.0" era (2023-2024) was characterised by custom-built solutions. Teams spent thousands of hours managing vector stores, writing manual chunking strategies for PDF parsing, and building bespoke embedding pipelines.

The result was often fragile, high-maintenance systems. Total cost of ownership was driven not by compute, but by continuous engineering labour. "Drift" in embedding models, the complexity of updating vector indices, and the difficulty of debugging retrieval failures turned RAG into a technical debt generator.

The response was the RAG 2.0 revolution, exemplified by Google's Gemini File Search and similar fully managed systems from platform vendors. These productised the entire retrieval category, abstracting away the operational complexity of file storage, optimal chunking strategies, embedding generation, vector storage, and dynamic context injection.

This shift forced a fundamental "build versus buy" reassessment. One critical breakthrough was the integration of hybrid search. Pure semantic search often failed in enterprise contexts when dealing with "out of domain" vocabulary: proprietary product numbers, internal project codenames, or specific acronyms lacking semantic meaning in general training data. Managed platforms solved this by combining vector similarity search with traditional token-based keyword search.

Data leaders began mandating managed RAG for all non-differentiating, general-purpose enterprise knowledge applications (HR bots, IT helpdesks). This freed engineering resources for "strategic reservation" projects, custom RAG builds limited to applications representing core competitive advantages or requiring absolute data sovereignty.

RAG 2.0 transformed retrieval from a code-heavy engineering challenge into an architectural configuration task. It allowed teams to stop debugging vector indices and start focusing on data quality, the "garbage in, garbage out" principle applied to the age of AI.

The Fourth Day: The Vibe Coding Paradox

Throughout the first quarter, a cultural phenomenon known as "vibe coding" went viral in Silicon Valley. The term described a new style of programming where developers acted as "directors" rather than "writers," using natural language prompts to guide AI models like GitHub Copilot, Cursor, and Replit to generate entire applications.

The promise was intoxicating. CEOs and media outlets heralded a future where "10 engineers can do the work of 100," accelerating product development cycles and reducing barriers to entry. Andrej Karpathy characterised it as a fundamental shift in the developer's role.

However, by April, cracks began to show. A series of high-profile data breaches were traced back to platforms built via vibe coding. Security experts discovered embarrassingly basic vulnerabilities: SQL injection flaws, hardcoded credentials, and unencrypted data transmission in code that had been generated by AI and blindly accepted by developers.

The issue wasn't that AI was incapable of writing secure code, but that "vibe coders," often lacking deep foundational knowledge, couldn't distinguish between functioning code and secure code. The democratisation of coding had outpaced the democratisation of security literacy.

For data engineering teams, vibe coding created a dual trajectory of risk and opportunity. The role of the senior data engineer evolved into that of an "AI curator" or reviewer. Senior engineers spent less time writing code and more time rigorously vetting AI-generated logic. The "time to generate" had dropped to near zero, but the "time to review" skyrocketed as code volume exploded.

The trend created a generation of "junior engineers" capable of rapid prototyping but lacking fundamental understanding of system architecture, data consistency, or secure coding principles. This "hollow skill set" posed a long-term risk to organisational capability. Vibe coding accelerated the accumulation of technical debt. Data pipelines built this way became "black boxes" that were impossible to debug when they inevitably failed.

The trend served as a stark warning: While AI could democratise creation, it could not democratise comprehension. The resulting security and maintenance crisis became a primary driver for the enhanced governance trends seen later in the year.

On the Fifth Day We Got: The Golden Thinking Machine

In late March, Anthropic released Claude 3.7 Sonnet, a model representing a quantum leap in AI architecture. Prior to this, models were generally categorised as either "fast and cheap" for simple tasks or "slow and smart" for complex reasoning. Claude 3.7 dismantled this dichotomy by introducing a hybrid reasoning architecture.

The model operated in two distinct modes: a standard rapid-response mode and an "extended thinking mode." In the latter, the model mimicked human cognition by taking additional time to deliberate, plan, and critique its own logic before generating a response. This process was visible to the user, showing step-by-step thinking that led to higher accuracy.

Crucially, the model gave engineers unprecedented control over the "thinking budget" via the API. Developers could allocate up to 128,000 tokens solely for the model's internal reasoning process, tuning cognitive depth based on query complexity: spending more "thought tokens" on a complex SQL optimisation problem and fewer on a simple data classification task.

While the reasoning capabilities were impressive, the most tangible impact for data professionals was in data visualisation. Claude 3.7 demonstrated the ability to ingest raw datasets and autonomously generate sophisticated, interactive dashboards. In one demonstration involving healthcare data, the model analysed the dataset, identified key trends, wrote appropriate visualisation code (using libraries like D3.js or Plotly), and executed it to present a fully functional dashboard, all without human intervention.

This capability lowered the barrier to entry for advanced analytics. Non-technical stakeholders could now generate insights visually without waiting for BI developers or data scientists. The model's low hallucination rate (2.3%) and extended thinking capabilities challenged the long-held dogma that a human must always be in the loop for code generation.

This trend marked the point where AI ceased to be just a text generator and became a reasoning engine capable of end-to-end analytical workflows, from raw data to visual insight.

Sixth Day: The Terminal Takeover

By June, the vibe coding trend had matured into something far more powerful and technical: the terminal takeover. Anthropic's launch of Claude Code represented a paradigm shift from simple code completion in an integrated development environment to an "agentic partnership" directly within the command line interface.

Unlike previous assistants that only saw the open file or a small context window, Claude Code possessed "agentic search" capabilities. It could navigate the file system, comprehend the intricate web of dependencies (Python scripts, SQL queries, dbt models, Dockerfiles, Terraform configurations), and understand how they fit together.

It didn't just read code; it understood project structure. It could see that a change in a Python ingestion script would break a downstream SQL view and a Tableau dashboard, offering holistic awareness that no previous tool could match.

This was a force multiplier for data engineers, whose work typically involves navigating a labyrinth of heterogeneous scripts and infrastructure configurations. Anthropic's internal testing revealed that Claude Code could successfully complete complex engineering tasks in a single pass that would typically demand over 45 minutes of focused manual work.

A data engineer could paste a cryptic error message from a failed dbt run into the terminal. Claude Code would autonomously traverse the project, identify the broken dependency, check the git history to see who changed it, propose a fix, and run the test suite to verify the solution. The tool excelled at managing Terraform and Kubernetes configurations, areas notorious for subtle syntax errors and dependency hell.

This automation freed engineers from the grunt work of syntax and dependency checking. The cognitive load of remembering every variable name and file path was offloaded to the agent, allowing human engineers to focus on high-value architectural design and system optimisation.

The terminal takeover was the precursor to the fully autonomous agents that would dominate the latter half of the year. It moved AI from being a "copilot" that made suggestions to being a "crew member" that executed tasks.

As we reach the halfway point of our journey through 2025, a pattern emerges. Each development built upon the last: the efficiency revolution enabled decentralisation, which fuelled agentic workflows, which demanded better retrieval systems, which exposed security gaps, which required smarter reasoning, which enabled autonomous execution.

The first six months laid the groundwork for what was to come. But if you think the first half was transformative, wait until you see what the second half delivered.

Join us in part two as we explore how context windows exploded, governance became paramount, and AI moved from the world of words into the physical realm.

That’s a wrap for this 6 days

Happy Engineering Data Pro’s