Kafka and Materialize Are Being Assigned Jobs They Do Not Perform

Dear Reader…

The RAG project always begins the same way, and it always demos well. A few hundred PDFs, a chunking strategy someone copied from a blog post, an off-the-shelf vector store. The chatbot summarizes the Q3 earnings report. The executives nod. The budget clears.

The trouble arrives about a quarter later, when the business stops asking about documents. Now it wants the model to review security footage, mine the call-center audio, cross-reference the CAD files. And the stack that summarized a PDF turns out to have been built on an assumption nobody wrote down: that "data" means text and numbers, and that everything else is an opaque file sitting in a bucket while some external application does the actual work.

That assumption is now the constraint. Rewiring around it is a genuine architectural project. But the reference diagram circulating to describe that project — lakehouse, Kafka, Flink, Materialize, done — does not survive contact with the documentation of its own components. Three of the four boxes are being asked to do something other than what they do.

What is actually being built

The idea itself is sound and the money is real. LanceDB raised a $30M Series A in June 2025, led by Theory Ventures with participation from CRV, Y Combinator, Databricks Ventures and RunwayML, explicitly to build what it calls the multimodal lakehouse. Its customer list — Midjourney, Runway, Harvey — is not a list of companies with a text-only data problem.

The distinction worth holding onto, because vendors blur it: Lance is the format; LanceDB is the platform. Lance is an open columnar format optimized for random access as well as scan, with a file spec, a table spec and a lightweight catalog spec. LanceDB is a lakehouse implementation on top of it. The company itself draws the analogy — the relationship is Iceberg to Polaris, or Delta to Unity Catalog — and in November 2025 moved Lance toward independent community governance, listing UBS, Netflix, Uber, Databricks, AWS and ByteDance among the parties around it. That governance split is the single most important due-diligence fact in the space, and it is the one the enthusiasm skips.

The technical argument is straightforward. Today a video lives in S3, an ML pipeline generates embeddings, a vector database stores them, and a relational catalog holds the metadata. Three systems, three consistency boundaries, three sets of credentials. A unified format collapses that: one hybrid query can run a metadata filter, a full-text match and a vector similarity search in a single plan, over one copy of the data.

A correction about the deletion argument

The pitch usually lands on a compliance example: a user deletes their account, and now you must orchestrate a distributed delete across three systems or face GDPR and EU AI Act penalties.

The architectural point is fair. The legal citation is not.

The right to erasure is GDPR Article 17. The EU AI Act does not create one. Its Article 10 imposes data-governance and quality obligations on training, validation and test sets for high-risk systems, and Article 2(7) states plainly that the Act operates without prejudice to GDPR — it layers on top, it does not extend deletion rights.

If anything, the AI Act pushes the other way. It requires retention of technical documentation and logs for accountability, which is why practitioners now write about reconciling the two regimes rather than citing them as one. Invoking the AI Act as a deletion mandate in a compliance review will get the whole architecture second-guessed by someone in legal, which is a bad way to lose an argument you were winning on the merits.

"Batch is deprecated" is not true, and the people saying it are running batch

The claim that batch processing has been largely deprecated for AI workloads is the weakest sentence in the standard pitch. It is contradicted by the flagship deployments of the exact architecture being advocated.

Netflix's widely cited setup pairs Ray for elastic batch inference with LanceDB for zero-copy data evolution, curating petabyte-scale multimodal datasets. Batch is not vestigial there; it is the compute-intensive core. Embedding backfills, re-embedding after a model swap, training-set curation, periodic index rebuilds — these are batch by nature, and they are where the GPU bill lives.

The defensible version of the claim is narrower and more useful: retrieval freshness has bifurcated from training throughput. A support agent answering questions about an order status genuinely cannot tolerate a 24-hour-stale index. A model curating a training corpus does not care. Architectures that fail are usually the ones that picked one latency posture for both.

Where the reference architecture breaks

Kafka does not carry video. This is the error most likely to detonate in production. Kafka's default broker limit, message.max.bytes, is 1MB, and outsized messages are widely treated as an anti-pattern: they inflate broker JVM memory pressure, and a single large record can evict thousands of hot records from page cache. The ceilings on managed tiers are hard architectural constraints, not tuning suggestions — Azure Event Hubs will not go past 1MB, and Confluent Cloud Basic/Standard and MSK Serverless cap around 8MB.

The correct pattern has a name: claim check. The media lands in object storage; Kafka carries a small reference plus metadata; the consumer fetches the payload. Say "Kafka ingests the video stream" in a design review and you have told the room you have not sized a broker.

Flink is where inference belongs — and only recently. This part of the pitch has become true faster than most people realize. ML_PREDICT arrived in Flink SQL in version 2.1, letting a streaming job call a remote model; Flink 2.2, released December 2025, added VECTOR_SEARCH for similarity lookup inline in the stream. On Confluent Cloud there are dedicated AI_EMBEDDING and ML_PREDICT functions, with models registered as first-class SQL resources via CREATE MODEL. The pre-2026 pattern — a Python microservice wedged between Kafka and the store — genuinely does collapse into SQL now.

Two footnotes the marketing omits, both of which will bite. Remote inference means network calls inside your dataflow, so batching, timeouts and error handling become the job. And Flink's exactly-once guarantees cover sink writes; LLM side effects are at-least-once. You will pay for some tokens twice.

Materialize is not an embedding engine. This is a category error, and it is in nearly every version of this article. Materialize is a Postgres-compatible streaming database built on Timely Dataflow that performs incremental view maintenance with strict serializability over relational streams. It is superb at keeping a complex join or aggregate continuously correct as inputs change. It does not vectorize video frames. Listing it as an alternative to Flink for computing embeddings suggests the diagram was assembled from vendor landing pages rather than from documentation.

The honest placement: Flink (or Confluent's managed Flink) for inference and embedding on the stream; Materialize for the derived relational state — the continuously correct feature values, entitlements and aggregates the retrieval layer filters against. Different jobs. Both legitimate. Not substitutes.

The vector-database objection, taken seriously

Your engineers will ask why they can't just buy a managed vector database. The standard answer — that standalone vector stores are an architectural dead end — is too strong, and stated that way it is easy to dismiss.

The real critique is narrower and survives scrutiny: embeddings in an isolated system sit outside your governance plane. They are not covered by the RBAC that governs the source records, they drift from the transactional data, and keeping them aligned means owning a synchronization pipeline that no one wants to be on call for. A vector store is a excellent index. It is a poor system of record.

But apply that same test symmetrically, because it is the question a good reviewer will ask. A Lance-based lakehouse standing outside your Iceberg tables and your catalog's grants recreates precisely the fragmentation it was brought in to solve — you have simply moved the seam. The multimodal lakehouse earns its keep when it is inside the governance perimeter, not when it becomes a second, better-marketed silo.

The scorecard

Parsing a PDF is a commodity, and it was a commodity before the current wave. The scarce thing is a system that ingests a live stream, generates embeddings without a bespoke microservice holding it together, joins the result against governed relational data, and hands an LLM context that is fresh where freshness matters and cheap where it doesn't.

That system is buildable in 2026. It is not buildable from the diagram everyone is drawing, because the diagram has Kafka moving video and Materialize computing vectors. Get the boxes right first. The architecture argument is strong enough that it doesn't need the embellishment.

That’s a wrap for this week

Happy Engineering Data Pro’s