The Warehouse Wars are Over. Welcome to the Catalog Wars.

Dear Reader…

The consensus arrived quietly, the way consensus usually does. By 2026, Apache Iceberg had become the open table format that every major platform agreed to read and write. Snowflake, Databricks, AWS, Google — all of them ship Iceberg support. The pitch is by now familiar to anyone who has sat through a vendor deck: your data, your object storage, an open format, queryable from anywhere.

The demo does work. The problem starts about two weeks into production, when someone asks who is allowed to see column seven.

A directory of Parquet files in an S3 bucket is inert. It is storage, not a database. Something has to tell a compute engine which metadata pointer represents the current valid state of a table, and whether the querying principal has the right to read it. That something is the catalog. By commoditizing the file format, the industry did not eliminate the lock-in it had spent a decade complaining about. It relocated it — from the storage layer, where it was visible and much-discussed, to the governance layer, where it is neither.

This is a story about where the leverage went.

What the REST spec actually standardizes

The Iceberg REST Catalog specification, introduced in Iceberg 0.14 back in 2022, is the document everyone points to as proof the layer is open. In fairness, it delivers more than skeptics allow. It defines a language-agnostic HTTP API for namespace and table operations, snapshot commits, and view management. It specifies OAuth2 bearer-token authentication. And — a detail routinely lost in the retelling — it specifies credential vending: on a table load, the catalog can return short-lived storage credentials scoped to that table's path, so no engine holds a standing key to the bucket. Remote request signing goes further still.

So the common claim that the spec covers "communication only" is not quite right, and it is worth getting right, because it changes where you should be looking for the trap.

What the spec does not define is the authorization model. Who holds which grant. How roles inherit. Column-level masking. Row filters. Tag-based policy. Lineage. The spec standardizes how a catalog hands out access; it says nothing about how a catalog decides whom to hand it to. That gap is not an oversight — it is the commercial surface area, and vendors have built on it accordingly.

The practical consequence is the part architects underestimate. A vendor can implement the REST API to a standard-compliant polish and still hold you, because the 5,000 grants, masking rules and policy tags you authored are expressed in a proprietary model with no export path. The Parquet files remain open. Your governance model does not port. Changing engines means rewriting the security posture of the enterprise, and that is a project no CISO signs off on casually.

Apache Polaris: the neutral option, with an operator's bill attached

If the mandate is neutrality, Polaris is currently the strongest claim to it. Snowflake initiated the project and, with Dremio, donated the code to the Apache Software Foundation in August 2024. It spent roughly eighteen months in the Incubator and graduated to Top-Level Project status in early 2026 — the ASF announced the graduation, alongside Apache Gluten, on 5 March 2026, following an incubator vote recorded as 27 in favour with no objections.

Graduation is a governance fact, not a marketing one, and it is the fact worth internalizing: the project now has independent PMC governance and a contributor base spanning competing vendors, which is materially harder for any single company to reverse than a feature lead is to close.

The feature set has moved fast. Generic table support — cataloging Delta Lake, Hudi and Paimon alongside Iceberg — reached general availability in 1.3.0 in January 2026, which also brought Open Policy Agent integration and native Iceberg metrics reporting. Federation of external catalogs, experimental at 1.0, has since matured; 1.5 added BigQuery Metastore federation, and releases have continued on a roughly monthly cadence through 1.7.0 in August 2026.

The caveat is unglamorous and non-negotiable: unless you consume Polaris through a managed service, you are operating your organization's control plane yourself. Not a service. The service. When the catalog is unreachable, no engine can resolve the current state of any table, and every pipeline and dashboard in the company stops at once. Neutrality is a real property. It is also an on-call rota.

Your Competitors Just Hired an Entire GTM Team. No Humans.

AI agents are doing the work of full GTM teams. Pipeline, content, customer service. The startups using them aren't waiting for headcount approvals.

Get the free Practical Guide to Agentic GTM for Startups and see exactly where to start.

Unlock the Playbook

Project Nessie: brilliant idea, contested future

Nessie's premise remains one of the most elegant in the field. Rather than treating governance purely as grants, it brings Git semantics to the catalog: branch the entire lakehouse state, run transformations in isolation, validate against contracts, and merge only when checks pass. A corrupt load is not a restore-from-backup exercise; it is a revert to a prior tag.

Two caveats, one cultural and one strategic.

The cultural one is well known: branching is a paradigm shift for the analyst who only wants to run a SELECT. Governance-as-version-control is coherent for a large engineering organization and alienating for a business consumer.

The strategic one is the one that belongs in a due-diligence memo. Dremio, the company behind Nessie, stated publicly in October 2024 that it would consolidate on Polaris and fold Nessie's capabilities — catalog-level versioning, Git-like semantics, multi-table transactions — into that project, with Nessie itself eventually retired. That work has been proceeding since. Adopting Nessie today as a strategic destination is therefore a bet against its own sponsor's stated roadmap. Adopting the pattern — branch, validate, merge — and waiting for it to land in Polaris is the more defensible position.

Unity Catalog: the tightest integration, and a correction

Databricks Unity Catalog is the case for depth over neutrality: automated lineage, predictive optimization, liquid clustering, credential vending, all wired directly into the execution environment. Databricks open-sourced a version in June 2024 and placed it with the LF AI & Data Foundation, where it remains — per the project's own site — a sandbox project. The open-source implementation trails the managed tier substantially, and that gap is the honest caution to raise.

But one criticism now circulating is simply out of date, and repeating it will cost you credibility in an architecture review. Unity Catalog does expose a standard Iceberg REST Catalog endpoint — /api/2.1/unity-catalog/iceberg-rest — and, per Databricks' own documentation, it supports reads and writes from external Iceberg clients such as Spark, Flink and Trino against Unity Catalog managed Iceberg tables, with credential vending. It is in Public Preview, gated behind enabling external data access on the metastore and the EXTERNAL USE SCHEMA privilege, but it exists and it works; Snowflake-to-Databricks write interoperability over that endpoint is documented.

The real asymmetry runs the other way, and it is more interesting than the one people keep repeating. Iceberg tables governed by another catalog are classified in Databricks' documentation as foreign tables: readable, refreshable on demand, but not writable from Databricks, and not eligible for credential vending. Outbound, Unity Catalog is a well-behaved REST server. Inbound, it is a reluctant client. That is the shape of the gravity well — and it is a design choice, not a missing feature.

The two-catalog temptation

Every nervous architect eventually proposes the hedge: run the platform-native catalog, sync it to a neutral one, keep options open.

Don't. Two catalogs writing over one data lake is a split-brain waiting for a Tuesday. Engine A commits through Catalog A, Engine B through Catalog B, and within a day the metadata pointers disagree, grants drift apart, and the lakehouse state is silently wrong — silently being the operative word, because nothing throws an error.

Note that the vendors themselves have converged on the alternative rather than the hedge. Polaris federates external catalogs behind a single REST surface. Snowflake's catalog-linked databases sync against Unity Catalog. Databricks federates foreign tables for reads. Every one of those designs preserves a single commit arbiter and makes everything else a consumer. That is the pattern to copy: one catalog owns the commit, everything downstream connects to it over REST and reads.

The scorecard

Storage is free. Execution is commoditizing. The catalog is the last place in the stack where a vendor can still charge rent, which is precisely why the marketing there is loudest and the specifications quietest.

An open format proves your files are portable. Only a governance plane you can actually move proves your platform is. Those are not the same claim, and the difference shows up three years later, in the size of the rewrite.

That’s a wrap for this week

Happy Engineering Data Pro’s