Dear Reader…

Survive Contact with Production

Part three of three, on building an AI pipeline you can actually trust.

Every AI pipeline works in the demo. The trouble starts the day real users arrive. Response times crawl as requests queue behind one another. The monthly API bill posts a number your manager forwards upward with a single question mark attached. And somewhere in the middle of it, the system begins returning confident nonsense, and nobody can say when it started, because nothing was watching. None of these are model failures. They are operational ones, and they are where most pipelines quietly die on the road from a successful proof of concept to a system a business will actually rely on.

We have built a foundation worth querying and a layer that turns it into output you can trust. This is the part almost no tutorial covers: keeping the whole thing alive, on budget, and honest, once it meets real traffic.

Getting a demo running is the easy twenty per cent. Serving it under load, governing what it spends, and being able to see inside it are the other eighty, and they are the architect's work. They are also, not by accident, exactly the parts a managed black box keeps out of your hands. Three open-source tools give them back.

Serve it: Ollama, and its ceiling

Ollama makes running an open-weight model locally almost trivial, which is why it is everywhere in prototyping. In production, that simplicity has edges, and you need to know where they are.

Its throughput is governed by three environment settings: OLLAMA_NUM_PARALLEL (how many requests run at once per model), OLLAMA_MAX_QUEUE (how many wait before it starts returning 503 errors, 512 by default), and OLLAMA_MAX_LOADED_MODELS (how many models sit in GPU memory together). Leave these at their defaults and you will either starve throughput or fall over under the first real spike.

The deeper limit is architectural. To protect latency for the users it is already serving, Ollama makes the rest wait, so time-to-first-token climbs as concurrency rises. A purpose-built serving engine such as vLLM does the opposite, using continuous batching to scale throughput as load increases, hundreds of tokens per second where Ollama manages tens. And when a model spills out of GPU memory into system RAM, hand-tuned back ends can be many times faster, and noticeably cheaper to run, than Ollama's automatic allocation.

The lesson is not to avoid Ollama. It is to prototype on it happily, and to know the moment your concurrency needs mean it is time to graduate to a production serving engine. Choosing when is the engineering decision, and making it deliberately is the mark of someone who has run one of these before.

Govern it: LiteLLM

Once you are serving models, some local, some from cloud providers, you need one point of control in front of them all. LiteLLM is that gateway. It exposes a single OpenAI-compatible endpoint and translates to over a hundred provider back ends behind it, so the rest of your code talks to one interface.

What earns its place is governance. It caps token spend and request rates per key and per team, so one runaway job cannot quietly drain the budget. It load-balances across identical endpoints and fails over to a backup provider or a local model when one is rate-limited or degraded, so a single upstream outage does not take you down with it. And it centralises key and credential management instead of scattering secrets through the codebase.

One production detail worth stealing: do not log every call synchronously to your database, or the gateway will throttle itself under load. Buffer the spend metrics and traces in a fast tier such as Redis, or a Kafka topic, and flush them asynchronously with background workers. That single choice is the difference between a gateway that scales and one that becomes your bottleneck.

Watch it: Langfuse

You cannot manage what you cannot see, and because these systems are non-deterministic, capturing only the final input and output is not enough. To know why an answer was good or bad you have to trace the steps in between: the retrieval, the reasoning, the tool calls. Langfuse is the open-source platform built for exactly that.

Its architecture is worth borrowing as a template for any high-volume observability. A relational store (PostgreSQL) holds accounts, keys and versioned prompt configurations. A columnar analytical store (ClickHouse) absorbs the flood of traces, spans and metrics, so dashboards can query billions of production events without slowing to a crawl. Caching and blob tiers handle queues and raw payloads. It is open-core: the tracing, datasets and prompt tooling are MIT-licensed, while enterprise features such as single sign-on and custom retention are paid, and it supports fully air-gapped deployments for regulated environments. One telling detail: it was acquired by ClickHouse in 2025, aligning its future squarely with that columnar-telemetry performance.

With it in place you can finally track cost per feature, debug a bad answer by replaying its exact trace, and catch drift before a user does.

Why not just let a platform do all this?

The fair objection to the whole series: why assemble this yourself when a managed platform will serve, govern and monitor for you? For some teams that is the right call, and there is no shame in it. But understand the trade. A managed black box hides precisely where your latency, your spend and your errors come from, prices it in ways you cannot fully audit, and makes leaving expensive. The entire argument here is that owning the stack means owning that visibility and control. These three tools are what turn "we think it is fine" into "we can see that it is."

The whole arc

That is the pipeline, end to end. A foundation that decides what the system can know. A build layer that makes its output trustworthy. And a production layer that keeps it fast, affordable and honest. Assemble all three from open-source parts and you are no longer renting someone else's judgement. You own the cost, the quality and the trust.

Which is the real point of the last three issues. The engineer who can call a hosted model is now everywhere. The one who can architect, build and maintain the whole pipeline, and stand behind every part of it, is the one this market has started to reward. That is the seat worth taking, and now you have the parts list.

A demo proves your pipeline can work. Production proves it can be trusted. Only one of those keeps the lights on.

That’s a wrap for this week

Happy Engineering Data Pro’s