The LLMOps stack: the four layers of production LLM tooling, and why they leak

Why OpenTelemetry is the dividing line

The single axis practitioners argue about most is OpenTelemetry support, because it decides portability. OpenTelemetry (OTel) is the vendor-neutral standard for traces, metrics, and logs, and it now has a GenAI semantic convention for LLM spans. A tool that is OTel-native lets you instrument once and point the data at any compatible backend; if you outgrow the vendor, you keep your instrumentation. A tool with a proprietary-only SDK does the opposite: your traces are written in its dialect, and switching means re-instrumenting your whole application.

It is worth being precise, because the marketing blurs three tiers. Some tools are built on OTel as the native data model (Traceloop / OpenLLMetry is the reference implementation; Arize Phoenix uses OpenInference on top of OTel; Datadog LLM Observability and Langfuse v3 align to the GenAI convention). Some ingest OTLP but ship their own SDK as the primary path. And some are late or partial: LangSmith added OTel export well after launch and remains LangChain-centric; tools like Lunary use non-OTel SDKs that create switching risk. When a vendor claims "OpenTelemetry support," ask which tier. In the matrix below we only mark a tool OTel-native where we could confirm it from the tool's own documentation.

The comparison matrix: OpenTelemetry, self-host, and pricing model

Nineteen tools across the four layers, on the five axes that decide interoperability and cost. yes and no are read from each vendor's own documentation; partial means tier-limited, proxy-based, or unconfirmed-native. Pricing model is the column to read at scale, not the headline price. This table is published as an open dataset under CC-BY (see methodology).

Highlighted rows are tools that span multiple layers cleanly, the leakage the thesis describes. Pricing models and OTel status reflect each vendor's public documentation as of June 2026 and change often; verify on the vendor's own page before a purchase decision.

This comparison is published as an open dataset (CC-BY) with a permanent DOI: DOI 10.5281/zenodo.20738671. Browse the full dataset landing page or download the machine-readable JSON.

Layer 1: the gateway, and the question of who holds the key

The cleanest way to split this layer is the one most articles skip: who holds the provider API key. A true gateway sits in your request path and you keep your own keys, getting fallback, load-balancing, caching, and budget control. An aggregator resells model access; you buy their credits, they hold the keys, and you get one bill across many models. They solve different problems, and conflating them is how teams pick the wrong one.

Portkey is the strongest open-source true gateway: an Apache-2.0 core that routes across a very large model catalog, self-hostable, and OpenTelemetry-native, with guardrails and observability layered on top, which is why it shows up as a multi-layer tool in the matrix. Portkey open-sourced its gateway after operating it at large production scale (see Portkey's own statements for current figures). LiteLLM is the OSS SDK-and-proxy that normalizes 100-plus providers into the OpenAI format; it is the pragmatic default when you want full control and will carry the self-host operations yourself. Cloudflare AI Gateway and Kong AI Gateway are the right picks if you already live on those platforms (edge caching for Cloudflare, plugin governance for Kong) and a tax otherwise. On the aggregator side, OpenRouter resells 300-plus models on a flat percentage of credit purchases, and AIMLAPI has quietly assembled one of the broadest catalogs in the category: by its own account, 400-plus models across every modality, text, image, video, and audio, behind one OpenAI-compatible endpoint and one bill. If your problem is "I want to try many models, including non-text, without managing many keys," an aggregator is the honest answer; if your problem is "I need a production control plane I govern," a true gateway is.

Layer 2: observability, where the OTel line is sharpest

This is the layer most teams should buy first, because you cannot fix what you cannot see, and it is where OpenTelemetry support most changes your future options. Langfuse is the open-source default: MIT-licensed, fully self-hostable for free, spanning tracing, evaluation, and prompt management, and OpenTelemetry-native since v3 (with an OTLP backend), so it interoperates instead of locking you in. The honest caveat is that its unit-based billing (a trace, an observation, and a score each count) can surprise high-emission teams, and the v3 self-host is heavier to run (Clickhouse, Redis, object storage). Arize Phoenix is the other strong open-source pick, built on OpenInference over OTel, excellent as a developer and eval tool, though production RBAC pushes you toward paid Arize. Traceloop / OpenLLMetry is the reference OTel implementation but is a library, not a turnkey platform: you bring your own backend and UI. LangSmith is polished and tightly integrated with LangChain, but it is closed, its OTel export arrived late and partial, and per-trace-plus-seat pricing can spike. Datadog LLM Observability is the natural choice only if you already run Datadog, with the usual cost and lock-in trade-off. Helicone takes an observability-first, gateway-style proxy approach that is fast to adopt but introduces a hop in your path.

Layer 3: evaluation, the layer that decides "is it good"

Evaluation is where the category is moving fastest, and where the leakage is most visible: most eval tools also do observability, because you cannot score production quality without the traces. Maxim is the most full-stack of the cohort: experimentation, agent simulation across thousands of scenarios, evaluation, observability, and even its own open-source gateway, Bifrost (Apache-2.0), which Maxim's published benchmarks put at roughly 20 microseconds of added latency at 5,000 requests per second. The simulation-across-scenarios approach to agent eval is a genuine differentiator few eval tools match; the trade-off is the usual breadth-versus-depth question when one tool competes on many fronts. Braintrust is the polished closed-source eval-plus-observability platform with strong CI ergonomics. Promptfoo is the open-source, local-first, pure-eval and red-team CLI: the right pick for pre-deploy testing in CI, with no live production monitoring of its own. DeepEval / Confident AI pairs an OSS eval framework with a commercial cloud for the production layer. The decision here is offline-versus-online: if you mainly need pre-deploy testing, Promptfoo; if you need a unified online-plus-offline platform, Maxim or Braintrust; if you already run Langfuse or Phoenix, start with their built-in evals before adding a fourth tool.

Layer 4: guardrails, enforced in real time

Guardrails are the runtime enforcement layer: input and output checks for prompt injection, PII, toxicity, and factuality, applied on the live request rather than after the fact. Guardrails AI is the open-source validation framework with a Hub of community validators and OTel telemetry; its strength is output-quality validation more than adversarial defense. NeMo Guardrails (NVIDIA) is the programmable option, using its Colang language to script rails, which is powerful but carries a learning curve. Lakera is the API-first commercial guard focused on prompt-injection and PII defense, with cost that scales with traffic. Prediction Guard takes a distinct path: rather than bolting a guard library onto someone else's model, it offers privacy-conserving hosted inference with guardrails native to the path (PII, injection, toxicity, factuality), running on Intel Gaudi 2 accelerators, an early pioneer of running paying-customer LLM workloads on that hardware. Its founder, Daniel Whitenack, hosts the long-running Practical AI podcast, which tells you the posture: opinionated, security-first, built for teams that cannot send data to a public API. First-party options exist too (AWS Bedrock Guardrails, Azure AI Content Safety, Vertex Model Armor), but they are cloud-locked and weaker on adversarial benchmarks.

Which layer, and which tool, for your situation

Build down the request path, but rarely all at once. The order that works for most teams: observability first (you need to see), then a gateway (control and fallback), then evaluation (a quality bar before you scale), then guardrails (enforcement as exposure grows). Match the move to your situation:

The honest per-layer verdicts

• Best open-source observability

  layer 2

  Langfuse -- the rare platform you can fully self-host for free that also became OpenTelemetry-native, so it interoperates instead of trapping you. Watch unit-based billing on the cloud tier and the heavier v3 self-host.

• Best self-governed gateway

  layer 1

  Portkey for an OSS, OTel-native, production-proven control plane you hold the keys for; LiteLLM if you want a thinner proxy and will run the ops yourself.

• Best breadth across models and modalities

  layer 1 (aggregator)

  AIMLAPI -- 400-plus models across text, image, video, and audio behind one OpenAI-compatible endpoint and one bill. The honest framing is that it is a marketplace, not a control plane you govern.

• Best agent / full-stack evaluation

  layer 3

  Maxim -- experimentation, simulation, eval, and observability in one, plus an OSS gateway (Bifrost). The trade is breadth versus depth. Promptfoo remains the cleanest OSS pick for pure pre-deploy CI testing.

• Best privacy-first guardrails

  layer 4

  Prediction Guard -- guardrails native to a private inference path rather than bolted onto a public API, for teams that cannot send data out. Guardrails AI is the OSS default for output validation.

Methodology and conflict disclosure

If you build one of these tools and want to check that we have represented it fairly, or that a pricing or OTel detail has moved, we would genuinely welcome the correction. The goal of this page is to be the one map of the category that is not written by a vendor ranking itself first.

Adjacent reading on Nesyona: the best AI agent frameworks that sit above this stack, the best AI coding assistants for the teams building it, and the best local and self-hosted AI for the privacy-first end of the spectrum.

Frequently asked questions

The bottom line

LLMOps is not a leaderboard, it is a stack. Map your problem onto the four layers, buy the one you need first, and let OpenTelemetry connect the rest so you are never re-instrumenting to escape a vendor. Read the pricing model, not the headline number, because per-event and per-seat tools scale very differently at production volume. And remember the thesis: the layers leak, so the leading tool in one layer often saves you a purchase in the next. Start with observability (Langfuse is the open-source default), add a gateway you govern (Portkey or LiteLLM) or an aggregator for breadth (AIMLAPI), put a quality bar in CI (Promptfoo, or Maxim for agents), and enforce at the edge (Prediction Guard or Guardrails AI) as your exposure grows.

Sources

1. OpenTelemetry, GenAI semantic conventions and OTLP specification, opentelemetry.io (accessed June 2026).
2. Langfuse documentation and pricing, langfuse.com (v3 OpenTelemetry support, MIT license, self-host requirements).
3. Portkey gateway repository and OpenTelemetry docs, portkey.ai and github.com/portkey-ai/gateway.
4. Maxim and Bifrost gateway repository and published benchmarks, getmaxim.ai and github.com/maximhq/bifrost.
5. AIMLAPI model catalog and documentation, aimlapi.com.
6. Prediction Guard documentation and Intel developer write-up on privacy-conserving LLM inference on Intel Gaudi 2.
7. Arize Phoenix (OpenInference) and Traceloop / OpenLLMetry repositories.
8. Datadog LLM Observability and its GenAI OpenTelemetry semantic-convention blog.
9. OpenRouter, LiteLLM, Kong AI Gateway, Cloudflare AI Gateway, Braintrust, Promptfoo, DeepEval / Confident AI, Guardrails AI, and NeMo Guardrails official documentation and pricing pages.
10. TechCrunch, on Anthropic's August 2025 acqui-hire of the HumanLoop team.