One-sentence summary
For context-heavy (agentic) workloads, IBM research data shows that self-hosted llm-d 1.0 distributed inference can cost 5 to 10 times less per token than equivalent managed model APIs; llm-d's strengths are open-source control, high prefix-cache hit rates (about 98%) and zero-downtime operations, while managed APIs offer hands-off operation and elastic quotas.
Object Introductions
- llm-d 1.0: an open-source distributed LLM inference project founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, now governed by the CNCF (Cloud Native Computing Foundation); version 1.0 was released on October 20, 2026, orchestrating inference across multiple model servers including vLLM and SGLang.
- Managed model APIs: closed-source inference services hosted by model or cloud vendors (for example, model APIs from OpenAI or Anthropic), billed per token, with infrastructure operated by the vendor.
Core Parameter Comparison
| Dimension | Self-hosted llm-d 1.0 | Managed model API |
|---|---|---|
| Cost per million tokens (blended) | About USD 0.11 (IBM Research quantum error correction use case) | N/A: pricing varies by vendor; IBM states 5-10x higher per token for equivalent workloads |
| Prefix-cache hit rate | About 98% (IBM Research use case) | N/A: not uniformly disclosed |
| Peak throughput (demo) | 6.6M output tokens/min (GLM 5.2, 753B parameters, 3,000 concurrent agents, zero preemptions) | N/A: depends on vendor quotas and load |
| Zero-downtime upgrades | Supported (new in 1.0: hot standby + rolling updates) | Provided by vendor |
| Data and model control | Open source (CNCF-governed), self-hosted, data stays in domain | Closed-source hosted; data passes through vendor path |
| Hardware options | Own or multi-cloud GPU/TPU (AMD Instinct, NVIDIA, Google TPU) | Bound to vendor infrastructure |
Key Differences
- Cost structure: IBM's official press release (October 20, 2026) states that on context-heavy agentic workloads, llm-d deployments cost 5 to 10 times less per token than equivalent commercial API pricing, with the largest savings on input-heavy traffic patterns.
- Context management: llm-d uses prefix caching and cache-aware routing to reduce recomputation; the IBM Research use case reported about 98% prefix-cache hit rate and a blended serving cost of about USD 0.11 per million tokens.
- Operations model: llm-d 1.0 adds multi-tenant QoS, zero-downtime operations and a v1 configuration API for production; managed APIs offload operations to the vendor but limit customization and control.
- Open ecosystem: llm-d has more than 500 contributors from over 25 organizations; Google Cloud has integrated it into Google Kubernetes Engine (GKE).
Buying Advice
- If your team already has GPU clusters, workloads are dominated by long-context agentic tasks, and data control and cost matter, evaluate self-hosting llm-d 1.0 first (using IBM's published cost and throughput data).
- If your team lacks infrastructure operations capability, workloads fluctuate and you need to launch quickly, managed APIs are more suitable.
- A hybrid approach works: llm-d officially supports running on cloud GPUs (CoreWeave Kubernetes Service, GKE), letting you choose self-hosted or managed per workload in the same cluster.