Executive summary
On November 5, 2026, Baidu released and open-sourced ERNIE-5.5-Flash-Base, a sparse Mixture-of-Experts foundation model with 280 billion total parameters and 14 billion active per token. It was pretrained from scratch on 33.5 trillion tokens, uses a 128K training context, supports up to 1 million tokens for fine-tuning and inference, and ships as an untrained pretraining endpoint rather than a chat-ready product.
What happened
Baidu published ERNIE-5.5-Flash-Base on November 5, 2026, together with the pretraining checkpoint weights. The company positioned the release as a common starting point for domain adaptation, instruction tuning, reinforcement learning and agent training, not as a finished conversational model.
That distinction matters. The released weights have not gone through mid-training or post-training. Baidu says a post-trained ERNIE 5.5 Flash will follow. Developers who download the checkpoint inherit both the flexibility and the cost of building the final model themselves, including alignment and safety work.
The release ships with mainstream serving and training support: FP8 and BF16 weights on Hugging Face, a dedicated vLLM plugin, an SGLang deployment cookbook, and a Megatron-Bridge fine-tuning example in NVIDIA NeMo. Baidu also published a technical blog covering architecture, training infrastructure and evaluation methodology.
Key facts
| Item | Detail |
|---|---|
| Release date | November 5, 2026 |
| Publisher | Baidu |
| Model | ERNIE-5.5-Flash-Base |
| Architecture | Sparse Mixture-of-Experts (MoE) |
| Total parameters | 280 billion (280B) |
| Active parameters per token | 14 billion (14B) |
| Pretraining data | 33.5 trillion tokens (33.5T) |
| Pretraining context | Up to 128K |
| Fine-tuning / inference context | Up to 1 million tokens (1M) |
| Network shape | 43 Transformer layers, hidden size 4096, 64 attention heads |
| Attention design | Hybrid sparse attention combining HCA (heavily compressed attention) and MLA (multi-head latent attention) |
| Expert routing | Multi-Router Latent MoE, multiple routers deciding in a 2048-dimensional latent space |
| Training objective | ECHO, reusing the last backbone layer to predict future tokens without an extra MTP module |
| Training system | Distributed Muon optimizer, FP8 mixed precision, FlashMask-4 attention kernel |
| Open formats | FP8 and BF16 weights; vLLM plugin, SGLang cookbook, Megatron-Bridge example |
| Follow-up | A post-trained ERNIE 5.5 Flash is planned |
Why a Base model, and what it costs
A base checkpoint and a shipped model are separated by the entire post-training pipeline. Pretraining compresses language, knowledge, code and long-context statistics into parameters. Mid-training and post-training add instruction following, conversational behavior, tool use, preference alignment and safety constraints.
By releasing the pretraining endpoint, Baidu hands developers the decision about which direction to continue training. Teams building vertical models can inject domain corpora; agent teams can run reinforcement learning on tool use; evaluation researchers can compare post-training recipes from an identical initialization. The trade-off is that users must budget for continued training and safety alignment, and cannot treat the checkpoint as a deployable product.
The sequencing matches how Baidu has handled the ERNIE line before: foundation first, product later. For the ecosystem, open base weights lower the barrier to entry for teams that could never pretrain from scratch. For Baidu, it creates a shared starting point that ties developers to its model family.
Three structural changes built around efficiency
Baidu describes efficiency per token as the design thread, expressed in attention, expert routing and the training objective.
First, hybrid sparse attention reduces long-context cost. The model combines HCA and MLA. Of its 43 layers, 37 attend over sequences compressed by a factor of 128 and carry long-range history; the remaining 6 use MLA-DSA to preserve fine-grained retrieval. The immediate benefit is lower KV-cache and attention compute, while a high-resolution path remains available when exact spans must be located. Baidu also adds low-rank gated attention and virtual head attention (VHA), and says gating projection parameters drop by 75%.
Second, multi-router latent MoE stabilizes expert selection. Conventional MoE relies on a single router to assign tokens to experts; a routing error concentrates compute on a few experts. Multi-Router Latent MoE lets several independent routers decide jointly in a 2048-dimensional latent space, reducing the impact of a single misstep while keeping expert compute in check.
Third, ECHO moves future-token prediction back into the backbone. Common multi-token prediction (MTP) adds a dedicated module outside the main network, so multi-step loss mostly trains the new module. ECHO instead reuses the last backbone layer, applying multi-step loss directly to the parameters that generate the next token. As a side effect, that layer can serve as a built-in draft model for speculative decoding without extra parameters.
Two smaller choices round out the design. A roughly 200,000-vocabulary R2L tokenizer splits numbers right-to-left in groups of up to three digits, targeting arithmetic and digit-heavy text. A MultiMax output head applies learnable segmented temperatures to suppress improbable long-tail tokens while retaining diversity among plausible candidates.
Training data and engineering
Pretraining used 33.5 trillion tokens, with three data directions deliberately built out:
| Data type | Processing | Target capability |
|---|---|---|
| Web | Multilingual crawling, high-fidelity text extraction, model-driven multi-dimensional quality filtering | Knowledge coverage and language breadth |
| Code | Deduplication and grading of repository and web code, plus commits, issues, pull requests, code review, generated tests and CI-verified fixes | Code engineering and execution validation |
| Long documents | Filtering natural long documents, concatenating semantically similar distractor text, synthetic retrieval and citation QA, multi-hop QA | Long-context comprehension and retrieval |
On the infrastructure side, the distributed Muon optimizer merges cross-machine communication for dense and MoE parameters into a single collective, reducing latency and jitter at thousands-of-GPUs scale. The FlashMask-4 attention kernel is built on CuTe-DSL, supports SM90 and SM100 architectures, and handles head dimensions up to 576 (qk) / 512 (v), which allows direct use on HCA's 512-dimensional wide heads.
Evaluation scope and caveats
Baidu evaluated ERNIE-5.5-Flash-Base on 30 benchmarks spanning seven capability groups: general, knowledge, science, reasoning, multilingual, code and long context. It compared the model with DeepSeek-V4-Flash-Base (284B-A13B), Hy3-preview-Base (295B-A21B) and MiMo-V2.5-Base (310B-A15B). Baidu states that all four models used their publicly released checkpoints under a single evaluation framework.
According to Baidu's results, ERNIE-5.5-Flash-Base reaches state-of-the-art performance among base models of the same size, with particular strength in reasoning, code, academic subjects and long-context tasks. Baidu's published figures put HumanEval+ 12.8 percentage points above DeepSeek-V4-Flash-Base and MBPP+ 14.3 points above it.
These numbers are vendor-reported. A shared framework is not the same as independent third-party reproduction, and post-training differences can change downstream behavior substantially. Treating "same-size SOTA" as proof that a finished product is stronger would overstate the result. A more reasonable use is to shortlist candidates, then re-test on your own task set.
What it changes for developers
First, a 1M-token inference window puts long documents, code repositories and multi-turn agent traces into one context. The real cost depends on KV-cache compression and serving-framework support. HCA's 128x compression is Baidu's architectural answer; actual throughput still needs measurement under specific vLLM or SGLang configurations.
Second, base weights plus mainstream serving support fill the gap between pretraining from scratch and fine-tuning a finished model. For teams with domain data and post-training capability but no GPU cluster, this is the more realistic path. For teams with no training capability at all, waiting for the post-trained release or calling an API is cheaper.
Third, competition among Chinese foundation models has shifted from parameter count to active parameters per token, long-context cost and reusability of the base. ERNIE-5.5-Flash-Base at 280B/14B active, DeepSeek-V4-Flash at 284B/13B active and Hy3-preview at 295B/21B active sit in the same bracket, so efficiency and data quality become the natural comparison axes.
Official sources
- Xinhua report: Baidu open-sources ERNIE 5.5 Flash Base
- Baidu ERNIE technical blog: ERNIE 5.5 Flash Base Release
- Hugging Face weights (FP8): baidu/ERNIE-5.5-Flash-Base
- Hugging Face weights (BF16): baidu/ERNIE-5.5-Flash-Base-BF16
- vLLM plugin: baidu/vLLM-ERNIE
- SGLang cookbook: ERNIE-5.5-Flash Cookbook