Moonshot AI Releases Kimi K3: 2.8 Trillion Parameter Open MoE Model With Kimi Delta Attention and 1M Content

Moonshot AI has just been released For me K3. 2.8-trillion parameter model with native view and 1 million token context window. Moonshot calls it the world’s first open 3T-class model.

What is Kimi K3?

The Kimi K3 is a small Mixture-of-Experts (MoE) model built on two architectural revisions. Those are Kimi Delta Attention (KDA) and Attention Residuals (AttnRes). Both change the way information flows through the sequence length and depth of the model. K3 targets long-horizon coding, knowledge work, and thinking.

The Moonshot team says K3 is the first open model to access 2.8 trillion parameters. For nine of the last twelve months, Kimi models set the upper limit of open model sizes.

The Moonshot also goes straight to where the K3 sits. Overall performance still trails the more powerful models, the Claude Fable 5 and GPT 5.6 Sol. Across Moonshot’s evaluation suite, the K3 consistently outperforms other models tested.

Architecture Under

Kimi Delta Attention (KDA) is a hybrid linear attention technique. Moonshot claims to enable recording up to 6.3x faster for 1 million token instances.

AttnRes works on another, deeper axis. It selectively finds representations in depth rather than uniformly accumulating them. Moonshot claims AttnRes delivers nearly 25% training efficiency at less than 2% additional cost.

Sparsity is the third safeguard. K3 uses Stable LatentMoE, successfully unlocking 16 out of 896 experts. In that sparsity, routing and optimization become challenges of the first order. Quantile Balancing derives expert allocation directly from router-score quantiles. That eliminates heuristic updates and sensitive hyperparameter estimation. Per-Head Muon extends Muon by preparing the attention heads independently. Sigmoid Tanh Unit (SiTU) and Gated MLA improve the control of activation and selective attention respectively.

Refined training and data recipes accompany those structural changes. Together they produce an average of 2.5x better than Kimi K2.

Those decisions lead to worship. K3 uses quantization training from the SFT stage onwards. It uses MXFP4 switches with MXFP8 activation for broad hardware compatibility. The Moonshot team recommends a supernode configuration with 64 or more accelerators. Because KDA poses new challenges for startup preservation, Moonshot has contributed to implementation in vLLM.