Thinking Machines Lab Releases Inkling: 975B-Parameter Open-Weights Multimodal MoE with 41B Functional Parameters and Controllable Thinking Effort
The Thinking Machines Lab has just been released The Inklingtheir first model trained from scratch, weights on, opens up nicely for Tinker. The lab sets it as a base for customization.
What is an Inkling?
Inkling is a Mixture-of-Experts converter with a total of 975B and 41B active parameters. It supports a context window of up to 1M tokens. Pre-training includes 45 trillion tokens of text, images, audio, and video. Input accepts text, images, and audio; the output is UTF-8 text only.
The research team also previewed the Inkling-Small, a 276B parameter MoE with 12B active parameters. It matches or surpasses its larger sibling in most benchmarks, and its weights come when testing is complete. Because customization/customization is an important distinction, design is very important here.
Inside Architecture
The model structure consists of a 66-layer decoder-only transformer with a small MoE feed core. Each MoE layer manages 256 routing specialists and 2 joint specialists. Six routing experts activate each token, and both shared experts activate all tokens. A sigmoid-based route handles the selection, using an auxiliary-loss-free load balancing bias. The averaged and distributed scores are normalized together, and then used to measure the combined results. The design of the MoE closely follows the DeepSeek-V3.
Attention is drawn from the meeting. Sliding window and universal layers are combined in 5:1 ratio with 8 KV heads. The position uses relative position embedding rather than RoPE, which the lab reports is more expressive. Short convolutions are used after key and value guesses, and on residual branch results.
Multimodality does not encode. The noise comes in as dMel spectrograms, and the images become 40×40 pixel patches with a four-layer hMLP. A lightweight embedding layer works both, and the video is processed in conjunction with the text tokens.
Training was carried out by Muon with large matrix weights and Adam for other parameters, on NVIDIA GB300 NVL72 systems. Post-training bootstrapped from SFT on synthetic data, including data generated by Kimi K2.5. Multi-computer to asynchronous RL, rated for the past 30M release, improves the log sequentially everywhere. That RL run also produced the main control point for the model.
