Thinking Machines Lab launches Connectionism research blog
The inaugural post identified batch-size-dependent kernels, not floating-point concurrency, as the real cause of non-reproducible LLM outputs, and showed a fix producing identical completions across 1,000 runs.
- Ideas & essays
- Minor
Thinking Machines Lab, the research company founded by former OpenAI chief technology officer Mira Murati, published the first post on its research blog, “Connectionism,” roughly seven months after Murati founded the company. The post, by researcher Horace He, addressed why large language model inference produces different outputs for identical inputs.
The prevailing explanation for this non-determinism had been that floating-point arithmetic combined with concurrent GPU execution made bitwise-identical results unachievable at scale. The post argued this was not the true cause: common inference kernels are not “batch-invariant” — their numerical output depends on how many requests happen to be batched together at the moment they run, and because server load varies unpredictably, so does batch size, and so do results, even though any single kernel run is itself deterministic. The post demonstrated batch-invariant implementations of three core operations — RMSNorm, matrix multiplication and attention — reporting identical completions across 1,000 repeated inference runs, against roughly 80 distinct completions from a standard vLLM setup on the same prompt.
The finding had practical stakes beyond reproducibility: the post showed the same batch-invariant kernels also closed a gap between training and inference numerics that had introduced subtle errors into on-policy reinforcement learning, where a model’s own outputs compute its training signal. As Thinking Machines’ first public output, the post was a statement of the company’s research register — infrastructure-level and aimed at a problem other large labs shared but had not published on — ahead of any model release from the company itself.