Eliminating Nondeterminism in LLM Inference with Integer-Only Arithmetic

A while back, Thinking Machines published Defeating Nondeterminism in LLM Inference, which explained why greedy decoding isn’t deterministic in production. In theory, at temperature 0 a model always picks its single most likely next token, so the same prompt gives the same answer every time. In practice, when they sent one prompt 1,000 times, they got 80 different answers.¹

The problem is that floating-point addition is not associative. $(a + b) + c$ and $a + (b + c)$ can round differently. A GPU sums thousands of numbers for every token, and the order it sums them in depends on things like how many other requests are in the batch. Change the order, change the rounding, and occasionally change which token wins.

Their fix was to pin the order: rewrite the kernels so that a given GPU always adds numbers in the same order, no matter what else is in the batch. That works, but only on that GPU with those kernels. An H100 and an A100 use different matrix-multiply kernels, a CPU uses different instructions entirely, and each one rounds its own way. Run the same model on all three and you still get three different answers.

I always wondered, what if we just didn’t use floats? Integer addition is exact, so the order can’t matter. If every operation in the model used integers, the output should be bit-identical everywhere: any batch size, any GPU, any CPU, any vendor.

Eventually I found time to work on this during a SpaceXAI hackathon. The project, DetLLM, worked well enough to become a paper at two NeurIPS 2026 workshops.² In this post I’ll walk through why LLM inference is nondeterministic, why integers fix it, and how to run every part of an LLM (matmuls, softmax, normalization, rotary embeddings) without a single floating-point operation.

The animation above is the whole result. The same prompt runs in nine configurations across three machines; the fp16 runs drift apart from the first token, and at step 457 one of them forks onto different text. The integer runs never move.

Why are GPUs not deterministic?#

Every float addition rounds#

A floating-point number stores a fixed number of significant digits and a separate exponent. When you add two numbers with very different exponents, the small one has to be shifted to line up with the big one, and the digits that fall off the end are gone. Every addition rounds. The consequence is that $(a + b) + c$ is not always equal to $a + (b + c)$:

>>> (0.1 + 1e20) - 1e20
0.0
>>> 0.1 + (1e20 - 1e20)
0.1

An LLM’s forward pass is mostly sums: every matmul, softmax, and layer norm adds up long lists of numbers, rounding at each step. The order of those additions is up to the kernel. Below, the same eight fp32 values are summed two ways: left to right, and as two halves combined (what a “split-K” matmul kernel does).³

Why kernels don’t always add in the same order#

The natural guess is that GPU threads finish in a random order, so the sum lands in a random order. That can happen with atomic adds, but it’s mostly not what goes on in an LLM forward pass. Run the same matmul on the same inputs a thousand times and you get the same bits a thousand times. Most inference kernels are run-to-run deterministic.

What they aren’t is batch invariant. cuBLAS doesn’t have one matmul kernel; it has a catalog, and it picks one based on the problem’s shape. A small batch can’t keep every streaming multiprocessor busy, so the library splits the reduction into chunks and sums them separately. A big batch has enough rows to go around, so each row is reduced in one pass. Your row is the same either way. The order it gets added in is not.

On a serving endpoint, batch size depends on how many other people are sending requests at that moment. So your output depends on traffic you can’t see or control. Batch size isn’t the only cause, either. Prefill processes your whole prompt in one big matmul while decode handles one token at a time, so the same tokens go through different kernels. And if the KV cache is split into chunks differently, attention gets summed in a different order.⁴

Across devices, there is no shared order#

On a single GPU, the problem is solved. Thinking Machines’ batch-invariant kernels fix every reduction’s order regardless of batch size or chunking, at about a 20% matmul slowdown, and all 1,000 of their completions came back identical.⁵

Across devices, there’s nothing to pin. An A100 and an H100 have different tensor cores, accumulation widths, and kernel catalogs, and a CPU doesn’t do fp16 math like a tensor core at all. PyTorch’s reproducibility notes promise nothing across platforms. Reproducible floating-point libraries try to close the gap,⁶ but only if every kernel on every platform is written to their standard.

What if we don’t use floats at all?#

Integer addition is associative. It is also commutative. $(a + b) + c = a + (b + c)$ holds for every $a$, $b$, $c$ as long as you do not overflow, and overflow is something you can bound ahead of time. That means every reduction, in any order, on any correct hardware, yields the same bits. Here are the same eight values as before, scaled by 4 so they are integers, going through the same two devices.

With integers, determinism comes from the arithmetic itself. You don’t need special kernels or a pinned order, and two vendors only have to agree that integer multiply-add is exact.

But int8 inference already exists (SmoothQuant, LLM.int8, OmniQuant), and it isn’t deterministic across hardware. That’s because it’s simulated quantization: matmuls run in int8, but everything between them (softmax, RMSNorm, SwiGLU, rotary embeddings) drops back to float. Every one of those float steps is a place for reduction order to leak back in.

Integer storage is not integer arithmetic. One float step anywhere breaks it, so the entire forward pass has to be integers, from input ids to logits.

DetLLM#

DetLLM runs Qwen3-0.6B with exact integer arithmetic for every operation between the input ids and the int32 logits: every matmul, normalization, activation, softmax, and rotary embedding. The integer operators come from I-LLM, part of a line of work (I-BERT, I-ViT, I-LLM) on integer-only transformers for edge hardware that cared about efficiency, not determinism.⁷ At 8 bits it needs no fine-tuning, just a calibration pass and two closed-form smoothing tricks.⁸ The pieces that matter:

Dyadic scales. Every tensor carries a scale $m / 2^k$, with an integer mantissa $m$ and shift $k$, so rescaling is an integer multiply and a shift with one fixed rounding rule. Exp, softmax, RMSNorm, and SwiGLU are shift-based integer algorithms on these values, and RoPE uses precomputed int16 sine and cosine tables.

Batch invariance by construction. Each token’s activations are quantized from that token’s own values, so its bits can’t depend on the rest of the batch, with no special kernels needed. The exception is the KV cache, where a per-token scale would land inside the softmax or the probs·V sum, so K and V use fixed scales calibrated offline. A cached value is quantized once and never again, so prefill and decode see the same bits.

One hardware-specific function. Only the int8 × int8 → int32 matmul dispatches on hardware (torch._int_mm on CUDA, a plain int32 matmul on CPU). Since integer accumulation is exact, matching across devices comes down to whether that matmul returns exact results, which the tests check. A TorchDispatchMode guard turns any float tensor in the forward pass into an error, and argmax breaks ties toward the lowest token id instead of trusting the backend.

One prompt, nine configurations, one hash#

The demo takes the first 64 tokens of Sonnet 18, greedy-generates 512 more, and chain-hashes the logits at every step. Each configuration targets a different source of nondeterminism: batch 1; batch 8, with seven random prompts sharing the batch; a prefill split in half, with the second half fed one token at a time; and a pure-CPU backend with no cuBLAS, Triton, or CUDA graphs. They run on three machines: an A100 server, an H100 server, and a MacBook.

Nine integer runs, one hash. Nine fp16 runs, nine hashes, every pair disagreeing from the first token. Four vendors and three instruction sets produce bit-identical logits, with nothing tuned per platform. The arithmetic has no freedom left to disagree.⁹

Divergent logits, matching tokens#

There’s a twist in the fp16 column. The demo also hashes the generated token ids, and eight of the nine fp16 runs produce the same 512 tokens despite disagreeing on the logits at every step. Greedy decoding is an argmax, and an argmax ignores any perturbation smaller than the gap between the top two candidates. On this prompt, batch composition shifts the step-0 logits by at most 0.094, while the top-two gap is 2.66, a 28× cushion.

That cushion is unusually generous. Most steps are much closer: 45 of the 512 have a top-two gap under 0.5. At step 457, " introduces" leads " is" by just 0.03 on the A100. On the M1 Pro the two land in an exact tie, the argmax picks the lower token id, " is", and the text forks:

A100: The first line introduces the question…

M1 Pro: The first line is “Shall I compare thee to a summer’s day?"…

Every generated token is another chance for a near tie, so the longer the generation, the more likely it forks. Float nondeterminism is usually latent, not absent.

Perplexity and speed#

The obvious worry is that removing every float hurts the model. It barely does: WikiText2 perplexity is 20.72 against 20.95 for fp16, and C4 is 0.4% worse.

Speed is a different story. The fp16 baseline runs on heavily optimized kernels like cuBLAS and FlashAttention. Integer inference has almost nothing: PyTorch ships an int8 matmul and that’s it, so softmax, RMSNorm, SwiGLU, and attention had to be hand-built out of small integer ops. After fusing the integer ops, adding CUDA graphs, and writing a fused Triton attention kernel, decode went from 0.8 tokens per second to 106.

Limitations and what’s next#

It’s a proof of concept, but I was surprised how well it held up: perplexity is on par with fp16, and with enough kernel work, decode isn’t stuck at eager speeds. Larger models, other architectures, integer sampling, prefill speed, and phone or NPU hardware are all open, and there are probably plenty of tricks left on the kernel side.

The guarantee is tied to one prepared artifact, since the float-to-int conversion isn’t itself reproducible. Code’s on GitHub if you want to try it. Run the int8 configuration anywhere and you should get 64430dd985f8. If you don’t, let me know.

Citation#

Please cite this work as:

Barry, Nathan. "Eliminating Nondeterminism in LLM Inference with Integer-Only Arithmetic".
nathan.rs (Oct 2026). https://nathan.rs/posts/detllm.

Or use the BibTex citation:

@article{barry2026detllm,
  title   = "Eliminating Nondeterminism in LLM Inference with Integer-Only Arithmetic",
  author  = "Barry, Nathan",
  journal = "nathan.rs",
  year    = "2026",
  month   = "Oct",
  url     = "https://nathan.rs/posts/detllm"
}