While validating attention kernels on a new Blackwell GPU, we got a result that seemed to say “flash-attn is wrong.” Following the discrepancy led us to the reference itself: PyTorch 2.7.1 fp32 attention. This article explains the conditions we measured, the versions where we could no longer reproduce the issue, and how to check your own environment.
What we found
At Nadoo Bio, we were investigating whether Evo 2, a DNA sequence model, could run on our own RTX PRO 4000 Blackwell server. Here is what the tests showed.
flash-attn 2.8.0.post2 passed our checks on this GPU. Its maximum error against a CPU float64 reference was 0.0092 in bf16, consistent with the rounding error expected in these tests.
PyTorch 2.7.1 fp32 attention silently returned incorrect results under specific conditions. With fp32 tensors, a head dimension divisible by 4 in the range 65–128 (such as 68, 72, 96, or 128), and sequence length of at least 25, the default
torch.nn.functional.scaled_dot_product_attention(SDPA) path differed from the reference by as much as 0.76. There were no error messages or NaNs. We observed the same issue in PyTorch 2.7.0.We could not reproduce it from PyTorch 2.8.0 onward in the versions tested. Official images for 2.8.0, 2.9.1, 2.11.0, 2.14.1, and 2.15.0 all had errors below 0.00001 in the same tests.
A trained model's output changed too. Loading the public OLMo-1B language model, which has a head dimension of 128, in fp32 under PyTorch 2.7.1 produced a maximum logit difference of 5.66. The top next-token prediction changed at 1 of 40 token positions.
All measurements in this article were taken on November 4, 2026, on a single RTX PRO 4000 Blackwell in Nadoo Bio's GPU server.
A few terms before we start
An omics foundation model is an AI model pretrained on large amounts of biological data, such as DNA sequences or gene expression measurements from cells. Evo 2 reads DNA base sequences—A, C, G, and T—and can be used to score how plausible a sequence containing a mutation is relative to the original. Attention, also used in language models, is one of the computations relevant to these models; the exact architecture and workload vary by model.
Attention is split across multiple heads. The length of the vector handled by one head is its head dimension, or head dim. For a standard attention configuration, divide hidden_size by num_attention_heads in the model configuration. A hidden size of 2048 with 16 heads gives a head dimension of 128.
PyTorch SDPA selects from several kernels, the implementations that actually execute on the GPU. These include the MATH path built from ordinary matrix operations, a memory-efficient path (EFFICIENT), and a FlashAttention path. The choice depends on the input dtype and shape [1]. flash-attn is the separately installed FlashAttention library.
sm_120 denotes compute capability 12.0, used by Blackwell workstation GPUs such as the RTX PRO 4000 Blackwell. When a new architecture arrives, libraries need builds that target it. For a while, “it runs” and “it computes the right answer” deserve separate checks.
Why we started checking
The Evo 2 repository recommends flash-attn 2.8.0.post2 for running the 7B model in bf16, and its installation example uses torch==2.7.1 with CUDA 12.8 [2]. Our GPU is sm_120, so our first question was whether that combination produced correct results on this hardware.
The build configuration for flash-attn 2.8.0.post2 includes sm_120 by default with CUDA 12.8 or later [3]. Its official wheel (flash_attn-2.8.0.post2+cu12torch2.7cxx11abiTRUE-cp311) also installed directly into pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime. Numerical correctness was the remaining question.
The first test made flash-attn look wrong
Figure 1. How we went from suspecting flash-attn to finding an incorrect reference. Measured on RTX PRO 4000 Blackwell with PyTorch 2.7.1+cu128 and flash-attn 2.8.0.post2.
We generated random Q, K, and V tensors—the three attention inputs—with batch size 2, sequence length 1024, and 16 heads. We varied the head dimension across 64, 128, and 256, comparing flash-attn in bf16 and fp16 with PyTorch fp32 SDPA on the same GPU. Since fp32 has more precision than bf16, we treated its output as the reference.
The results looked odd. Maximum differences stayed below 0.01 at head dimensions 64 and 256, but rose to 0.43 in bf16 and 0.48 in fp16 at head dimension 128. Both were causal tests: each position could attend only to itself and earlier positions. These differences were far larger than the rounding errors we expected. We were close to concluding that flash-attn computed head dimension 128 incorrectly on sm_120.
Changing the reference changed the suspect
Our supposed ground truth was still a GPU computation. We moved the same inputs to the CPU, calculated attention in float64, and used that result as the new reference.
Implementation (CPU float64 reference) | Head 64 | Head 96 | Head 128 | Head 256 |
|---|---|---|---|---|
flash-attn, bf16, causal | 0.0086 | 0.0081 | 0.0092 | 0.0086 |
Original reference (PyTorch fp32 SDPA), causal | 0.0000 | 0.4223 | 0.4613 | 0.0000 |
flash-attn stayed within the expected bf16 error range at every head dimension tested. The variable-length function, flash_attn_varlen_func, had an error of 0.0086; rotary position encoding, apply_rotary_emb, had an error of 0.0133. A causal forward pass with sequence length 8,192 and 32 heads took about 5 ms. The reference from our first test was the incorrect result.
Which conditions trigger the error?
Figure 2. Affected fp32 SDPA conditions and version comparisons. Bars show maximum error against CPU float64 for PyTorch 2.7.1 with causal attention and sequence length 1024. “Pass” means error below 0.00001 in the version tests.
We varied the inputs to fp32 SDPA under PyTorch 2.7.1, still using CPU float64 as the reference.
Of the first eight head dimensions, 32, 64, 160, and 256 had errors reported as zero, while 80, 96, 112, and 128 showed errors of 0.31–0.66 with causal attention and 0.09–0.21 without it. These are the bars in Figure 2.
A finer sweep revealed a more specific boundary. Within 65–128, only dimensions divisible by 4 were affected: 68, 72, 76, 84, 88, 100, 104, 116, 120, 124, and so on. Tested dimensions not divisible by 4, such as 66, 70, 74, 90, 110, and 126, were normal, as were the tested dimensions at or below 64 and at or above 129. Common model head dimensions—64, 80, 96, and 128—are divisible by 4, making this interval relevant in practice.
Sequence length had a boundary too. At head dimension 128, lengths 16 and 24 passed, but length 25 produced an error of 0.44. Testing only short inputs could easily miss this.
Forcing individual kernel paths narrowed it down further. On the same head-128 input, the MATH path had an error of 0.000002, while the memory-efficient path had an error of 0.447. The default path matched the memory-efficient error, indicating that it selected that path for these fp32 inputs. The cuDNN and FlashAttention paths did not support fp32 in this configuration and were not candidates.
We also checked TF32. Since Ampere, GPUs can accelerate some fp32 matrix multiplications using lower-precision TF32, so it was an early suspect. Toggling torch.backends.cudnn.allow_tf32 did not change the error. The matrix-multiplication TF32 setting was off by default. TF32 did not explain the observed discrepancy.
Changing dtype changed the outcome. With the same head-128 inputs in bf16, the default path had a normal error of 0.0094. In these tests, the problem was specific to fp32 inputs.
Which versions passed?
We ran the same test across six versions of the official PyTorch Docker images.
PyTorch | CUDA build | Image digest (first 12 characters) | fp32 SDPA, head 128, causal |
|---|---|---|---|
2.7.1 | 12.8 |
| 0.476 (incorrect) |
2.8.0 | 12.8 |
| 0.00000 |
2.9.1 | 12.8 |
| 0.00000 |
2.11.0 | 12.8 |
| 0.00000 |
2.14.1 | 13.0 |
| 0.00000 |
2.15.0 | 13.2 |
| 0.00000 |
For the tested versions from 2.8.0 onward, errors remained below 0.00001 across the tested head dimensions (32–256) and sequence lengths (16–1024). The PyTorch 2.8.0 release notes include several memory-efficient attention fixes [4], but we did not find an entry that directly identified our symptom. Our conclusion is therefore limited to “we could not reproduce it from 2.8.0 onward in the tested versions.”
A public report describes a similar silent error in memory-efficient attention in a custom PyTorch build on DGX Spark (sm_121) [5]. That report attributes the issue to build configuration and also describes incorrect bf16 results. Those differences matter; we do not treat it as the same root cause.
Does it change a real model's output?
A kernel discrepancy does not, by itself, tell us how much survives through a model.
We first ran a randomly initialized nn.MultiheadAttention module in fp32, with d_model 1024, 8 heads, and head dimension 128. Under PyTorch 2.7.1, the maximum attention-output error was 0.0057, about 27% of the output standard deviation of 0.021. Yet a randomly initialized model with 12 encoder layers of the same configuration produced final outputs with cosine similarity 0.999998. Residual connections and layer normalization appear to have diluted the attention error in this random model. Testing only that model could have led us to dismiss the issue.
We then loaded a trained public model: OLMo-1B (allenai/OLMo-1B-hf, head dimension 128), in fp32 with Hugging Face Transformers 4.53.3 [6]. We supplied one English sentence of 40 tokens and compared logits on the same GPU, changing only the attention implementation between sdpa and eager (ordinary matrix operations). Logits are the scores assigned to candidate next tokens before conversion to probabilities.
Measurement | PyTorch 2.7.1 | PyTorch 2.8.0 |
|---|---|---|
Maximum logit difference (logit standard deviation: 3.7) | 5.66 | 0.0000 |
Matching top next-token predictions | 39 / 40 | 40 / 40 |
Maximum tokenwise probability-distribution difference (KL divergence) | 0.215 | 0.0000 |
Mean negative log-likelihood; lower is better (eager / sdpa) | 2.825 / 2.847 | 2.825 / 2.825 |
Transformers uses SDPA by default for supported models and environments unless another implementation is specified [7]. The default dtype varies by version; current documentation describes following the dtype in the model's config.json [8]. A model stored or explicitly loaded in fp32, with a head dimension divisible by 4 in the affected interval, can therefore reach this path under PyTorch 2.7.1 on the tested sm_120 GPU and return different results without an error.
What does this mean for omics models?
Evo 2 7B is documented to run in bf16 [2], which differs from the fp32 path affected here. Our bf16 kernel tests passed. We have not yet traced Evo 2's full inference path, however. The problem can also arise outside the model: fp32 reference code used for validation may itself be wrong, as in our first test. Other models installed in the same environment may take the affected path when run in fp32. The Evo 2 installation example uses this very PyTorch version, 2.7.1.
Meaningful comparisons between omics predictions depend on consistent computation under the same conditions. Changes in variant scores can alter candidate rankings; changes in cell embeddings can affect clustering. Those are possible consequences, not outcomes measured here. This error is troublesome because it produces plausible numbers without NaNs or warnings, leaving little to distinguish a model problem from a kernel problem.
We audited all 35 execution images deployed by Nadoo Bio. Ten used PyTorch 2.7.0 or 2.7.1 on GPUs and therefore had a potentially affected environment. Tracing the actual tool paths found none that met the affected conditions. ProteinMPNN, RFdiffusion, RFantibody, and DiffAb did not use SDPA. ESMFold ran its ESM-2 component in fp16 with head dimension 64. Chai-1 called SDPA thousands of times, but only at head dimensions 24, 32, 48, and 64. The difference between predicted structures with the default configuration and with memory-efficient attention disabled—coordinate RMSD 0.09–0.14 Å—was comparable to variation between repeated runs with the same configuration (0.10–0.15 Å). An affected environment does not imply affected output if the tool never enters the problematic path. Adding a new model calls for checking that path again.
Check your own environment
Figure 3. A four-step check. The numbers in step 3 are the output from running the script below on our server.
Start with the PyTorch version (torch.__version__), GPU architecture (torch.cuda.get_device_capability()), and model dtype. Next, calculate the head dimension from the model configuration. These steps indicate whether the conditions may apply. We tested only the RTX PRO 4000 Blackwell, so on other GPUs it is better to run the script than to infer correctness from the hardware name. The check uses a CPU float64 computation on the same inputs as its reference; runtime depends on your CPU as well as your GPU.
Save the following as sdpa_fp32_check.py:
import torch
import torch.nn.functional as F
from torch.nn.attention import sdpa_kernel, SDPBackend
torch.manual_seed(0)
print(torch.__version__, torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))
for hd in (64, 96, 128, 160):
q, k, v = (torch.randn(2, 16, 1024, hd, device="cuda") for _ in range(3))
# Reference: the same inputs computed on the CPU in float64
ref = F.scaled_dot_product_attention(q.cpu().double(), k.cpu().double(), v.cpu().double(), is_causal=True)
out = F.scaled_dot_product_attention(q, k, v, is_causal=True) # Default path
with sdpa_kernel([SDPBackend.MATH]):
safe = F.scaled_dot_product_attention(q, k, v, is_causal=True) # MATH path
err = lambda t: (t.cpu().double() - ref).abs().max().item()
print(f"head {hd:3d} default {err(out):.5f} math {err(safe):.5f}")To run it in the official image:
docker run --rm --gpus device=0 -v "$PWD":/w:ro \
pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime python /w/sdpa_fp32_check.pyOur server, with PyTorch 2.7.1 and RTX PRO 4000 Blackwell, printed:
2.7.1+cu128 NVIDIA RTX PRO 4000 Blackwell (12, 0)
head 64 default 0.00000 math 0.00000
head 96 default 0.41907 math 0.00000
head 128 default 0.35089 math 0.00000
head 160 default 0.00000 math 0.00000A default error above 0.001 flags the discrepancy observed in this test; it is not a diagnosis of the underlying kernel defect by itself. Error magnitude depends on the random inputs. For head dimension 128 with causal attention, our measurements ranged from 0.35 to 0.48. The tested versions from PyTorch 2.8.0 onward stayed below 0.00001 under the same conditions.
If your environment is affected
The simplest option is to upgrade PyTorch to 2.8.0 or later. Every version we tested from 2.8.0 through 2.15.0 passed. Extensions such as flash-attn have wheels tied to particular PyTorch versions, so update those together.
If an upgrade is not practical immediately, force fp32 attention to use MATH. It uses more memory and is slower, but its results were correct in our tests.
from torch.nn.attention import sdpa_kernel, SDPBackend
with sdpa_kernel([SDPBackend.MATH]):
outputs = model(inputs)For a Transformers model, loading with attn_implementation="eager" is another option where supported [7]. If the model is suitable for bf16, that path also passed our tests.
When validating numerical results on a new GPU, an independently computed reference helps. We initially trusted fp32 on the same GPU and blamed a working flash-attn implementation. Even a smaller CPU float64 test can separate a faulty implementation from a faulty reference.
What we have not established
We tested only one GPU model, the RTX PRO 4000 Blackwell (sm_120). We do not know whether other sm_120 cards or Ampere and Hopper GPUs show the same symptom under PyTorch 2.7.1.
We tested six official PyTorch image versions and a production image with PyTorch 2.7.0 installed through pip. We did not test versions earlier than 2.7.0.
We have not identified the kernel-level root cause, why divisibility by 4 matters, or which change in 2.8.0 removed the symptom in our tests.
We have not run full Evo 2 7B inference. Our flash-attn conclusion concerns kernel-level numerical checks.
Reproduction details
Item | Value |
|---|---|
GPU | One NVIDIA RTX PRO 4000 Blackwell 24GB; compute capability 12.0; driver 595.91.07 |
PyTorch images | Official |
flash-attn | Official wheel |
Trained-model test |
|
Reference | CPU float64 on the same inputs; for OLMo, the eager implementation on the same GPU |
Production audit | 35 deployment execution images; Chai-1 compared over three runs using 76-residue ubiquitin and the same seed |
References
PyTorch. torch.nn.functional.scaled_dot_product_attention. Official documentation. Accessed 2026-11-04.
Arc Institute. evo2 README. Repository documentation, installation and FP8 requirements sections. Accessed 2026-11-03.
Dao-AILab. flash-attention v2.8.0.post2 setup.py. Source code,
cuda_archsand sm_120 gencode sections. Accessed 2026-11-04.PyTorch. PyTorch 2.8.0 Release Notes. Release notes. Accessed 2026-11-04.
NVIDIA Developer Forums. DGX Spark / sm121: silent SDPA EFFICIENT_ATTENTION corruption in a custom PyTorch build. User forum report. Accessed 2026-11-04.
Allen Institute for AI. allenai/OLMo-1B-hf. Hugging Face model card. Accessed 2026-11-04.
Hugging Face. Attention interface. Transformers documentation source, main branch. Accessed 2026-11-04.
Hugging Face. Loading models — Model data type. Transformers documentation source, main branch. Accessed 2026-11-04.