Reevaluating 16 TYK2 candidates with Boltz-2
Imagine planning experiments for 16 molecules that might become drug candidates. With limited time and resources, which should come first? One hoped-for role of AI in drug discovery is to help set that order, prioritizing promising candidates for the next experiments.
We investigated this question with Boltz-2, an AI model that predicts how proteins and small molecules may bind and what their binding properties may be. We selected 16 molecules with published historical experimental results, calculated predictions, and compared them with those measurements—checking the model on a problem whose experimental answers were already available.
Two of the three strongest candidates by experimental measurements also appeared in the model's top three. The model followed much of the overall trend, but ranked the experimentally fourth-place candidate eleventh. Computation finished in about 23 minutes on one GPU. We then examined where the rankings differed and by how much, using numbers and figures together.
Checking predictions against existing experiments is called retrospective evaluation. We did not develop a new medicine or test effects in people. We examined model judgments on historical data. Whether the same findings hold for untested candidates requires further research.
Start with the protein a candidate is intended to affect
Proteins perform many tasks in the body: transporting substances, transmitting signals, and facilitating reactions. Drugs can affect proteins and change their activity. The biological object whose activity researchers intend to modulate is the target.
Our target was TYK2, tyrosine kinase 2. It participates in cellular signaling and belongs to an enzyme family that adds chemical markers called phosphate groups to other proteins. Enzymes are proteins that facilitate particular chemical reactions. Here we compare inhibitor candidates intended to reduce TYK2 activity. [1]
A molecule binding a protein is called a ligand. Here, the ligands are the small molecules under comparison. We call them candidates while discussing the research; they are not approved drugs.
Protein binding is often compared with a key fitting a lock. The analogy captures the importance of compatible shapes. Actual molecules, however, move and are influenced by surrounding water and electrical properties. A visually good fit alone does not establish strong binding or a good medicine.
Boltz-2 predicts both arrangement and binding properties: “How might these molecules be positioned?” is a structure question; “What might their binding properties be?” is an affinity question. Binding affinity describes how readily two molecules bind. Pose and affinity are related, but their predictions are separate outputs. [1]
Figure 1. We kept the protein fixed and changed the candidate molecule, then compared predictions with historical experiments. “2/3” means two overlapping candidates among the top three, not a probability of becoming a drug.
Why recalculate candidates already tested experimentally?
Evaluating the model requires experimental reference data. We used the TYK2 dataset from the Open Forcefield Group's protein-ligand-benchmark. A benchmark provides a common reference problem for comparing methods. This dataset connects a protein structure, 16 molecules, and historical experimental values. [2]
The Boltz-2 paper also evaluated TYK2. We took related public candidates and independently checked how well the model distinguished their relative strengths. The paper includes much larger searches and generation of new molecules; this article focuses on 16 existing candidates. [1]
Input preparation details and the execution environment were not matched exactly to the authors'. Using the same model does not constitute full replication of their conditions. This is an independent reevaluation of one target associated with the paper.
Throughout, “predicted values” mean outputs calculated by the AI, and “experimental values” mean historical measurements in the public dataset. We used existing inhibition measurements; the new measurements we collected were computation time and GPU usage.
Turn the protein and candidate molecules into inputs
Names alone are insufficient model inputs. The model needs to know which components connect in which order and arrangement.
Proteins are chains of amino acids. Writing their order as characters gives the sequence. We extracted the region needed for comparison from a public structure. The input's 288 amino acids describe that region, not all of TYK2.
Each small molecule was supplied as a SMILES string, encoding atom types, connectivity, rings, and stereochemistry. This makes a molecular drawing's connectivity machine-readable. Protein sequences primarily encode the order of building blocks; SMILES encodes atom connections.
Stereochemistry matters. Even with the same connected atom types, different spatial orientations can change protein interactions. We preserved the arrangement recorded in the source. Removing it merely to simplify file handling could mean evaluating a different molecule.
We also supplied an MSA, multiple sequence alignment, which aligns corresponding positions in related protein sequences. Patterns of changes over evolution help infer structure. Every candidate used the same protein sequence and MSA; only the candidate molecule changed.
Each input paired the protein with one candidate. We made 16 such inputs, requesting structure and affinity together. Code and settings appear in the execution notes below. [3]
Figure 2. Connectivity of the 16 candidates. EJM and JMC identify source series. Lines are bonds; N, O, and Cl denote nitrogen, oxygen, and chlorine. Where carbon symbols are omitted, line bends and endpoints represent carbon atoms. Triangles and hexagons indicate rings. These drawings do not depict actual three-dimensional poses.
Even without reading every chemical detail, the similarities are visible. Some connections or atoms differ. We examined how those small differences related to experimental strength and whether the model distinguished them.
Why smaller experimental values indicate stronger inhibition
Below each molecule in Figure 2 is Ki, the inhibition constant. It expresses inhibitor strength within a defined model. Under comparable experimental conditions, lower Ki indicates stronger inhibition—the opposite direction from a score where higher is better.
Ki has concentration units. Concentration describes the amount of substance in a volume. Micromolar (µM) and nanomolar (nM) express small concentrations: 1 µM = 1,000 nM.
The source mixed units. The 12 EJM candidates used µM, while the four JMC candidates used nM. Their experimental values originate in separate 2013 papers by Liang and colleagues; we used the values organized by OpenFF. [2] [4] [5]
Two actual candidates illustrate the conversion.
Candidate | Source Ki | Ki in µM |
|---|---|---|
EJM 31 | 0.096 µM | 0.096 µM |
JMC 23 | 2.5 nM | 0.0025 µM |
Numerically, 0.096 is smaller than 2.5. After conversion, however, JMC 23 has the lower Ki. Sorting mixed centimeters and millimeters by number alone would create the same problem. Matching units does not make the experimental conditions in the two papers identical, so we also analyzed the sources separately.
Annotations mattered too. EJM 44's numeric field contained 3.5, but its note said Ki was greater than 3.5 µM.
Measuring exactly 3.5 differs from knowing only that the value exceeds 3.5. The latter specifies a boundary without an exact value—a censored measurement. The candidate is not missing all information; its value is bounded rather than precisely known.
We ran all 16 candidates but included only 15 in the primary analysis requiring exact numbers. EJM 44 retained its boundary information and was displayed separately. Its effect on the evaluation is discussed below.
Comparing AI scores with experiments
The Boltz-2 affinity output used here was affinity_pred_value: lower values indicate candidates predicted to bind more strongly. Knowing its name and direction helps interpret the figures. [3]
To compare it with experiments, first align scales. Official documentation describes a logarithmic representation of IC50, the concentration producing half inhibition in an assay. A log scale represents tenfold changes at equal intervals. On the scale used here, 1 µM maps to 0, 0.1 µM to −1, and 0.01 µM to −2. These are conversion examples. Negative log values do not mean negative concentrations or failed calculations.
For EJM 31, the predicted value was approximately −0.928. Converting its experimental Ki of 0.096 µM gives approximately −1.018. The numbers are close, but that alone does not establish a small prediction error: the model's IC50-related output and the source's Ki are different quantities.
Ki is a constant within an inhibition model; IC50 is the concentration producing half inhibition in a particular assay. IC50 can change with assay conditions for the same molecule. Sharing concentration units does not make these identical measurements. We examined whether experimentally stronger candidates were also rated stronger by the model, rather than treating the numbers as exact equivalents.
Another output, affinity_probability_binary, classifies binding candidates. EJM 31 scored about 0.762. This is a classification output for a particular purpose, not a 76.2% probability of becoming a medicine. Becoming a drug requires answers about effects in cells and the body, safety, and much more.
Did AI also favor experimentally stronger candidates?
We used the Pearson correlation coefficient to compare overall trends: how consistently the two logarithmic values varied together. The question was whether the model tended to favor candidates that were stronger experimentally.
The coefficient ranges from −1 to 1. Near 1 indicates a strong positive linear relationship, near 0 a weak linear relationship, and near −1 an inverse relationship. Both axes here treat lower as stronger, so a high positive correlation is the expected direction. A correlation of 0.815 describes a relationship; it is not 81.5% accuracy.
The coefficient for the 15 exact experimental measurements was 0.815. Reading the points alongside that summary helps make sense of the result.
Figure 3. Each point is a candidate. The horizontal axis is experimental and the vertical axis predicted. Left means stronger experimentally; down means stronger by the model. Blue circles and orange squares distinguish sources. The open circle and arrow show EJM 44's bound instead of an exact measurement.
Stronger experimental candidates mainly appear toward the lower left and weaker ones toward the upper right. The broad trends agree, but the points do not form one line. Candidates with similar measurements received different predictions, and some departed from the trend.
JMC 28 stands out toward the left but relatively high: experimentally strong, yet rated less strongly by the model. Because the axes describe different quantities, distance from a diagonal is not an absolute concentration error.
We also compared a simple rule that heavier molecules would be stronger. Molecular weight sums the masses of a molecule's atoms. That rule produced a correlation of 0.118. On this dataset, Boltz-2 followed experimental trends better than that simple baseline. This does not establish understanding of every binding mechanism.
Treating EJM 44 as exactly 3.5 µM and including all 16 raises the correlation to 0.860. The number looks better, but an unknown exact value has been treated as known. We therefore retained 0.815 from 15 candidates as the primary result. Interpreting experimental records affects performance evaluation alongside the calculation itself.
How much did the top three overlap?
Returning to the original question, suppose three candidates were to be tested first. Similar overall trends do not guarantee identical priorities. We performed no new assay; we compared the two top-three lists.
Rank | Experimental Ki | Model prediction |
|---|---|---|
1 | JMC 23 | EJM 46 |
2 | EJM 46 | JMC 23 |
3 | JMC 27 | EJM 54 |
JMC 23 and EJM 46 appear in both. The third differs. Two of three overlap describes candidate selection, not two successful drugs out of three.
Connecting the full rankings shows where judgments diverge.
Figure 4. Experimental ranking is on the left, predicted ranking on the right. Lines connect the same candidate. Higher means stronger; light blue marks the top three. Nearly horizontal lines indicate similar ranks, steep lines larger discrepancies. Orange JMC 28 moves from experimental fourth to predicted eleventh.
Correlation summarizes the relationship across all candidates. This figure reveals differences that matter when prioritizing individual ones. Either alone can miss an overall pattern or an important exception.
Experimental rankings also come from sorting measured numbers. When values are close, measurement error can make their ordering uncertain. This dataset did not let us quantify that uncertainty, so small rank differences were not treated as definitive biological superiority.
A closer look at JMC 28
JMC 28 had the largest rank discrepancy: fourth experimentally and eleventh in prediction. Yet its structural confidence measure, ipTM, was 0.985.
ipTM describes model confidence in the arrangement between a protein and its partner. “This relative placement looks plausible” differs from “this candidate acts more strongly than another.” A value of 0.985 is not a 98.5% experimental success probability.
Figure 5. JMC 28's predicted structure. The long blue curve is the protein backbone; orange points locate the candidate. The right view enlarges the candidate from another angle. Actual predicted coordinates were projected onto a plane, with a smoothed backbone for readability. Bond lines indicate connectivity. This is neither a microscopy image nor experimentally confirmed binding.
High structural confidence coexisted with a large ranking discrepancy. The picture alone cannot tell us whether the structure was wrong, the affinity prediction missed a difference, or assay conditions contributed. Follow-up comparison with experimental structures or other measurements is needed.
For developers, this is also a data-display problem. Labeling structural confidence, predicted affinity, and experimental inhibition constants simply as “scores” invites readers to treat them as equivalent. Data and screens should distinguish what each value predicts or measures.
What was—and was not—computed in 23 minutes
We used one GPU for the model calculations, keeping protein and auxiliary data fixed while processing the 16 candidates sequentially. All completed. Five possible structures and affinity results per candidate yielded 80 structure files and 16 affinity results. The five structures represent model-generated possibilities, not five repeated experiments.
Total runtime was 1,394.4 seconds, about 23.2 minutes. Per-candidate runtime ranged from 85.9 to 89.1 seconds, with a median of 87.0 seconds. Timing ran from job startup to completion with model files and inputs already prepared.
Figure 6. Each bar is a candidate: runtime on the left, observed GPU memory on the right. Runtimes were similar. Sampling memory every five seconds may miss intervening peaks.
GPU memory holds models and data during computation. Maximum observed usage was about 14.0 GiB. This can inform hardware preparation, but requirements change with input length and settings.
The 23.2 minutes exclude model downloads, preparation of protein auxiliary data, and analysis. They also exclude making molecules or measuring inhibition. This is model computation time, not developing a new medicine in 23 minutes.
We used the model image employed by Nadoo Bio's catalog—a package containing the model and environment. We did not test the complete service path from catalog submission through billing and result tables. Versions, hardware, and settings are recorded below.
Fifteen candidates can produce an unstable summary
The 0.815 coefficient belongs to these 15 candidates. To examine sensitivity to their composition, we removed one candidate at a time and recalculated correlation. It ranged from 0.755 to 0.854. This changed the aggregation, not model execution or training.
We also used bootstrap resampling: repeatedly drawing 15 pairs of experimental and predicted values, with replacement, from the existing 15 pairs. This examines how the summary changes with sample composition. The middle 95% of 10,000 repetitions lay between 0.559 and 0.932.
Resampling related candidates for one target cannot predict performance on other proteins. It does show how much a summary can vary with few candidates. Reporting three decimal places does not make performance definitive.
Separating sources gave 0.844 for the 11 exact EJM values and 0.712 for four JMC values. A four-candidate correlation is particularly sensitive to individual candidates. We did not conclude that one series is generally easier to predict.
What this case shows about AI's role
The model followed much of the experimental trend, and two top-three candidates overlapped. Important exceptions such as JMC 28 remained. Its ranking can inform experimental planning without being a final determination of candidate properties.
Performance on other proteins requires separate evaluation. The paper describes separating affinity evaluation data from training data, but also states that the structure-prediction module used public TYK2–ligand structures for training. Data for learning activity strength and data for learning molecular arrangements are distinct. This case cannot be generalized as performance on a protein the model had never encountered. [1]
Developers have concrete responsibilities: preserving molecular information in model inputs, applying the same conditions to candidates, and linking outputs back to experiments. That includes retaining units and inequality signs. Here, replacing “greater than 3.5” with “3.5” alone made apparent model performance improve.
Alongside correlation, we recorded successful priorities and ranking discrepancies, together with inputs and calculation conditions. These records let researchers examine the recommendation and decide what to test next.
Execution notes for developers
The following details support recalculation. They are not all needed to understand the observations, but inputs, versions, and settings must remain linked to reproduce the calculation.
We pinned OpenFF protein-ligand-benchmark release 0.2.1 to commit fe6f96916b2e28f9c14398d77c838d6515e931b0. The reference structure identifier was 4GIH. Removing computational terminal caps ACE/NME and normalizing histidine-state labels to H yielded a 288-amino-acid sequence. Coordinates were not supplied as a ground-truth template. We fixed an A3M MSA obtained through ColabFold using the public sequence.
The protein was chain A, ligand B, and binder: B requested affinity. Variables below hold the actual sequence, SMILES, and MSA path. [3]
payload = {
"version": 1,
"sequences": [
{"protein": {
"id": "A",
"sequence": tyk2_sequence,
"msa": str(msa_path),
}},
{"ligand": {"id": "B", "smiles": ligand_smiles}},
],
"properties": [{"affinity": {"binder": "B"}}],
}We used Boltz 2.2.1, PyTorch 2.11.0+cu128, and one RTX PRO 4000 Blackwell. Each candidate started in a new container. Settings were identical and were not tuned after inspecting results. Only one seed was used, so variation across repeated model runs was not evaluated.
Setting | Value |
|---|---|
Structure recycling | 5 |
Structure generation | 200 diffusion steps; 5 structures per candidate |
Affinity path | 200 sampling steps; 5 diffusion samples |
Parallel samples | 1 |
Random seed | 20261026 |
Molecular-weight correction | Enabled |
Execution option |
|
Network | MSA prepared beforehand; model container network blocked |
Structure settings referenced paper section B.5.1 and molecular-weight correction B.5.2. The paper combines two affinity models and applies molecular-weight correction; we recorded this correction before running. The authors' curated candidate list, MSA, and environment were not replicated exactly. Their TYK2 Pearson R=0.83 and four-target mean R=0.66 are different aggregates, separate from our 0.815. ABFE in the paper's evaluation of generated candidates estimates absolute binding free energy; we did not interpret it as new experimental measurements in that version. [1]
The primary analysis compares log10(Ki / µM) with affinity_pred_value for 15 exact Ki values. These metrics summarize the same results from different perspectives.
Evaluation | Result | Interpretation |
|---|---|---|
Pearson R | 0.815 | Linear relationship between log experimental and predicted values |
Spearman rank correlation | 0.796 | Similarity in rank rather than numeric spacing |
Kendall tau-b | 0.657 | Agreement on ordering when comparing candidate pairs |
Top-three overlap | 2/3 | Shared candidates among the first three selected |
Molecular-weight-only Pearson R | 0.118 | Untrained baseline scoring with negative molecular weight |
Structure and affinity results were stored separately. JMC 28's ipTM=0.985 came from the saved rank-zero structure, not necessarily the first structure generated. Input/protocol hashes matched pre-run records; 16 completion records and 80 structure files were confirmed. This checks file linkage and integrity, not experimental structural accuracy.
Resource observations covered container runtime and whole-GPU memory sampled every five seconds, with a maximum of 14,341 MiB. Other GPUs on the same host were also active, so these observations were not generalized to dedicated-host performance or service cost.
All 16 results, analysis summaries, and the pre-run protocol were stored separately. Raw outputs include affinity JSON, 80 structures, execution logs, and resource observations.
Data and usage terms. The Open Forcefield Group benchmark uses CC BY 4.0. Research and commercial reuse are permitted with attribution, license identification, and indication of changes. We converted units, separated inequality information, and created molecular diagrams and analysis figures. This does not apply the same license to the original papers' text or figures. [7] The reviewed official Boltz README states that code and weights are MIT-licensed. Preserve copyright and permission notices when using them for research or commercial purposes; this does not replace separate data or server terms. [6]
References
Passaro et al. Boltz-2: Towards Accurate and Efficient Binding Affinity Prediction. bioRxiv v1, 2025. Structure/affinity prediction and evaluation paper.
Open Forcefield Group. protein-ligand-benchmark, pinned commit fe6f969. Public TYK2 inputs and experimental values in release 0.2.1. Checked October 26, 2026.
Boltz developers. Prediction documentation. Official input/output documentation snapshot. Checked October 26, 2026.
Liang et al. Lead identification of novel and selective TYK2 inhibitors. European Journal of Medicinal Chemistry, 2013. Original EJM measurements.
Liang et al. Lead Optimization of a 4-Aminopyridine Benzamide Scaffold To Identify Potent, Selective, and Orally Bioavailable TYK2 Inhibitors. Journal of Medicinal Chemistry, 2013. Original JMC measurements.
Boltz developers. Boltz README. Official code and weight terms. Checked October 26, 2026.
Creative Commons. Attribution 4.0 International. CC BY 4.0 terms applied by the dataset repository; checked with its license text on October 26, 2026.