calculation report
ChEMBL-Derived Benchmark Dataset and Computational Results of Boltz-2-Based Binding Affinity Prediction
2026 年 26 巻 p. 11-27
詳細
2026 年 26 巻 p. 11-27
Accurate prediction of protein–ligand binding affinities is a critical step in accelerating drug discovery by reducing experimental costs and development times. Recently developed co-folding AI models predict how multiple biomolecules fold and interact with each other in three-dimensional space. The emergence of a new co-folding model, Boltz-2, has made highly accurate and efficient predictions of protein–ligand binding affinities increasingly feasible. However, the generalization and reliability of these models remain unclear due to the absence of standardized and target-wide benchmark datasets. In this study, we constructed an independent external benchmark dataset derived from ChEMBL version 35 to rigorously evaluate Boltz-2’s performance for affinity prediction. The dataset includes 356 unique protein targets and 10,933 compounds, carefully collected to ensure no overlap with the Boltz-2 training data. Binding affinity measurements were standardized into pChEMBL values and linked to the compound SMILES and protein UniProt accessions. Using this benchmark dataset, we compared the performance of the original Boltz-2 model and its NVIDIA Inference Microservice (NIM) implementation. The results showed that the original Boltz-2 and NIM achieved comparable and fair predictive performance across targets (mean absolute error of approximately 0.9), while NIM reduced computational time by approximately 60–90%. The error analysis indicated that no clear correlation existed between the prediction errors and sequence or compound novelty relative to the Boltz-2 training data, underscoring the model’s broad coverage. This work provides a transparent and reproducible benchmark for evaluating AI-driven affinity prediction models and offers valuable insights into Boltz-2’s applicability, limitations, and potential as a practical tool for data-driven drug discovery.
The accurate prediction of binding affinities for novel compounds is crucial for eliminating bottlenecks in the early stages of drug discovery, from hit identification to lead optimization, and for significantly reducing experimental costs and timelines. Generative model-integrated structure prediction approaches have recently attracted attention due to their potential to accurately predict protein–ligand complex structures, referred to as co-folding methods [1, 2]. The recently released Boltz-2 has been reported to not only predict complex structures but also efficiently estimate the binding affinities (pIC50-like values) of their interactions with accuracy comparable to that of free energy perturbation [3].
However, without rigorous external evaluation of the generalizability, robustness, and reproducibility of these predictive models, their advantages remain limited, hindering translation into practical applications or comparison with other methods. Therefore, developing a publicly available benchmark dataset that ensures target coverage [4] and validating it using appropriate evaluation metrics, such as mean absolute error (MAE), root mean squared error (RMSE), Pearson’s correlation coefficient (Pearson r), Spearman’s rank correlation coefficient (Spearman ρ), and Kendall’s rank correlation coefficient (Kendall τ) are essential. These evaluations should also consider the ability to extrapolate based on protein sequence similarity of targets, the novelty of compound structures, and the prediction reliability of complex structures (confidence score).
In this study, we present a comprehensive evaluation of the predictive performance of Boltz-2 using a diverse dataset derived from bioactivity records in the ChEMBL database [5]. By establishing and sharing a benchmark dataset, we aimed to clarify the model’s extrapolation ability using compound–target affinity data not used in Boltz-2 training, as well as to delineate its limitations. This work enhances transparency regarding the effectiveness of AI-based co-folding models and provides the research community with a reproducible foundation for comparative assessment. Ultimately, it contributes to accelerating the practical implementation of computational predictions in drug discovery and improving the reliability of data-driven decision-making.
2.1 Calculation conditions for affinity prediction
In this study, Boltz-2 affinity predictions were performed using the original Boltz-2 v2.2.0 [6] and NVIDIA Inference Microservice (NIM) for Boltz-2 Release 1.2.0 [7]. All computations were executed on a Linux operating system (Ubuntu 22.04.3 LTS) using an AMD EPYC 7352 CPU (512 GB RAM) and an NVIDIA A100 GPU (80 GB memory). NIM for Boltz-2 is an alternative implementation of the Boltz-2 model provided by NVIDIA. It is optimized for NVIDIA GPUs and enables fast and efficient execution of the Boltz-2 AI model, offering high-throughput affinity prediction as well as structure prediction of proteins, ligands, antibodies, and related biomolecules.
Multiple sequence alignments (MSAs) of target protein sequences were generated using the original Boltz-2 with the use_msa_server option. The affinity prediction for each compound was performed using its target protein sequence as a monomer with the pre-computed MSA. The original Boltz-2 was executed in the default setting, while the Boltz-2 NIM was tested under two different configurations: (i) the original setting (NIM-original), equivalent to the Boltz-2 default settings, and (ii) the light setting (NIM-light), in which four parameters, sampling_steps, sampling_steps_affinity, diffusion_samples_affinity, without_potentials, were set to 50, 50, 1, and false, respectively, while the other parameters remained identical to the original setting.
2.2 Dataset
Data sources
To evaluate the Boltz-2 performance in affinity prediction, we designed an external benchmark dataset that consists of data not used in the Boltz-2 training. The training data for Boltz-2 were primarily derived from public databases: Protein Data Bank (PDB) [8] for protein structure information; ChEMBL and BindingDB [9] for binding affinity regression values; and PubChem [10], CeMM [11], and MIDAS [12] for binding affinity classification. The benchmark dataset was also constructed from these sources, enabling comprehensive evaluation of the generalization performance of Boltz-2 for new compounds and targets. The workflow of the dataset preparation is summarized in Figure 1.
Figure 1. Workflow for dataset preparation
The workflow illustrates the process from data collection, preprocessing, and conversion to the model input format.
The binding affinity data were obtained from ChEMBL version 35, which includes new records absent from Boltz-2 training. To verify compound novelty in the Boltz-2 training data, the compounds in ChEMBL version 34, BindingDB, and PubChem (as of June 2023), which were used in the training, were also retrieved. The amino acid sequences of the target proteins were obtained from PDB as of September 3, 2025.
Extraction of benchmark compounds
To identify compounds that were not included in the Boltz-2 training data, compound differences between ChEMBL version 35 and version 34 were verified based on ChEMBL IDs, and the compounds only in version 35 were obtained. The compounds overlapping with those in BindingDB or PubChem were removed based on exact InChIKey matches using UniChem [13]. The remaining compounds were defined as benchmark compounds for the external test.
Chemical structure standardization
The compound structures were obtained as SMILES strings. The structures were desalted and standardized using the ChEMBL Structure Pipeline version 1.2.2 [14]. For multi-component compounds, only the largest fragments were retained.
Affinity values
Boltz-2 is trained on all available affinity records irrespective of activity type. Therefore, its predicted affinity should be interpreted as a generic affinity estimate rather than as corresponding to any single activity type. Experimental affinity measurements were obtained as pChEMBL values, defined as the negative logarithm of activity values (Ki, Kd, IC₅₀, EC₅₀, XC50, AC50, ED50, and potency in molar units) [15]. When multiple affinity data points existed for the same compound–target pair, the median pChEMBL value was used to reduce inter-assay variability; for an even number of data points, the median was calculated as the mean of the two middle values. Each record included a compound structure in the SMILES format, a protein target identifier annotated by its UniProt accession, and the affinity value for the compound–target pair.
Target proteins
Only single protein targets with ≥ 10 compounds were included. The amino acid sequences of the target proteins with ≥ 50 amino acids were obtained from the PDB based on the UniProt IDs, as provided in the ChEMBL target information, using PostgreSQL. The targets absent from PDB were excluded. The targets with ≥ 1,500 amino acids were excluded due to the high computational costs. The relational data for the UniProt IDs and sequences used are provided in Supplemental Data S1.
Dataset for evaluating implementations and parameter settings
To assess the differences between implementations (original Boltz-2 vs. NIM-original) and parameter settings (NIM-original vs. NIM-light), two small subsets (datasets A and B) were selected by narrowing down targets to reduce the computational costs. Dataset B included 91 targets with ≥ 30 calculable compounds each, resulting in 8,350 data points. Dataset A comprised 16 representative targets in terms of the sequence length, randomly selected from dataset B, and contained 963 data points.
Benchmark dataset and its novelty
The maximum structural similarities of the benchmark compounds relative to the compounds in ChEMBL version 34 (a large part of the training data of Boltz-2) were calculated using Morgan Fingerprints (radius=2, bit=2,048) with RDKit version 2024.09.2 [16] in KNIME version 5.7.0. [17].
To calculate the sequence novelty of the targets used in the benchmark dataset relative to those used in the Boltz-2 training, protein BLAST (2.17.0+) search [18] was performed against the PDB proteins released before June 1, 2023. The sequence identity of the search result for the highest bit score was used as the novelty index.
The final benchmark dataset comprised 356 unique target proteins (UniProt ID) and 10,933 unique compounds (ChEMBL ID), each associated with experimentally measured pChEMBL values (total 15,465 compound–target pairs, Supplemental Data S2). The benchmark dataset provides an independent resource for evaluating the generalization, extrapolation, and reliability of Boltz-2 affinity predictions. A KNIME workflow for creating the benchmark dataset is available in Supplemental Data S3.
2.3 Performance metrics
Performance metrics comparing predicted affinity values with experimental pChEMBL values, including MAE and RMSE, were calculated using Scikit-learn version 1.6.1 [19], while Pearson r, Spearman ρ, and Kendall τ were calculated using SciPy version 1.16.2. [20] as:
| (1) |
| (2) |
| (3) |
| (4) |
| (5) |
where n = number of pairs of values;
3.1 Comparison of the calculation time and affinity prediction results between the original and NIM implementations and between the two parameter settings
To verify the effects of the implementation and parameter differences, calculations were performed using three settings: original Boltz-2, NIM-original, and NIM-light. To compare the predictive behavior between the implementations (original Boltz-2 and NIM-original) and between the parameter settings (NIM-original and NIM-light), we analyzed their computational efficiency and predicted affinity values using datasets A (963 data points) and B (8,350 data points), respectively.
Calculation time
The calculation time for the total predictions, including both structure and affinity predictions, was measured (Figure 2 and Supplemental Data S4). The results for targets of different sequence lengths (73–1,280 amino acids) indicated that the longer the target protein sequence, the longer the calculation time. The calculation using NIM was approximately 1.9–7.6 times faster than the original Boltz-2. The use of NIM-light setting further decreased the calculation time compared to the NIM-original setting; the NIM-light was approximately 2.4–14.1 times faster than the original Boltz-2.
Figure 2. Comparison of calculation time in the different implementations and parameter settings
The average calculation time per compound is plotted for each target. The calculation time for the MSA creation is not included. The results for the original Boltz-2, NIM-original, and NIM-light are plotted as blue triangles, orange circles, and green squares, respectively.
Affinity prediction values and performance
The predicted binding affinity values, binding probabilities, and confidence scores were calculated for all compound–target pairs in the dataset A and B under the three settings (Supplemental Data S5). The predicted affinity value of the original Boltz-2 showed a high linear correlation with that of the NIM-original for the dataset A (Pearson r = 0.942), with a mean absolute difference of 0.242 ± 0.242 (Figure 3A and S1A). The differences in the predicted values between the two implementations were comparable to the variations caused by different random seed values, suggesting no substantial discrepancy. Furthermore, the prediction accuracy (MAE) for the affinity values of all the calculated compound–target pairs in the dataset A was similar between the original Boltz-2 and NIM-original, representing 0.891 and 0.907, respectively.
The NIM-light setting also exhibited a high correlation with the NIM-original setting (Pearson r = 0.974 and the mean absolute difference of 0.149 ± 0.166 for the 8,350 pairs, Figure 3B, S1B, and Supplemental Data S5) for the dataset B and maintained comparable prediction accuracy (MAE = 0.905 and 0.906 for the NIM-original and NIM-light, respectively). In addition, the three calculation settings were compared from the perspective of the binding probability and confidence score, and showed similar trends (Figure S2 and S3). These results indicate that the NIM-light offers the most efficient balance of computational speed and predictive accuracy among the three settings. Therefore, affinity predictions in the subsequent predictive performance analyses were performed using the NIM-light setting (Supplemental Data S6).
Figure 3. Comparison of affinity prediction results in the different implementations and parameter settings
(A) Comparison of affinity prediction results in terms of the implementations (original Boltz-2 and NIM-original). (B) Comparison of affinity prediction results in terms of parameter settings (NIM-original and NIM-light).
3.2 Predictive performance for each target
To evaluate the predictive performance of binding affinities of the compound–target pairs for each target and to obtain statistically reliable accuracy estimates, only targets with ≥ 30 compounds and at least 10 active and 10 weakly active/inactive compounds, classified based on pChEMBL value threshold of 7, were selected after removing not calculable compounds. This yielded 99 targets and 9,218 compound–target pairs for affinity prediction (Supplemental Data S6 and S7). The range of the pChEMBL values was 4.01–10.73 (Figure S4). The five predictive performance metrics (MAE, RMSE, Pearson r, Spearman ρ, and Kendall τ) were calculated for each target (Figure 4 and S5, and Supplemental Data S8). Spearman ρ and Kendall τ values showed trends similar to Pearson r, and RMSE showed trends similar to MAE (Figure S6). The Pearson r values for approximately 21.2% (21/99), 26.3% (26/99), and 32.3% (32/99) of targets resulted in > 0.6, 0.4–0.6, and 0.2–0.4, respectively (Figure 4A), indicating that the affinity prediction accuracy was moderate to acceptable for many targets. In addition, approximately 71.7% (71/99) of targets had a lower 95% confidence bound for Pearson r greater than zero (Figure 4A). Similarly, the MAE values for many targets of the benchmark dataset were moderate to acceptable, for approximately 68.7% (68/99), 29.3% (29/99), and 2.0% (2/99) targets resulted in < 1.0, 1.0–2.0, and > 2.0, respectively (Figure 4B). The distributions of Pearson r and MAE values for the benchmark dataset were shifted toward lower and larger values compared with those for a ChEMBL 34 compound–target pair dataset included in the Boltz-2 training, respectively (Figure S7).
Performance metrics were calculated using the median pChEMBL value as the ground truth when multiple affinity measurements were available for the same compound–target pair. Because the maximum value is also commonly used as a representative affinity measure, analogous calculations were performed using the maximum value (Supplemental Data S6). The performance metrics were comparable between the median- and maximum-based analyses for most targets, but differed slightly for some targets (Figure S8 and Supplemental Data S8).
pChEMBL values aggregate activity measurements obtained from different assay types such as IC50 and EC50. Accordingly, activity measurements from multiple activity types were used both to select a representative ground-truth value for each compound–target pair and to construct ground-truth values for each target. In the former case, only approximately 3.0% (273/9,218) of compound–target pairs had representative labels derived from multiple activity types (Table S1 and supplemental Data S2), suggesting a minimal impact of activity-type mixing on overall performance estimates. In the latter case, IC50, Ki, EC50, and Kd were the dominant activity types contributing to mixed activity-type records for individual targets (Table S1). Among these, IC50, Ki, and Kd reflect binding affinity, whereas EC50 represents functional potency. To evaluate potential systematic errors arising from the inclusion of EC50, analyses were repeated after excluding EC50 records. The evaluation dataset was reconstructed, and predictive performance was reassessed for targets selected using the same criteria as in the primary analysis (Supplemental Data S6). Predictive performance was largely comparable with and without EC50 records for most targets; however modest differences were observed for a subset of targets (Figure S9 and Supplemental Data S8), likely reflecting biases depending on activity types (e.g., most EC50 records exist in a limited range of activity values) and inconsistencies between EC50 and binding-affinity measurements.
3.3 Predictive performance for each target class
Predictive performance of affinity values was analyzed across sixteen ChEMBL target classes. Seven classes (kinase, family A G protein-coupled receptor, ion channel, lyase, transporter, transcription factor, and protease) overlapped with those in the original Boltz-2 benchmark [3], while nine target classes (hydrolase, oxidoreductase, phosphatase, transferase, membrane receptor, epigenetic regulator, cytochrome P450, other nuclear protein, and unclassified protein) were newly included. The predictive performance varied (mean Pearson r ranging from −0.373 to 0.829 and mean MAE from 0.848 to 1.784) according to the target classes and also varied within each target class (Figure 5 and Table 1).
Figure 4. Prediction accuracy of the affinity predictions for each target
The target IDs were sorted in descending order of prediction accuracy (A: Pearson r, B: MAE). Bars represent 95% CIs.
Figure 4(continued). Prediction accuracy of the affinity predictions for each target
Figure 5. Prediction accuracy of the affinity predictions for each target class
Pearson r (A) and MAE (B) for each target class are plotted as box plots. When the number of targets in a target class < 5, all the targets were plotted as dots.
3.4 Relationship between the affinity prediction errors and target proteins
The relationship between the prediction errors (actual pChEMBL values − predicted values) and the target sequence identity relative to the Boltz-2 training data is shown in Figure 6A and Supplemental Data S9. The prediction errors did not correlate with sequence identity (Pearson r = −0.046). Notably, even when the sequence similarity of the target approached 100%, large absolute prediction errors of > 2 were observed for several compounds. The prediction errors also did not correlate with the confidence score of the protein structure predictions (Pearson r = −0.092) (Figure 6B and Supplemental Data S10). The sequence identity did not correlate with the confidence score (Pearson r = 0.150, Figure S10).
3.5 Relationship between the affinity prediction errors and compound features
The relationship between the affinity prediction errors and compound features, including structural similarities to compounds in the reproduced Boltz-2 training data, molecular weights (MW), number of rotatable bonds (RB), number of heavy atoms (HA), hydrogen bond donors (HBD), and hydrogen bond acceptors (HBA), is shown in Figure 7 and Supplemental Data S11–16. No clear trend was observed between the prediction errors and the molecular features (Pearson r = 0.036, −0.027, −0.016, −0.024, −0.093, and 0.004 for compound similarity, MW, number of RB, HA, HBD, and HBA, respectively). The proportion of compounds with absolute prediction errors < 1.0 was 59.6, 60.2, and 67.6% for the compound similarities of 0.3–0.5, 0.5–0.7, and ≥ 0.7, respectively, suggesting the substantial generalization ability of the Boltz-2 affinity prediction model. Interestingly, although compounds with > 50 heavy atoms were not included in the Boltz-2 training, prediction errors for these compounds were not larger than those with ≤ 50 heavy atoms.
Table 1. The status and predictive performance result of each target class
Figure 6. Relationship between prediction error and target
Prediction error is plotted as box plot against target sequence identity (A) and model confidence score (B).
In this study, we developed a ChEMBL-derived benchmark dataset to independently evaluate the predictive performance of the Boltz-2 affinity prediction module. By constructing a systematic pipeline for removing overlapping compounds from the estimated training data, we ensured full data independence and reproducibility. The benchmarking results provide a valuable resource for analyzing Boltz-2 predictive accuracy and computational efficiency across a broad chemical and biological space.
The NIM implementation of Boltz-2 demonstrated a predictive performance comparable to the original Boltz-2 implementation while substantially improving computational cost. Furthermore, our analyses suggest that Boltz-2 exhibits broad applicability in terms of chemical structure similarity, delivering fair predictive performance. However, challenges remain in accurately predicting affinities for a considerable proportion of compound–target pairs.
This benchmark dataset represents an important step toward establishing standardized protocols for assessing AI-based molecular interaction models. By making data, pipeline, and results publicly available, we aim to facilitate fair comparisons among emerging predictive approaches and to promote transparent, reliable integration of AI-based models into drug discovery workflows.
Figure 7. Relationship between affinity prediction error and compound features
Prediction error is plotted as box plot against compound features: Tanimoto similarity (Morgan fingerprints of radius=2 and bit=2,048) of the compounds to the reproduced Boltz-2 training data (A), MW (B), and number of RB (C), HA (D), HBD (E), and HBA (F). When the number of compound–target pairs < 5, all the pairs were plotted as dots.
This study was supported by the Japan Agency for Medical Research and Development Basis for Supporting Innovative Drug Discovery and Life Science Research (JP25ama121028); and the Japan Science and Technology Agency National Bioscience Database Center (JPMJND2401). This study was supported in part by the JSPS KAKENHI Grant Number 22K15258. We gratefully acknowledge NVIDIA Corporation for supporting the use of NIM in this work.
The authors declare no conflict of interest.
Supplemental Figures S1–S10 and Table S1 are available in supplemental_information.docx.
Supplemental Data (S1–S16) are downloadable at https://doi.org/10.5281/zenodo.18669539.
