F.5 Chemical Novelty

One of the primary concerns with using LLM models in scientific discovery is their capacity to simply memorize and regurgitate their training sets instead of performing logical reasoning [105]. Since recent models like gpt- 4o are trained on massive, opaque datasets, a key concern is that the molecules TEXTGRAD generates may simply be duplicates of clinically approved molecules for their respective targets. To quantify and test the extent or existence of this "memorization" effect, we compare the molecules that TEXTGRAD generates with approved drugs for their respective protein targets, as well as unrelated but existing druglike chemical compounds.
We first compare TEXTGRAD generated molecules for a particular protein target with all the clinically approved small molecules found in Drugbank for the same target (see F.3 for filtering criteria) using the Tversky similarity score on RDKit daylight chemical fingerprints. The Tversky similarity score is not symmetric, and compares substructures between a variant and reference molecule [106]. In our application, this is preferred as it assigns a high score to generated (variant) molecules that contain most or all of the sub structures in the Drugbank (reference) molecule, even if there exist other extraneous substructures in the generated molecule that reduce the symmetric overlap between the two molecules. A Tversky score of 1.0 indicates that the TEXTGRAD molecule is a complete subset of the Drugbank molecule, while 0.0 indicates no overlap.
In order to analyze the most relevant chemical structures, we restricted our analysis to a set of high performing TEXTGRAD molecules, rather than all molecules from all iterations. In particular, using the overall score described in Equation 25, we selected the best molecule for each protein target and for each initial fragment, for a total of 87 generated molecules (29 drugable targets initial fragments) and 118 clinically approved drugs across 29 targets. For each target, we compute all pairwise Tversky scores between the 3 TEXTGRAD molecules and the Drugbank molecules approved for that target, setting the Drugbank molecule as the reference, and the TEXTGRAD molecule as the variant.
We observed that the distribution of Tversky similarity scores was quite broad, with a median of 0.42 but ranging from 0.14 to 0.90 (Figure 2 (b)). However, we observe that Tversky similarity between TEXTGRAD and DrugBank molecules is actually slightly anti- correlated with molecule performance as measured by the overall score, suggesting that TEXTGRAD 's optimization procedure is at best weakly influenced by prior knowledge of approved drugs. In fact, for a variety of generated molecules along a gradient of similarity scores, the generated molecules exhibit QED and Vina scores that match or even exceed their DrugBank counterparts (Figure 2 (d)).
Beyond the similarity to known drugs, we are also interested in observing if TEXTGRAD is generating novel molecules, or discovering previously unknown properties in existing compounds, for example a strong binding affinity to a protein target in a compound that has not been associated with the protein. To answer this question, we perform a Tanimoto similarity search across all of ChEMBL, a manually curated database of 2.4 million bioactive molecules with drug- like properties [107]. Unlike the Tversky score, the Tanimoto metric measures the symetric overlap between two chemical structures, where 1.0 indicates an exact match, and 0.0 no overlap. We classify a TEXTGRAD compound as "novel" if and only if there does not exist any molecule in ChEMBL with a with a Tanimoto similarity score over 0.80.
While the starting fragments are known molecules, we observe that as the number of iterations increases, TEXTGRAD generates molecules that are progressively less likely to be previously known compounds. By the 6th iteration, 95% of all the molecules generated by TEXTGRAD across all 58 targets and 3 starting fragments are novel using the criteria above (Figure 2 (a)). By observing the trajectories of generated molecules and analyzing the textual gradients, we hypothesize that the feedback provided by the Vina and QED scores encourages TEXTGRAD to explore chemical space beyond known molecules by progressively adding and removing combinations of functional groups. Since these chemical updates can form a combinatorial number of unique structures, it is reasonable that TEXTGRAD would reach previously unexplored regions of chemical space in a relatively small number of iterations.