Joe Olechno, Ph.D., is a senior research fellow at Labcyte Inc., Sunnyvale, CaLif. He has over 30 years of experience in the development of scientific instruments.
Everyone knows and, maybe unfortunately, almost everyone feels able to discern garbage from quality. But the Scottish comedian Rory Bremner may be more correct when he said, “If you put garbage in a computer, nothing comes out but garbage. But this garbage, having passed through a very expensive machine, is somehow ennobled and none dare criticize it.”
The annual global market for high-throughput screening (HTS) in 2017 is estimated to be $19.9 billion. But is that money being spent wisely? Or is it an exercise of “garbage in, garbage out” that is clothed in the emperor’s finery? Are the results from existing HTS studies providing biologically meaningful data—data that is predictive and directs research down fruitful paths?
A recent study, of which I am a coauthor, looked at dose-response (IC50) experiments performed either with the old standby of serial dilutions made with tip-based pipettes or those same experiments but with the concentrations made with a newer technique, direct dilution using acoustic liquid handling. The data in the study was produced by researchers at AstraZeneca and was published in a patent and patent applications. The differences in IC50 values were huge (some more than 250-fold different). But there was no obvious systematic error. There was no correlation with hydrophobicity, charge, solubility, or structure.
Of course, the pharmacophores generated by the two different structures were also very different. The critical result occurred when the two different pharmacophores were tested for predictability on a new set of compounds. The traditional serial dilution technique had zero predictive capability while the direct dilution/acoustic technique was able to correctly predict the order of activity.
The final nail in the coffin was the comparison of the two pharmacophores to pharmacophores developed via X-ray crystallography of the receptor. The pharmacophore developed from the serial dilution technique differed significantly from the X-ray generated model. The direct dilution/acoustic technique yielded a pharmacophore that substantially matched the X-ray structure.
What has been amazing is the response to the article—not a sudden rush to confirm the results or a change in techniques—but, rather, absolution of the serial dilution technique. To the suggestion that errors of this sort may be due to the compounds being very hydrophobic, we note that the compounds analyzed ranged in logP from 2.83 to 4.38 with the greatest deviations at logP of ~3.5. Is this a rare exception and unlikely to be repeated with other compounds and other targets? That is doubtful. The study points out other research encompassing other targets and over 10,000 compounds tested. In this other research there was a strong bias towards compounds showing greater potency when the direct dilution/acoustic technique was used. In one case, the direct dilution/acoustic technique identified 46% more active compounds than the traditional method did. Variability is often accepted in biological systems. But the variability is two- or three-fold. In the study, we showed differences greater than 200-fold. The argument that the automated pipette system has been calibrated and cannot generate errors of the magnitude seen misses the point. It is not that the volume transferred is incorrect; it is that solute is being lost so the assays do not contain the amount of compound expected. Finally, potential users have argued that the acoustic technology is too costly anyway. But I have to ask how that cost compares to running hundreds of thousands of meaningless assays or missing new potential blockbuster drugs.
The authors, including myself, suggest that metadata about the liquid handlers and dilution techniques be included in databases. We also urge pharmaceutical companies with more data to confirm or refute our findings.
How likely is this to have happened in your company or with your clients? The article suggests that this has been noted by major pharmaceutical companies but not reported in peer-reviewed journals. Have the dilution techniques/liquid handling processes led people away from the correct leads and down cul-de-sacs? Is this the exception or the rule?