As part of a project I was wondering about reports of surveys that collected chemists assessments of differnt things. More specifically, I wasn’t looking for crowd-sourcing efforts for data curation (such as the in the Spectral Game) or data collection. Rather, I was interested in reports where somebody asked a group of chemists what they thought of some particular molecular “feature”. Here, “feature” is pretty broadly defined and could range from the quality of a probe molecule to whether a molecule is complex or not.

Surveying the literature (and with pointers from @dgelemi, @baoilleach, Jun Li, @georgeisyourman and @DrBostrom) here’s the following papers:

• Brown et al, J. Med. Chem., 2015, “Understanding Our Love Affair with p-Chlorophenyl: Present Day Implications from Historical Biases of Reagent Selection“
• Li et al, Org. Biomol. Chem, 2015, “Current complexity: a tool for assessing the complexity of organic molecules“
• Franco et al, J. Cheminf., 2014, “The use of 2D fingerprint methods to support the assessment of structural similarity in orphan drug legislation“
• Sheridan et al, JCIM, 2014, “Modeling a Crowdsourced Definition of Molecular Complexity“
• Litterman et al, JCIM, 2014, “Computational Prediction and Validation of an Expert’s Evaluation of Chemical Probes“
• Kutchukian et al, PLoS ONE, “Inside the Mind of a Medicinal Chemist: The Role of Human Bias in Compound Prioritization during Drug Discovery”
• Hack et al, JCIM, 2011, “Library Enhancement through the Wisdom of Crowds“
• Oprea et al, Nat. Chem. Biol., 2009, “A crowdsourcing evaluation of the NIH chemical probes“
• Ertl et al, J. Cheminf., 2009, “Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions“
• Boda et al, JCAMD, 2007, “Structure and reaction based evaluation of synthetic accessibility”
• Lajiness et al, J. Med. Chem, 2004, “Assessment of the Consistency of Medicinal Chemists in Reviewing Sets of Compounds“
• Takoaka et al, JCICS, 2003, “Development of a Method for Evaluating Drug-Likeness and Ease of Synthesis Using a Data Set in Which Compounds Are Assigned Scores Based on Chemists’ Intuition”

The number of people surveyed across these studies ranges from less than 10 to more than 300. Recently there appears to be a trend towards developing predictive models based on the results of such surveys. Also, molecular complexity seems pretty popular. Modeling opinion is always a tricky thing, though in my mind some aspects (e.g., complexity, diversity) lend themselves to more robust models than others (e.g., quality of a probe).

If there are other examples of such surveys in chemistry, I’d appreciate any pointers