Reproducibility in experimentation – the implications for regulatory toxicology

Abstract

A paper based on the Sir William Paton lecture of the British Society of Toxicology, 2014.

Sir William Paton's classical exploitation of nitrogen in mixtures of gases for deep-sea diving is a good example of what would now be called “blue skies” research where the data obtained were used as the basis for subsequent hypothesis-driven work. Like his work on acetyl choline, the original observations were the basis for further painstaking and step-like progression by experimentation, where each I was dotted and T crossed as the establishment of a therapy or practice was built into a clinical framework or workplace. However, in recent times there has been a failure to follow a similar cautious and pragmatic practice of advancement of knowledge at the interface of Science and Policy. Inadequately investigated associations are made the basis of speculative therapeutic or dietary interventions in frameworks where lack of effect is often not demonstrable and where the possibility of harm is seldom considered, notably in the dietary field. In the same way, isolated findings in regulatory studies may lead to poor decision making or to extensive work to explain non-significant events. The Economist has pointed out that venture capitalists use the rule that only 50% of academic study results can be replicated and Begley and Ellis report that a study of 53 “landmark” papers in Oncology found that only 6 were reproducible even with co-operation of the original authors. So what notice should be taken of single adverse toxicological studies in animals? Of isolated, epidemiologically determined associations? What is the scientific value of an individual regulatory study? What is the probability of error? What is the role of chance and variation? Only an understanding of mechanisms of production of the effect in question can resolve these issues.