Reproducibility
In 1660, Boyle published his landmark book, New Experiments Physico-Mechanical, Touching the Spring of the Air and its Effect. In the book, he described the first controlled experiments of the effects of reducing the pressure of the air. He gave the purpose of the book:
” . . . I thought it necessary to deliver things circumstantially, that the Person I addressed them to might, without mistake, and with as little trouble as possible, be able to repeat such unusual Experiments . . . ”
— Boyle. New Experiments Physico-Mechanical, touching the Spring of the Air. (The page containing the quotation is here, courtesy of the Text Creation Partnerhsip).
Three hundred years later, in Caltech’s 74 commencement address, Feynman talked about science that lacked integrity and transparency. He started to coin the phrase “Cargo Cult Science” that became one of the most famous things that he had said:
“In the South Seas there is a cargo cult of people. During the war they saw air planes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to imitate things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas–he’s the controller–and they wait for the air planes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No air planes land.”
Regarding integrity and transparency, he went on to say:
“If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.”
“In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgement in one particular direction or another.”
Preproducibility
Science must be falsifiable and reproducible. A scientific result must be in sufficient detail for readers to repeat the research. Communicating the result requires enumerating, recording and reporting those things that cannot be omitted.
In 2018, Philip Stark observed that much modern communications do not meet the standard of falsifiable and reproducible. One contributing factor is the lack of standard terminology to describe the situation. Researchers give different meanings to a single term, such as, reproducible, replicable, repeatable and generalizable.
Philip Stark proposed the word preproducibility to reduce confusion.
“An experiment or analysis is preproducible if it has been described in adequate detail for others to undertake it. Preproducibility is a prerequisite for reproducibility, and the idea makes sense across disciplines.”
“The distinction between a preproducible scientific report and current common practice is like the difference between a partial list of ingredients and a recipe. To bake a good loaf of bread, it isn’t enough to know that it contains flour. It isn’t even enough to know that it contains flour, water, salt and yeast. The brand of flour might be omitted from the recipe with advantage, as might the day of the week on which the loaf was baked. But the ratio of ingredients, the operations, their timing and the temperature of the oven cannot.”
Epilogue
Philip Stark concluded:
“Science should be ‘show me’, not ‘trust me’; it should be ‘help me if you can’, not ‘catch me if you can’. If I publish an advertisement for my work (that is, a paper long on results but short on methods) and it’s wrong, that makes me untrustworthy. If I say: “here’s my work” and it’s wrong, I might have erred, but at least I am honest. If you and I get different results, preproducibility can help us to identify why — and the answer might be fascinating.”
“The commitment that Boyle made to the scientific community is even more crucial today.”
References
Richard P. Feynman. Cargo Cult Science. Engineering and Science, June 1974, pages 10 – 13.
Philip Stark. Before Reproducibility Must Come Preproducibility. Nature 557, 613 (2018)