The reproducibility crisis and what it means for the NIH

This post is a brief article by Dr Vinay Prasad, an American hematologist-oncologist and health researcher whose credentials are impressive:

The author of over 400 peer reviewed papers, including articles in the New England Journal of Medicine, JAMA, BMJ, Nature, the Annals of Internal Medicine and JAMA Internal Medicine. He has also written 2 peer reviewed books, published by Johns Hopkins University Press.

In the article quoted below he is taking on one of the primary ways fraud is committed in medical studies... reproducibility.

The words science and technology are frequently conjoined because technology is typically the end result of science. It's the primary test of science - can a technology that works be built? No? It's not science, even though it may continue to be called that. 

It is also extremely common that scientific theory is called science in mass media publications. Fraud is too strong a term for the misrepresentation of a theory to be science, the more accurate term for this very common practice would simply be lying.

Science is fact, but theory is only theory.

Well before the stage of working technology however is reproducibility. This is where other teams check the study by seeing if they can reproduce it in their labs. If they can't reproduce it, the study under test is "not science". At best it's "back to the drawing board", at worse it's a fail that won't be pursued further.

Not to belabor the point, the lack of reproducibility studies during the pandemic is one of the most egregious examples of this form of medical fraud ever.

Dr Prasad, writing on his substack, quoted below:

"The reproducibility crisis and what it means for the NIH

Why scientists broke the social contract"

When we say science doesn't reproduce, what does that mean?

It means that if you take the same experiment that's published in a top journal and you try to run it again--- the way you would take a recipe and try to bake the cake at home-- you don't get the same result— the cake flops.

How often does it happen?

A number of studies have found that this happens at least half the time, or, in some fields, even more often.

Why does it happen?

It happens because there's a culture where the people who do the daily work, often poorly paid postdocs and PhD students, are under immense pressure to generate results for their career. We don't incentivize truth, we incentivize discovery. Naturally, they may cherry-pick what data to show, or run the experiment many times and take the most favorable result, or even commit outright fraud.

What about oversight?

The senior investigator does not really care about the validity of the findings from their lab. I surmise this because if they did care they would make different people run the experiment many times, and they would try to replicate the results from their own laboratories, which they seldom do.

Is it still science if it doesn't replicate?

If a scientific finding does not replicate, it doesn't tell you anything true about the universe. It's just telling you about the idiosyncratic conditions under which it occurred, which is not knowledge, and pretty useless.

What does this mean for the social contract of Science?

Having such a large problem of reproducibility, and nearly no attempts to fix it, suggest that scientists have betrayed the social contract. They're happy to swallow up billions of dollars in grants, and have these billions increase over time, but they're not fundamentally interested in whether or not they're delivering true results.

But science still leads to cures?

This is absolutely true. Despite all the flaws in the system. There are still some true and useful discoveries. But that's not the question. The question is about whether or not each dollar can generate more true and useful results. That is almost certainly true.

Is reform needed?

It is a must