Most of us who are research scientists have had the experience of trying to reproduce another investigator’s experiment and failing. And if we admit it, we also sometimes have difficulty reproducing our own experiments (but rarely report it). In the biological sciences and especially in my field, ecology, reproducing the results of a study is difficult, if not impossible, due to the fact that we can rarely duplicate the precise conditions of the original experiment. Such is especially true of field experiments and even experiments conducted in a “controlled environment” such as a greenhouse. At least that’s always been my explanation for this non-reproducibility phenomenon.

Recently, there have been several news articles and blog posts about reproducibility of published scientific experiments. For example, two studies run by the pharmaceutical companies, Amgen and Bayer, showed that 70 to 90% of cancer studies published by academic scientists in reputable journals were not readily reproduced by other scientists assigned to repeat the experiments (see Reuters story for a summary and links). Those numbers are hard to believe, but such observations have prompted investigations into why so few studies are readily reproduced.

A recent paper proposed that one reason for the lack of reproducibility is weak statistical tests. Using a new method, Valen Johnson (Texas A & M) compared the strength of two types of statistical tests (read about the details in Nature News) and concluded that 17 to 25% of studies that use the common P-value of 0.05 may yield false results. Johnson suggests that the 0.05 cutoff, leading to false positive conclusions, may be the main reason for why subsequent experiments fail to reproduce the findings of the original study. In contrast, studies that use a P-value of 0.005 rarely fail to replicate. I’ve not read that paper but imagine that the news about the P-value of 0.05 will throw a lot of researchers into a tizzy.

Another hypothesis for non-reproducibility is offered by Moshe Pritsker (CEO of JoVE, the Journal of Visualized Experiments): lack of sufficiently detailed methods. His solution is for researchers to use video to publish scientific results (see his opinion piece in The Scientist). That is what JoVE publishes: peer-reviewed videos that are created from text-based research papers. The idea is that by showing experimental procedures and results in video rather than in (or in addition to) a written description, other scientists can more readily duplicate an experiment and confirm its findings.

Although JoVE is the only journal of its type, several other journals accept video as supplemental material, and such visual depictions of methods and experimental protocols should help to standardize techniques. My own experience is that even very detailed written descriptions of methods are not sufficient to allow someone else to replicate the study. Many important details are left out of the written description but that are readily visible in a video. How did the researcher position the sampling instrument? How hard or gently did they apply it? What did the field study site actually look like?

I’m obviously a big fan of video and its use in science. However, I don’t think video will necessarily eliminate the problem of non-reproducibility or that it can substitute for a well-written description. If you watch some of the free-access videos at JoVE, you see that the videos are really video abstracts (audio-visual summaries of the work) and that the text-based paper containing all the usual details and extended discussion of the results is published alongside the video. Both are necessary, in other words, for a full understanding of what was done, how it was done, what was found, and what it means.

My view is that video can be a critical element in research publications, but should be complimentary to the text in the same way graphs, diagrams, conceptual models, photographs, and other visualizations are in a more traditional article. That’s not to say that research cannot be reported entirely through the medium of video. Some types of research may lend themselves well to a video-type publication: new methods/standard protocols, descriptions of new species (or observed behaviors), and physical or biological phenomena that cannot be completely described in words.

Reading about how a heron uses bread as bait to catch fish is just not the same as seeing it (and some people will want visual evidence of a new or unusual phenomenon to accept it):

Whether video will affect reproducibility of studies must await further data. In the meantime, I think a better reason to use video is to more clearly depict methods and results and provide visual evidence of observed phenomena. In the past, video was usually out of the question for the average researcher because of the expense and skills required. Today, with inexpensive digital recording devices and simple but powerful editing software, anyone can create a reasonably good video to illustrate a method or other aspect of a research project.

In the next post, I’ll describe my own experience with respect to video and scientific methods.