Ellison Aaron M. - 2018;2017;

STEPPING IN THE SAME RIVER TWICE

OTHER BOOKS BY AYELET SHAVIT

One for All? Facts and Values in the Debates over the Evolution of Altruism (Jerusalem, Israel: The Magnes Press, 2008)

OTHER BOOKS BY AARON M. ELLISON

A Primer of Ecological Statistics (Sunderland, MA: Sinauer Associates, 2012)

A Field Guide to the Ants of New England (New Haven, CT: Yale University Press, 2012)

EDITED BY AYELET SHAVIT AND AARON M. ELLISON
WITH A FOREWORD BY W. JOHN KRESS

REPLICATION IN BIOLOGICAL RESEARCH

Copyright 2017 by Yale University. All rights reserved. This book may not be reproduced, in whole or in part, including illustrations, in any form (beyond that copying permitted by Sections 107 and 108 of the U.S. Copyright Law and except by reviewers for the public press), without written permission from the publishers.

This volume is a contribution of the Long Term Ecological Research (LTER) program funded by the U.S. National Science Foundation. The twenty-five LTER sites, which encompass diverse ecosystems from Alaska and Antarctica and include islands in the Caribbean and the South Pacific, make up the largest and longest-running ecological research network in the world. The LTER network of sites serves as a global model for transparent and reproducible ecological and environmental research.

Yale University Press books may be purchased in quantity for educational, business, or promotional use. For information, please e-mail (U.K. office).

Set in Scala and Scala Sans type by Westchester Publishing Services. Printed in the United States of America.

Library of Congress Control Number: 2016951790
ISBN 978-0-300-20954-9 (hardcover : alk. paper)

A catalogue record for this book is available from the British Library.

This paper meets the requirements of ANSI/NISO Z39.48-1992 (Permanence of Paper).

To Ran, Ofri, Tal, Rotem, and Frodo,
who remain unique and unreproducible,
and
To the past, present, and future practitioners of Open Science

History does not always repeat itself. Sometimes it just yells, Cant you remember anything I told you? and lets fly with a club.
John W. Campbell, from Analog Science Fiction and Science Fact (1965)

W. John Kress

PART ONE. INTRODUCTION: REPLICATION ACROSS
DISCIPLINES

Ayelet Shavit and Aaron M. Ellison

Yemima Ben-Menahem

Haim Goren

The Value of Natural History Collections

Tamar Dayan and Bella Galil

Rebecca J. Rowe

Repeatable Monitoring and Observations

Avi Perevolotsky, Naama Berg, Orit Ginzburg, and Ron Drori

Ron Drori, Naama Berg, and Avi Perevolotsky

Yonathan Shaked and Amatzia Genin

Replication and Experiments

Aaron M. Ellison

Jacob Pitcovski, Ehud Shahar, and Avigdor Cahaner

Meta-analysis and the Need for Repeatability

Leonard Leibovici and Mical Paul

Mical Paul, Yaara Leibovici-Weissman, and Leonard Leibovici

The Role of Metadata in Creating Reproducible Research

Kristin Vanderbilt and David Blankman

R Emery R. Boose and Barbara S. Lerner

Morgan W. Tingley

Barbara Helm and Ayelet Shavit

Aaron M. Ellison

Ayelet Shavit

Here is the idealized scientific method at work: (1) a testable hypothesis is constructed to determine a general principle that explains some initial observations; (2) an experiment is devised that will generate data to test the hypothesis; (3) the experiment is conducted and data are collected; (4) the data are analyzed; (5) the experiment is repeated until sufficient evidence allows one to fully accept or reject the hypothesis that explains the original observations; and (6) the accepted hypothesis may result in new insights that require additional testing. This basic method enables scientific progress to be made over time. In some cases the initial hypotheses can be narrow; in other cases they can be broad and substantial. In all cases the experiments, the data, and the analyses need to be replicable, repeatable, and reproducible. Good science that follows this basic method enables us to understand the world around us.

In todays society, in which science is responsible for all aspects of our lives, from our understanding and utilization of nature to our health, to our economies and livelihoods, and to our national security, scrupulously adhering to this scientific method is essential for civilization to progress and for human populations to prosper. Yet in the first decades of the twenty-first century, the integrity of this scientific process, and of the scientists who depend on it, has in some cases and by some individuals been called into question. Are the scientific method and the reproducibility of scientific results being compromised in the highly competitive nature of todays world?

The question of reproducibility is the topic of Stepping in the Same River Twice: Replication in Biological Research, superbly edited by Ayelet Shavit and Aaron M. Ellison, a philosopher of science and a field ecologist, respectively. The editors and other contributors to this volume provide a broad spectrum of thoughts and opinions on the topic. It is a subject highly relevant to the role of science in society.

First of all, as clearly presented in the book, replication, replicability, repeatability, and reproducibility must be defined. Replication is a general term, denoting different concepts and practices. A replicate results from the duplication of the methods, the results, and the analyses of an experiment at the same timeframe as the original procedures are performed. There is no variance, no error, and no secondary interpretation. Everything is the same. Repeatability is the repetition of the original experimental process under similar, but not exact, laboratory, field, and environmental conditions, which in essence tests the original conditions of the experiment under new and controlled settings. Everything is similar but not quite the same. Reproducibility is the attempt to obtain the same results of an experiment at a different time and in a different space. Exact reproducibility is difficult to achieve and for the most part is infrequently attempted from one lab to the next.

The notions of replication, repeatability, and reproducibility are necessary to successfully engage in the scientific method. Replicates are vital to reducing experimental error. Repeatability is core to understanding the laws of nature, as they allow us to quantify how processes and features, such as evolution, ecology, physiology, and morphology, vary from place to place, from one point in time to the next, and from one individual to a population of individuals, within and between species. Reproducibility allows scientists to question why experiments differ from lab to lab and between environments and therefore serves as a test of the credibility of our results.

However, it would be a big mistake to equate the lack of reproducibility with poorly conducted science or even fraud. Repeatability is simply the basic nature of the business of science. Exact reproducibility is in some ways the pipe dream of a scientifically illiterate society looking for absolute truth in what it does not understand. For scientists, and for those in society who understand the scientific process, science is working well when irreproducible results and conclusions are revealed, when they are not repeatable, and when new insights about nature are the result. It is critical that we distinguish between experimental and observational error, natural variability in environmental conditions, and fraudulent misconduct in the scientific method.