Blaise Cronin - Beyond Bibliometrics: Harnessing Multidimensional Indicators of Scholarly Impact

Beyond Bibliometrics

Beyond Bibliometrics

Harnessing Multidimensional Indicators of Scholarly Impact

edited by Blaise Cronin and Cassidy R. Sugimoto

The MIT Press
Cambridge, Massachusetts
London, England

2014 Massachusetts Institute of Technology

All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher.

Library of Congress Cataloging-in-Publication Data

Beyond bibliometrics : harnessing multidimensional indicators of scholarly impact / edited by Blaise Cronin and Cassidy R. Sugimoto.

p. cm.

Includes bibliographical references and index.

ISBN 978-0-262-02679-6 (hardcover : alk. paper)

ISBN 978-0-262-32329-1 (retail e-book)

1. Bibliometrics. 2. Bibliographical citations Evaluation. 3. Scholarly publishing Evaluation. 4. Scholarly electronic publishing Evaluation. 5. Scientific literature Evaluation. 6. Research Evaluation Statistical methods. 7. Communication in learning and scholarship Technological innovations. I. Cronin, Blaise, editor of compilation. II. Sugimoto, Cassidy R., editor of compilation.

Z669.8B49 2014

010. 72'7 dc23

2013027449

EPUB Version 1.0

d_r0

Preface

The etymology of bibliometrics consists of the Greek words for book (biblos) and measure (metron). In recent decades the subject matter of concern to bibliometricians has expanded well beyond books to include scholarly journals and journal articles, authors and institutions, bibliographic references, citations, acknowledgments, patents, and much more. With the advent of the Web, electronic journals, digital libraries, citation databases, and social media, we are now much better positioned to see what scholars produce (from monographs to videos, from datasets to tweets), where they publish, how they communicate with their various audiences, how their work is received, critiqued, and used, whether and in what ways their diverse contributions are acknowledged, how influential they are within and across different intellectual communities, and whether or not their oeuvre has an impact over time, and, if so, precisely what kind of impact. The we above refers, of course, to any interested party, be it an individual scholar, university administrator, budget director, federal funding agency, or national research council, curious to know more about the immediate or long-term outcomes and effects of research, whether personal or institutional in nature, funded or unfunded.

Such is the variety of digital trace elements that can be tracked and measured using currently available tools and platforms that the term bibliometrics can seem a mite anachronistic when placed alongside younger lexical cousins, such as webometrics and scientometrics. But in defense of the term bibliometrics we would merely note that both of these neologisms themselves have quite specific and limiting referents. For that reason, we have decided to stick with Bibliometrics (albeit prefaced with Beyond) in the title of this book, it being a term that has both pedigree and currency. We see it as the logical starting point for any serious discussion of the newer metricsoften grouped under the rubric alternative metricscurrently being proposed or already being developed and tested in the context of research assessment, faculty evaluation, or resource allocation exercises.

Bibliometrics, once the preserve of a small population of information scientists and mathematicians, is now a sprawling, still fast-growing specialty, a multidisciplinary mix of methodologists, conceptualists, policy analysts, software developers, evaluators, and application specialists, along with a sprinkling of critical theorists. The heady enthusiasm that sometimes characterizes the field should not, however, cause us to lose sight of the fundamentals. Algorithmic and statistical refinement matters not a jot if the metrics in question do not measure what they claim to measure or are applied inappropriately: fitness for purpose needs to be demonstrated, not presumed. Despite the many technical advances recorded in the literaturethink, for instance, of all the papers published in the last five years alone on Jorge Hirschs h-index and its numerous derivativesissues relating to validity (and, no less, reliability) continue to plague blibliometrics.

Do the tools we use and the indicators we favor measure what we claim or believe them to measure, and, if so, are those measures reliablethat is, capable of producing consistent, and, ideally, transparent results? Are we not sometimes so seduced by the incrementing sophistication of our procedures (data capture and cleaning, weighting, normalization, multivariate analysis, modeling, visualization) that the technical tail could almost be said to be wagging the disciplinary dog? Predictably, opinions are cleft and debate on the pros and cons of evaluative bibliometrics is as eristic as ever. But as the stakes rise, by which we mean that the progressive institutionalization of metrics-based assessment in both academe and science policy circles has become an ineluctable trend, so, too, does the need to explore the ramified ethical and cultural consequences of relying on metrics of one kind or another to capture putative evidence of research quality, scholarly impact, and academic influence.

Beyond Bibliometrics showcases a diversity of recent advances in metrics-based research, both theoretical and applied, and also presents a number of penetrating critiques of the subject. The chapters in this collection reflect the Janus-faced nature of (biblio)metrics research: the proliferation of techniques and applications demonstrates that the field is flourishing, yet critics highlight the considerable potential for misuse and abuse. It is our hope that Beyond Bibliometrics will help promote critical reflexivity within the field and foster a more enlightened appreciation of the pros and cons of metrics-based assessment among relevant communities of practice.

Acknowledgments

We are grateful to Jylisa Doney and Andrew Tsou for their assistance with bibliographic checking.

I

History

Scholars and Scripts, Spoors and Scores

Blaise Cronin

Est modus in rebus.

Horace, Satires 1.1.106

Communicating Science

In medieval Europe, scientific news traveled at a measured pace, as slow or as fast as the peripatetic scholars who journeyed from one center of learning to another. The medium was the messenger. With the introduction of postal services in the sixteenth and seventeen centuries the speed of dissemination picked up. Letter writing gradually became the dominant form of exchange between gentlemen scientists such as the chemist Robert Boyle, the statistician William Petty, and the other grandees of the Scientific Revolutions early days. Their correspondence was often channeled through what might today be termed clearinghouses, there being copied for wider distribution and subsequent reading at local scientific gatherings (Manten, 1980). In Paris, Friar Marin Mersenne, variously referred to as the mailbox of Europe (Hatch, 2000, p. 265) or chief philosophical intelligencer of his time (Dear, 2000, p. 668), was the energetic embodiment of a communications hub; he corresponded with many of the leading minds on the continent, including Descartes, Galileo, Huygens, and Pascal. In London, Henry Oldenburg performed a similar role with industry and distinction.

Members of these invisible collegesinformal networks of natural philosophers, unaffiliated with formal institutions of learning (e.g., Crane, 1972; Lomas, 2002; Wagner, 2008)would assemble at private dwellings, coffee shops, or taverns to conduct their business and exchange news of the latest scientific findings. From such loose arrangements evolved the national academies of the seventeenth century, the Royal Society in London and the Acadmie Royale des Sciences in Paris. Shortly after their establishment, both bodies created in-house journals, the