Marc W. Vinyard - Demystifying Scholarly Metrics: A Practical Guide

DEMYSTIFYING SCHOLARLY METRICS

A Practical Guide

Marc W. Vinyard and Jaimie Beth Colvin

Copyright 2022 by Marc W. Vinyard and Jaimie Beth Colvin

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, except for the inclusion of brief quotations in a review, without prior permission in writing from the publisher.

Library of Congress Cataloging-in-Publication Data

Names: Vinyard, Marc W., author. | Colvin, Jaimie Beth, author.

Title: Demystifying scholarly metrics : a practical guide / Marc W. Vinyard

and Jaimie Beth Colvin.

Description: Santa Barbara, California : Libraries Unlimited, [2022] |

Includes bibliographical references and index.

Identifiers: LCCN 2021028546 (print) | LCCN 2021028547 (ebook) | ISBN

9781440875939 (paperback) | ISBN 9781440875946 (ebook)

Subjects: LCSH: BibliometricsHandbooks, manuals, etc. | Scholarly

publishingEvaluation. | ResearchEvaluationStatistical methods.

Classification: LCC Z669.8.V56 2022 (print) | LCC Z669.8 (ebook) | DDC

020.72/7dc23/eng/20211028

LC record available at https://lccn.loc.gov/2021028546

LC ebook record available at https://lccn.loc.gov/2021028547

ISBN: 978-1-4408-7593-9 (paperback)

978-1-4408-7594-6 (ebook)

26 25 24 23 221 2 3 4 5

This book is also available as an eBook.

Libraries Unlimited

An Imprint of ABC-CLIO, LLC

ABC-CLIO, LLC

147 Castilian Drive

Santa Barbara, California 93117

www.abc-clio.com

This book is printed on acid-free paper

Manufactured in the United States of America

Contents

Why Are You Reading This Book?

Goodharts law When a measure becomes a target, it ceases to be a good measure.

Marilyn Strathern

This chapter is asking all the stakeholders with an interest in scholarly metrics (librarians, administrators and department chairs, and researchers) to stop and consider why they are interested in this topic and what they hope to gain from understanding journal ranking and other scholarly metrics. These are complex, but imperfect measurements. When used appropriately, metrics can help users make informed decisions or describe work accomplishments. When metrics are used to answer questions they were never designed to answer they become meaningless, or worse become an onerous burden.

LIBRARIANS

Dont push your agenda. Provide a service; dont create problems.

When we initially investigated how libraries might provide research evaluation services, we noticed two groups emerging from the literature. One group believed libraries should assist with bibliometric services because it was a natural expansion of library services, while the other group focused on the strategic benefits of providing such services. They argued that if libraries provide scholarly metrics services, they would demonstrate their worth and reestablish their value to administrators who increasingly rely on metrics. However, coupled with the voices warning that libraries risked becoming irrelevant if they didnt evolve to incorporate bibliometric analysis services was a second group warning that the strategic plan to stay relevant might backfire if faculty associate librarians with evaluation reports and failed promotions. Those expressing wariness didnt oppose libraries offering research evaluation services; instead, the message was to proceed with caution and make sure that faculty would welcome these services.

Since our initial investigation, the literature has shifted from Should libraries provide scholarly metrics services, and if so why? to Which services do we provide and how? The Association of College and Research Libraries (ACRL) updated their information literacy competency standards to a framework model that includes Scholarship as Conversation as one the frames, thus recognizing the cycle of research that goes beyond investigation (research) and creation (writing) to sharing (publishing) and communicating (citing, responding, etc.) (Brantley et al., 2017, p. 140). The scholarly communication framework illustrates the ongoing cycle of research, and librarians have recognized needs at each stage of this cycle that they previously focused onspecifically the stages of publishing and disseminating research, which have increasingly grown complex (Si et al., 2019; Ye, 2019). The need to navigate the expanding world of research metrics has created an opportunity for librarians to expand their services to include tracking impact and interpreting metrics (Brantley et al., 2017; Howie & Kara, 2020; Powell & Elder, 2019). The need is great enough that many libraries are hiring librarians specifically to help faculty navigate the realm of measuring scholarly output (Powell & Elder, 2019). As libraries increase their involvement in the stages of research evaluation and analyzing impact, they have the chance to chase the hottest trends or create faculty center services (Brantley et al., 2017).

As more libraries move to incorporate scholarly metric services, past cautions may seem irrelevant. Yet, we believe vigilance is still called for as libraries expand research support to include research evaluation and metric analysis services because these services are closely related to professional development and promotions. Librarians must assess whether the programs meet needs or create problems. For example, if librarians attempt to push mandates for faculty to publish in gold open access (OA) journals that either limit their publication options or require high article processing charges (APCs) faculty cant afford, theyve created another hurdle to publishing instead of making the process easier. But if faculty are interested in learning more about OA or already decided to prioritize OA journals, then a service that helps faculty publish in OA journals is providing a service that removes hurdles (see for tips on identifying faculty needs).

Meet a need; dont create a problem.

ADMINISTRATORS AND DEPARTMENTAL CHAIRS

Dont misapply metrics and create a hostile research environment.

Are you hoping a move toward emphasizing bibliometrics will make evaluating individuals, groups, and the whole institution easier? Are looking for the metrics that will help you evaluate your institutions scholarly strength in comparison to peer institutions? Are you looking to standardize scholarly metrics to help you compare departments? Are you looking to use metrics to identify research with the most impact potential so your institution allocates funding efficiently?

Consider the research environment your decisions will create. Pay close attention to the initial purpose of the metric and what its intended to measure before you start using it to measure something else entirely. Understand metrics within their context to avoid unfair comparisons. By ignoring the limitations of metrics, you might pick academic winners and losers by valuing subjects that are easier to measure. Remember college applications are a compilation of SAT scores, essays, references, and supplemental information in an attempt to create a complete picture. Additionally, there is a reason job applicants submit a cover letter, resume, and application and run the gauntlet of interviews: because employers dont see the full picture from one metric. Likewise, author-level metrics and research output reports cant tell the whole story. We learned that some larger research universities place a high emphasis on improving their institution ranking through scholarly output. These universities make appointment decisions based on evaluating the talent of candidates by producing reports that compare the papers of candidates with scholars in the same discipline; these analysis reports are forwarded to the human resources office (Ye, 2019). Metrics are helpful, but they are a piece of the puzzle, not the whole picture.