EVALUATION
A CULTURAL SYSTEMS APPROACH
MARY ODELL BUTLER
First published 2015 by Left Coast Press, Inc.
Published 2016 by Routledge
2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
711 Third Avenue, New York, NY 10017, USA
Routledge is an imprint of the Taylor & Francis Group, an informa business
Copyright 2015 Taylor & Francis
All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers.
Notice:
Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.
Library of Congress Cataloging-in-Publication Data
Butler, Mary Odell.
Evaluation : a cultural systems approach / Mary Odell Butler.
pages cm
Includes bibliographical references and index.
ISBN 978-1-62958-082-1 (hardback : alk. paper)
ISBN 978-1-62958-083-8 (paperback : alk. paper)
ISBN 978-1-62958-084-5 (institutional ebook)
ISBN 978-1-62958-085-2 (consumer ebook)
1. Social sciencesResearchEvaluation. 2. EvaluationMethodology. 3. AnthropologyMethodology. 4. Educational evaluation. I. Title.
H62.B88 2015
300.72dc23
2015012880
ISBN 978-1-62958-082-1 hardback
ISBN 978-1-62958-083-8 paperback
CONTENTS
Like many of us who reach this point in a book, I am overwhelmed by the number of friends who supported me at every step of the way in writing this book and, more importantly, helping me learn the things I needed to know to do it. It forms a mighty crowd, and if I leave your name out here, please accept my apologies and my gratitude.
I would like to extend my thanks to James Hersey who opened the door to evaluation and invited me in. To Susan Squires and Lenora Bohren, who helped me see evaluation within the framework of anthropology. To Eve Pinsker, who, as head of the Evaluation Anthropology Working Group within the National Association for the Practice of Anthropology, kept me focused on the fit between anthropology and evaluation and provided lots of technical support. And to my colleagues and clients over the years, who provided the space for me to learn evaluation and create new ways of doing it.
I am grateful to my editors at Left Coast Press, Inc.: Jack Meinhardt and, before him, Jennifer Collier, both of whom assured me that this was a feasible undertaking. And to Mitch Allen, the Big Man at Left Coast, who first convinced me this book was a good idea.
I acknowledge my debt to my classes in evaluation anthropology at the University of Maryland and the University of North Texas. Much of what is in this book they either taught me or helped me to teach myself.
Finally, thanks to all of my relatives and friends who listened over and over to my trials and perils in putting all of this together. Your patience knows no end.
T his book is about how to link evaluation and anthropology into a dynamic and flexible approach that brings out the vitally human nature of the ways people organize themselves to accomplish their goals. I present an approach to evaluation that merges anthropology and evaluation to make something new, an approach to valuing from the perspective of both the insiders who generate the things being evaluated and the outsiders who will use the evaluation. Evaluation anthropology, alone or in combination with other approaches, can generate explanations of human activities in ways that are both rigorous in their ability to explain and adaptable to changing circumstances. As we will see in the chapters that follow, evaluation is a large field and one requiring many different kinds of abilities.
Evaluation teams go far beyond anthropology. But anthropology has something important to contribute. By using anthropology, evaluators can tease out the multiple perceptions of the thing being evaluatedthe evaluandfrom the various kinds of people who are influenced by it, untangle the complex dynamics of communities in which programs are embedded, qualify the contextual influences coming from outside of programs yet affecting them often profoundly, and integrate large amounts of information to build descriptions of both individual implementations and widespread programs.
So what are we talking about here? What is it that can help us map diversity on diversity, relationship on relationship, in such a way that we get a three-dimensional view of what kinds of processes generate modern life? Capturing and translating the inside of human communities to the outsiders who observe them is the time-honored task of the ethnographer (Harris 1968; Stake 2006). Ethnography too must be re-interpreted and disciplined to meet the scientific demands of evaluation. This volume will explore how the two disciplinesanthropology and evaluationcan move together to help us see how people organize to get the jobs of living done in the complex systems in which all of us must function.
I will argue here that anthropology, as refined and used in this century, enables us to layer reality upon reality in human activity in such a way that the evaluator can assemble what can be known and map program activities onto it. In this way the value of a program in the natural community can be inferred. Evaluation adds the perspectives, the methods, and the science needed to do this in a way that is credible to those who rely on evaluation to make decisions.
WHY EVALUATION?
I chose to write about evaluation because evaluation is what I have done with myself for the past twenty-five years. Evaluation is a scientific endeavor conducted for the purpose of describing the worth, value, and/or effectiveness of some activity directed to serving a human need or solving a human problem. Evaluations may be focused on programs, projects, products, media campaigns, curriculaalmost anything that is delivered to communities of humans. I will focus on program evaluation in this volume because it is what I know best. However, the same approach can and has been used to evaluate projects, policies, and products.
One of the most important things about evaluation, however it is done, is the need for clear thinking and methodological rigor to protect the credibility of results. Decisions about policies, careers, and funding often rest on evaluation results. Evaluations can protect the public good while alerting authorities to things that arent working as they should. The scientific demands of evaluation require that anthropologists incorporate into their methods the safeguards needed to protect their results validity, reliability, and credibility.
I did not find evaluation; evaluation came to me because some people knew I could do ethnography and believed that ethnography would be helpful to their efforts to understand programs. In the early years I dont think my clients cared whether I was an anthropologistalthough they all said it must be interesting! Instead, they wanted program evaluation and knew that they couldnt understand how a program worked unless they could ask the people involved with it. They were pretty sure I could do that.