CQ Press
1255 22nd Street, NW, Suite 400
Washington, DC 20037
Phone: 202-729-1900; toll-free, 1-866-4CQ-PRESS (1-866-427-7737)
Web: www.cqpress.com
Copyright 2007 by CQ Press, a division of Congressional Quarterly Inc.
All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher.
Cover design: Jeffrey Everett/El Jefe Design
The paper used in this publication exceeds the requirements of the American National Standard for Information SciencesPermanence of Paper for Printed Library Materials, ANSI Z39.48-1992.
Printed and bound in the United States of America
10 09 08 07 06 1 2 3 4 5
Library of Congress Cataloging-in-Publication Data
Emison, Gerald A.
Practical program evaluations: getting from ideas to outcomes / Gerald Andrews Emison.
p. cm.
Includes bibliographical references and index.
ISBN-13: 978-0-87289-302-3 (alk. paper)
ISBN-10: 0-87289-302-2
1. Public administrationEvaluation. 2. Administrative agenciesEvaluation. I. Title.
| JF1351.E55 2007 |
| 352.439dc22 | 2006023238 |
PREFACE
P rogram evaluation is an important way to advance the public interest. It opens windows to improving the performance of public organizations. Performance in the public sector has always been a major concern, and the past decade has seen an increasing emphasis on it. Whether termed reinventing government, new public management, or results-based management, this new emphasis on results reflects the fact that achievement requires reflection, and program evaluation is institutionalized reflection. It enables the intellectual underbrush to be cleared away and performance improvements to be identified. This identification, however, is not enough for improvements to be realized; implementation is also necessary. This book concerns the practices that heighten the likelihood that a program evaluation will lead to implemented recommendations and subsequent improvement.
As a career member of the federal senior executive service for more than twenty years, I saw that two components typically made up successful program improvements. The first was rigorous, unbiased, and thorough analysis. The second was a series of practices that enabled decision makers to translate the analysis into change. The first component is the focus of most program evaluation texts and courses, whereas the second usually is left to education by osmosis when an evaluator begins work in the real world. The focus on the former at the expense of the latter is easy to understand. Learning the tools and methodology of program evaluation is not easy, so teaching competency in these skills is crucial. Seemingly, the latter is really just good common sense and can be more easily taught on the fly. But what seems so obvious is often not so for students first entering the workforce.
It is not necessary to leave a critical aspect of successful evaluation to happenstance. This book identifies those practices that savvy evaluators follow so that their evaluations get implemented. It adds another dimension to the preparation, reflection, and practice that compose the essentials of program evaluationa handy way to offer concrete advice and reinforce the practical.
The foundations of this book are my own experiences in the public sector and in the classroom. I initially conducted program evaluations as an analyst for the U.S. Environmental Protection Agency (EPA). As a manager, and later, as the director of the program evaluation division, I saw the effective, the ineffective, and the neglected as this group dealt with highly controversial issues. During this period I had many conversations with colleagues about what composed a truly worthwhile evaluation. Almost every practitioner spelled out successful change as the measure of a combination of rigor and practical action.
My work on evaluations led to my crossing over from evaluator to director of a large EPA regulatory program. As the director of air quality planning and standards, I found myself a customer of program evaluations and policy analyses. In this role I was able to observe, during my interactions with political executives and senior career appointee colleagues, what worked and what did not. This experience validated my belief that a combination of rigor with the practical is essential. When I moved to a regional office to become its senior career executive, my observations were reconfirmed in yet another venue. A good program evaluation needs conceptual rigor and practical application in order to be implemented.
Shortly after I left the regional office, I found myself teaching policy analysis in a university setting. As a practitioner I often had wondered why the practical skills essential for successful evaluations were so randomly distributed among newly graduated evaluators. I soon realized it was because most academic training in program evaluation emphasized conceptual preparation without much stress on practical pathways to success. When I retired from the senior executive service and became a full-time academic, I could not find a satisfactory text that exposed my students to this complementary aspect. So I wrote this book.
For those teaching introductory program evaluation courses, this book supplements the many fine core texts available. It introduces the practices essential to effectiveness in applied settings. It supplements, rather than replaces, the conceptual emphasis that is the staple of traditional program evaluation courses. The intention is to round out graduate students education and preparation. Its most useful place is early in a graduate program evaluation course, when students can employ this guidance on the content of the course throughout a semester. The book also can serve as an accessible reference to remind practicing evaluators in the rush of day-to-day work what is important for effectiveness.
The book is organized to promote accessibility. places the book in the terrain of the overall enterprise of program evaluation. The texts core lies in the next four chapters. Each examines a key attribute of successful practical program evaluations. The 4Csclient, content, control, and communicationare used to bundle the essential practices and to examine a series of related practices in a framework that students can return to easily.
Since this book is practice based, it is impossible to thank adequately everyone who played a part; it is my exposure to many dedicated public officials that enabled me to write it. Nevertheless, there are a number of people who contributed mightily to my ideas. Ron Brand, first as my boss, then as my mentor and friend, contributed extensively to most of the ideas found within. Stan Meiburg, as a staff assistant and then a colleague, has never failed to shed new insight upon public evaluation. John Thillmann, David Ziegele, and Tom Kelly were always able to bring me back to earth and remind me that in the long run if a practice does not improve program performance, it is not worthy of extensive effort. And the staffs of the EPAs Office of Air Quality Planning and Standards in Washington, D.C.; Research Triangle Park, North Carolina; and the Seattle, Washington, regional office consistently demonstrated that long-run improvement of the publics interest was why we were in the game.
