John Boulmetis - The ABCs of Evaluation. Timeless Techniques for Program and Project Managers

Copyright 2011 by John Wiley & Sons, Inc. All rights reserved.

Published by Jossey-Bass

A Wiley Imprint

989 Market Street, San Francisco, CA 94103-1741www.josseybass.com

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400, fax 978-646-8600, or on the Web at www.copyright.com. Requests to the publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, 201-748-6011, fax 201-748-6008, or online at www.wiley.com/go/permissions.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Readers should be aware that Internet Web sites offered as citations and/or sources for further information may have changed or disappeared between the time this was written and when it is read.

Jossey-Bass books and products are available through most bookstores. To contact Jossey-Bass directly call our Customer Care Department within the U.S. at 800-956-7739, outside the U.S. at 317-572-3986, or fax 317-572-4002.

Wiley also publishes its books in a variety of electronic formats and by print-on-demand. Not all content that is available in standard print versions of this book may appear or be packaged in all book formats. If you have purchased a version of this book that did not include media that is referenced by or accompanies a standard print version, you may request this media by visiting http://booksupport.wiley.com. For more information about Wiley products, visit us at www.wiley.com.

Library of Congress Cataloging-in-Publication Data

Boulmetis, John.

The ABCs of evaluation : timeless techniques for program and project managers / John Boulmetis, Phyllis Dutwin.3rd ed.

p. cm.(Research methods for the social sciences ; 46)

Includes bibliographical references and index.

ISBN 978-0-470-87354-0 (pbk.); 978-1-118-10364-7 (ebk.); 978-1-118-10365-4 (ebk.); 978-1-118-10366-1 (ebk.)

1. ManagementEvaluation. I. Dutwin, Phyllis. II. Title.

HD31.B633 2011

658.4032dc23

2011021314

Evaluation is a task that every program and project manager will face at one time or another: you may manage the funding process for an organization, or direct a self-study that requires evaluation, or write a grant proposal that includes an evaluation segment. Occasions vary, but they are inevitable.

Still, endorsing evaluation is a lot like endorsing regular visits to the dentist. People are quick to endorse both activities, but when it comes to doing either one, most people are very uncomfortable. The ABCs of Evaluation will reduce your discomfort by doing two things: First, the book will demystify the process of designing and conducting an evaluation by helping you understand the components of an evaluation design. Second, and more important, the book's aim is to convince you that you play an integral part in any evaluation process, and therefore you are a beneficiary of the results.

Who is the you that this book is designed to assist in the process of evaluation? You might be a manager, supervisor, team facilitator, analyst, or evaluator in the private sector. Then again, you might be in the public or private sector as an administrator, coordinator, facilitator, consultant, or evaluator. Or you might be a faculty member or student in any of a host of disciplines (education, management, human services, psychology, public affairs, labor relations, or health). For any of these situations, the book will provide you with a basic understanding of the steps to follow to design, conduct, and supervise a program evaluation.

This volume has been designed to serve as an introduction to evaluation for program and project managers who have little experience in this task. It presents time-tested evaluation principles and techniques, but it does not purport to reflect the totality of the professional field of evaluation, which encompasses a far greater range of techniques and technologies than are suited to the needs and interests of this book's readership.

As a professional, you know that your programs need to show evidence of success. For example, regardless of how important a communication skills training seminar might be to the overall team-building effort of a company, if you cannot show how it affected the employees teamwork skills or how improved teamwork added to company performance, the program might not be funded again. This realization may have originated with you, or the organization may require evaluations and demand accountability. If you have always assumed that evaluations are just too difficult for you to do, that you don't want to tackle difficult data collection or unfathomable data analysis, think again. The ABCs of Evaluation will dispel these misconceptions and show you what to do.

Both the person who is designing and performing an evaluation and the person who is participating in and receiving the findings of an evaluation need to prepare themselves to understand the basic processes involved. The evaluator, by understanding the thoughts, concerns, and questions of the evaluated, can better anticipate and prepare to address them up front. The evaluated, by anticipating the processes, the probing, and the data requests of an evaluator, can position themselves to make better use of the evaluation process and results.

Too often people feel that evaluation is something that is done to them rather than for them. Indeed, evaluation may be seen as something to get through and get around (Gray, 1998).

The ABCs of Evaluation is an easy-to-read explanation of the concepts and methods of the evaluation process. The chapters are designed to teach readers what the process offers, what it consists of, and how it is designed and implemented. In addition, the book will guide managers step-by-step throughout the process of conducting an evaluation, from the early question, Why evaluate? to the later one, What did the evaluation tell me? Consequently, the book will be useful to those who design and conduct the evaluation, as well as to those who supervise others in these tasks.

You may be experiencing downsizing, reorganization, or the introduction of high-performance work teams where you work. These new circumstances may require you to possess evaluation skills. Although you now find yourself in a position that requires the skills, you may not have them, or you may not have been trained to use them.

At the beginning of each chapter, a scenario sets the stage. Then the chapter presents new evaluation concepts followed by concluding exercises designed to challenge your understanding.