Ian Shaw - Evaluating Public Programmes: Contexts and Issues

Ian Shaw
School of Sociology & Social Policy University of Nottingham

First published 2000 by Ashgate Publishing

Reissued 2018 by Routledge
2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
711 Third Avenue, New York, NY 10017, USA

Routledge is an imprint of the Taylor & Francis Group, an informa business

Copyright Ian Shaw 2000

All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers.

Notice:
Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.

Publisher's Note
The publisher has gone to great lengths to ensure the quality of this reprint but points out that some imperfections in the original copies may be apparent.

Disclaimer
The publisher has made every effort to trace copyright holders and welcomes correspondence from those they have been unable to contact.

A Library of Congress record exists under LC control number: 00134014

ISBN 13: 978-1-138-73965-9 (hbk)

ISBN 13: 978-1-315-18415-9 (ebk)

Many thanks to Laura Ward-Swietlinska and Linda Poxon who undertook the preparation of the CRC for this work. Thanks also to Ashgate for being understanding of the pressures on my time. I must also thank Ray Pawson who, perhaps inadvertently, introduced me to the sociological approach to evaluation when I replaced his teaching for a year at the University of Leeds back in 1992.

When I started out studying public policy as a student I thought that policy formation was based upon sound information and analysis of the impact of policy programmes. I now know that this was somewhat naive and 'the facts' often have little to do with the formulation or the direction of a surprising number of public policies. Those evaluating public policy often find that a significant part of their role is in educating policy makers on the limitation of both their programme and what can be expected from a programme of evaluation research. One could of course argue that public policies have been subject to evaluation for many years. However, such evaluations are often to do with examining costs of a public service frequently in terms of a single criterion: value for money. Such 'evaluations' often focus almost exclusively upon achieving economy savings without exploring how, or even whether the service works. The NHS is perhaps the prime example and many of the case studies in this book refer to that service, though the lessons are wider:

Because of such misuse of the term, Evaluation has become a misunderstood and often abused term in relation to public programmes. The Collins English Dictionary defines evaluation as 'to find or judge the value of but, of course, does not explain how this can be achieved. This leads to a wide range of views on what evaluation is and even whether audit or monitoring can be seen as a form of evaluation. This was the subject of a wide ranging discussion on the American Evaluation Association's E-mail discussion list in January of 1998 - only a small section of which is reproduced here.

Richard Duncan argued that:

Patricia Rogers on the other hand argued that monitoring was more concerned with description than assessment (which is evaluation). Michael Quinn Patton responded that paradigm's have permutations and shifts and the distinction between monitoring and evaluation will often be context based. He went on:

We're inevitably caught up in attempting to make fine semantic distinctions (e.g. monitoring vs evaluation) that are no less arbitrary and variable. A dispute was reported in today's newspaper in which people are arguing with great passion about the differences between cider (soft not hard) and apple juice. Some perceived them as the same, some as different. Some experience them as different some don't. Some want to regulate the supposed differences. You can only call this cider (you can only call this evaluation). It is critically important to get beyond labels. Monitoring or audit in one setting will be evaluative in its use and intent, and in another will not be. The differences will depend, in part of the perceptions of those involved or the experiences of those involved, or both, or neither.

In my view the difference can be illustrated by way of an analogy. If your goal was to travel from Nottingham to London by car, evaluation would tell you that you had actually arrived and how you got there. Monitoring, on the other hand, can be seen to be a little like navigation as it would ensure that you travelled the right roads and would advise changes in direction if you were not.

Certainly it is the case that the term 'evaluation' has become a part of the modern day apparatus of management and policy making and has become a miss used and misunderstood part of management jargon. In such an environment 'evaluation' can often mean that senior management asks the junior management responsible for the day to day running of the project - 'how's it going?'. To which the junior replies according to how much he or she values their job. One of the aims of this work is to show that evaluation is often far from straightforward and that the theory and practice of evaluation are far from 'fool proof. There is also a distinction which should be made between evaluation of public policy and evaluation of public programmes. Programmes tend to be the result of policy and, as we shall see, there are different ways of evaluating each.