Ray Pawson - The Science of Evaluation: A Realist Manifesto

THE SCIENCE OF EVALUATION

SAGE has been part of the global academic community since 1965, supporting high quality research and learning that transforms society and our understanding of individuals, groups, and cultures. SAGE is the independent, innovative, natural home for authors, editors and societies who share our commitment and passion for the social sciences.

Find out more at: www.sagepublications.com

Connect, Debate, Engage on Methodspace

THE SCIENCE
OF EVALUATION

A REALIST MANIFESTO

RAY PAWSON

SAGE Publications Ltd
1 Olivers Yard
55 City Road
London EC1Y 1SP

SAGE Publications Inc.
2455 Teller Road
Thousand Oaks, California 91320

SAGE Publications India Pvt Ltd
B 1/I 1 Mohan Cooperative Industrial Area
Mathura Road
New Delhi 110 044

SAGE Publications Asia-Pacific Pte Ltd
3 Church Street
#10-04 Samsung Hub
Singapore 049483

Editor: Katie Metzler
Assistant editor: Anna Horvai
Production editor: Ian Antcliff
Copyeditor: Jennifer Hinchliffe
Proofreader: Jill Birch
Marketing manager: Ben Griffin-Sherwood
Cover design: Wendy Scott
Typeset by: C&M Digitals (P) Ltd, Chennai, India
Printed by: MPG Books Group, Bodmin, Cornwall

Ray Pawson 2013

First published 2013

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act, 1988, this publication may be reproduced, stored or transmitted in any form, or by any means, only with the prior permission in writing of the publishers, or in the case of reprographic reproduction, in accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries concerning reproduction outside those terms should be sent to the publishers.

Library of Congress Control Number: 2012942929

British Library Cataloguing in Publication data

A catalogue record for this book is available from the British Library

ISBN 978-1-4462-5242-0
ISBN 978-1-4462-5243-7 (pbk)

Ray Pawson is Professor of Social Research Methodology in the School of Sociology and Social Policy, University of Leeds. His main interest, perforce, is in research methodology and he has written widely on the principles and practice of research, covering methods qualitative and quantitative, pure and applied, contemporaneous and historical. Publications include A Measure for Measures (1989), Realistic Evaluation (1997) and Evidence-Based Policy: A Realist Perspective (2006). He was elected president of the Committee on Methodology of the International Sociological Association (199498). He has served much time in prison (for research purposes), being a former UK director of the International Forum for Education in Penal Systems (199597). He has held the post of visiting professor at the University of Rome, University of Victoria, Canada and the Royal Melbourne Institute of Technology, as well as visiting fellow at the UK Economic and Social Research Councils Centre for Evidence-Based Policy and Practice. He is best known for his writing on evaluation methodology, research synthesis and evidence-based policy, work which has been supported over the years by four ESRC senior fellowships. Research income exceeds 3.5 million. He has acted as researcher and consultant on programme evaluation for the various UK and European government departments and agencies.

God save you, dear reader, from an ide fixe, better a speck, a mote in the eye. Joaquim Maria Machado de Asiss The Posthumous Memoirs of Bras Cubas

Ray Pawsons ide fixe, which goes by the name of realist inquiry, has travelled through thirty years and three previous books one on measurement, one on evaluation research and one on research synthesis. And now here he is in 2013, with The Science of Evaluation, doggedly, stubbornly, unendingly writing manifestos in the name of realist social science. A number of institutions have supported this work and along the way he has been fortunate enough to receive several fellowships and visiting positions. He is world famous in Italy, Tasmania and Vancouver Island. Happily, the ide fixe remains controversial. Other bodies (unnamed for the purposes of this sketch) have not been so enamoured and social inquiry has fragmented into a thousand rival fragments. Undaunted, he is already working on a new book How to Think like a Social Scientist.

I am some kind of realist, some kind of critical, hypothetical, corrigible, scientific realist. But I am against direct realism, naive realism and epistemological complacency.

Donald T. Campbell, Methodology and Epistemology for the Social Sciences, 1988

What enables and yet constrains research? What is both medium and outcome of research? What do researchers reproduce without even knowing it? What is supposed to unite researchers but may divide them? What empowers researchers to speak but is never fully articulated? What is played out in the routine of research but can never be routinised? What is the responsibility of all researchers but for which none has a mandate?

The answer to all of these riddles is METHODOLOGY. Methodology provides what has been variously described as the procedures, the rules, the codes and the laws of scientific research. But methodological procedures are unlike the exacting drills followed, say, by airline pilots in readying a plane for take-off. Methodological rules are unlike, for instance, those in official rulebooks enforced by referees and followed by sports teams for fear of the penalty. Methodological codes are unlike, for example, the highway code with its compendium of regulations and advice designed as behavioural conventions for all road users. Methodological laws are quite unlike the legislative decrees sanctioned by governments, written in statute and sustained by enforcement regimes.

Scientific methodology shares with these other regimes the ambition to clarify and organise. But the crucial difference is the extent to which the procedures, rules, codes and laws are formalised. In all of our other cases, there is a manual to follow or a handbook to be thumbed or a rulebook to be consulted or an authoritative tome to be pondered. To be sure this guiding documentation is often in a state of steady evolution, with even the two hundred and fifty-year-old Marylebone Cricket Club, the driest and dustiest of regulatory bodies, responding with continued clarifications and amendments to the rules of that game. The crucial point in these other cases is that vested authority is marshalled via a codified set of decrees and propositions.

Consider, by contrast, the rules of the scientific game. Each time the researcher dreams up a project, responds to a tender, enters the field, draws conclusions, makes observations and pens a paper, that individual too will work under a set of expectations. Indeed many researchers consider that they operate under the sternest of requirements that of producing scientific truths about the fragment of the world selected for study. But whether they are peering down a microscope or into the human psyche, whether they are smashing atoms or counting heads, they do not begin the day by consulting the manual. They have no rulebooks to which to refer, no regulators to affirm standards and no sanctions to keep them honest.