First published 2000 by Westview Press
Published 2018 by Routledge
711 Third Avenue, New York, NY 10017, USA
2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN
Routledge is an imprint of the Taylor & Francis Group, an informa business
Copyright 2000 Taylor & Francis
All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers.
Notice:
Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe.
A Catalog-in-Publication record for this title is available from the Library of Congress.
ISBN 13: 978-0-8133-9782-5 (pbk)
Ken Meier dedicates this book to Laura Langbein.
Jeff Gill dedicates this book to James A. Thurber.
This book is intended to introduce a new methodological approach, substantively weighted analytical techniques (SWAT), to analyzing data. This approach is focused on investigating subgroups in a given sample with the idea that what makes them different can be important. The differences are often of key substantive importance, such as which programs or agencies are performing at an unexpectedly high level given their available resources. This investigation of "what works" is typically interesting to both researchers and practitioners in public administration. Consequently we have written this book in such a way as to appeal to both audiences. This orientation relies heavily on real and practical examples as a way of illustrating SWAT techniques.
The SWAT philosophy is new to data analysis and will be uncomfortable to some readers. Underpinning this way of thinking is the idea of a "hypothetical population" of interest rather than the actual, existing population from which the data are drawn. We posit the idea that investigating typical cases is not only uninteresting in some settings, but also provides little prescriptive advice.
Many of the methods used in the analysis of policy or programs are inappropriate, and some are actually misleading. The methods used in public administration and public policy should be determined by the needs of the analyst and the desire to make policy recommendations, not by tradition in writing econometric textbooks. We think a great deal of progress can be made by consciously thinking about what we want quantitative methods to tell us and then seeking out or inventing methods that do exactly that. One of the major problems with contemporary policy analysis is the overemphasis on average cases. Although the average case contains a great deal of useful policy information, it is often not the case or set of cases that concern us the most. More often, we care about exceptional casesagencies that perform well, programs that are likely to collapse, or programs that can meet more than one goal simultaneously, for example. SWAT was invented to look at such cases and to tease out how they differ from the average case. We think there is a great deal of policy-relevant information in such cases. Whenever one of us demonstrates SWAT methods to an individual, we get one of two predictable reactions. First, many people respond positively, noting that, "Yes, SWAT is telling me something I would like to know." A second group of people take refuge in the language of statistics and make statements such as "Those have to be biased estimators" or "I'll bet heteroscedasticity is a major problem." The second response troubles us because it immediately assumes that our goal is estimation and inference in the classical manner of statistics (despite our frequent and emphasized statements that SWAT does neither) and because it represents what we see as a rigid and uncreative approach to methods.
Complex public problems require creative and flexible analysis. The rigid application of techniques that make heroic assumptions presents one picture of public programs, but it is only one picture, and perhaps not even an interesting picture. SWAT is not only a technique but also a philosophy of analysis. It seeks to guide analysts in looking at their data in different ways to see what those looks reveal that is new and interesting. This book presents SWAT in the context of exploratory data analysis. We view it as a qualitative tool that uses quantitative data. The exploratory data analysis approach is only one of four ways that SWAT can be viewed. It can also be viewed as an estimation technique for hypothetical populations, as a merger of sensitivity analysis with traditional techniques, or as a cross-sectional version of time-varying-parameter techniques. A brief discussion of these other ways to motivate SWAT is merited to illustrate the logic of the technique.
Classical inference asks: given a population P, what is the probability that sample S was drawn from it? SWAT reweights outlier cases (or any cases of special interest) so that SWAT results cannot say anything about whether or not S could be drawn from P. The coefficients in SWAT quite clearly cannot be inferential estimates of any population parameters in P. However, one might envision a different population, say, P*, that contains ten times the number of high-performing cases and only one-tenth the number of non-high-performing cases. SWAT, by reweighting cases accordingly, might well provide parameter estimates for P*, but P* is a hypothetical population, not one that currently exists. That hypothetical population has an assumed composition based on the SWAT weights assigned and the results will strongly reflect those assumptions.
The second alternative way to think about SWAT is simply that it is a constrained form of sensitivity analysis. Statistics are estimates with an associated variance. Any specific point estimate has an associated standard error. The sensitivity of the point estimate to changes in the composition of the data tells the researcher a great deal about the robustness of a relationship. Influence and leverage diagnostics are constructed via this logic. SWAT coefficients can be viewed in the same light. By altering the composition of the sample, especially with the iterative weighting process, one can judge how stable or unstable coefficients are. Although stability is usually highly valued when sensitivity analysis is done, in SWAT we seek the instability because that is also informative.
A third possible way to think about SWAT is the flexible generalized least squares approach to time-varying parameters. That approach suggests that not only do relationships between variables vary over time, but also that the relationship might actually be unique for each time period. SWAT in essence takes this idea and applies it to cross-sectional patterns. Relationships in the SWAT context are expected to vary across different types of organizations. The cluster of the different relationships and how they diverge from the average relationship is useful information to the analyst. Our emphasis on flexible approaches to analysis is illustrated throughout the book as we change weighting schemes, look at data in different ways, change the types of values we wish to emphasize, and compare various ways of accomplishing similar methodological tasks. Flexibility in our mind avoids bad habits. Analysis should be constantly adapted to the substantive needs of the problem. The reader will quickly note that we avoid significance testing in this text. In part, this reflects our interest in flexible approaches to analysis; traditional hypothesis testing is generally done by rote with little thought about what is actually being tested. But more important, we believe that hypothesis testing as it is done in the social sciences is both misguiding and misleading.
