Jason W. Osborne - Best Practices in Quantitative Methods

Best Practices in
Quantitative Methods

Best Practices in
Quantitative Methods

Edited by Jason W. Osborne

North Carolina State University

Copyright 2008 by Sage Publications, Inc.

All rights reserved. No part of this book may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.

For information:

Printed in the United States of America.

Library of Congress Cataloging-in-Publication Data

Best practices in quantitative methods / edited by Jason Osborne.

p. cm.

Includes bibliographical references and index.

ISBN 978-1-4129-4065-8 (cloth)

1. Social sciencesResearchMethodology. 2. Social sciencesMathematical models. I. Osborne, Jason.

H62.B467 2008

This book is printed on acid-free paper.

07 08 09 10 11 10 9 8 7 6 5 4 3 2 1

CONTENTS

13. Best Practices in Data Transformation:
The Overlooked Effect of Minimum Values

20. Creating Valid Prediction Equations in Multiple Regression: Shrinkage, Double Cross-Validation, and
Confidence Intervals Around Predictions

USING BEST PRACTICES IS A MORAL AND ETHICAL OBLIGATION

J ASON W. O SBORNE

I magine youre a doctor, and your patient is dying. You have three choices: no treatment, standard practice, and an experimental treatment. Standard practice and the new procedure put the patient through a great deal of pain and trauma and, if unsuccessful, dramatically reduce that patients quality of life and financial situation in his or her remaining days. Standard practice is effective 33% of the time. The experimental treatment is expected to be more effective but is still experimental.

What do you do? Do you let the patient quietly ease into death, or do you make heroic efforts to save that patient, gambling on a positive outcome being worth the expense, hassle, and pain and suffering?

Now imagine youre the superintendent of a school district and you have (severely) limited resources and students not achieving the level you desire. You are mulling three potential options for helping your struggling students. One is to give teachers a large raise, theoretically encouraging talented individuals to enter the teaching profession and retaining valuable master teachers. Another is to purchase an expensive new reading program that is getting rave reviews. And your teachers and school board members are telling you that if you just give every student a laptop (or PDA, or iPod), everyone will meet expected yearly growth.

Where do you put your resources? Do you spend your resources on flashy gimmicks that may be the Next Great Thing in education, or do you go with the mundane (but solidly effective) pay raise for teachers?

R ESEARCHERS A RE R OMANTIC F OOLS . R ESEARCH I S M AGICAL .

Some of you may be scratching your heads, cleaning your glasses, wondering if you read that right. Yes, you did. Research is magical, research is romantic.

The word research (especially the word statistics) and the words romantic and magical seem antithetical, yet if you distill the research process to the most basic process, you find people who believe in miracles, who believe in the magic of research. Researchers are not (always) cardigan-wearing, ivory-tower recluses interested in only their esoteric, irrelevant theories, or myopic clipboard-toting socially inept nerds in lab coats.

Research is magical. The creation of knowledge is a uniquely human endeavor, and those of you who have participated in this compelling process know its charms. To perform quantitative analysis is to be present at the creation of new knowledge, and that is a magical thing, a romantic endeavor. We researchers create knowledge. What an amazing, powerful, exciting, maddening, profound process that can be!

This book is dedicated to you, the researcher, who labors (often in obscurity) attempting to change the world, or at least your small corner of it. But why do we need yet another research methods book when there are already approximately a billion research methods books already on the market? There are two answers to that question. The first, and most direct, is because I couldnt find a book that gave me and my students the answers we sought to some very basic questions. Most of the chapters and other articles I have written over the years are in direct response to wondering how to do something, not finding it clearly laid out in any text I had, and writing it down so my next semesters students could use it.

More important, I think we can do better. I know how difficult it is to do high-quality research, to spend months, sometimes years, gathering data, and then what? Do a median split on a continuous variable and perform a t test to see if something worked? Plod through a repeated-measures ANOVA despite violating all assumptions that were never checked, losing half your data due to missing values? No, my fellow Knights, we can do better. We owe it to ourselves to do better. We owe it to the decision makers of the world to do betterand better does not mean maximizing the probability of getting p < .05. It means getting the most accurate picture of what is really happening out there in the world. It means getting the most generalizable, replicable result so that others (like our hypothetical doctor and superintendent introduced above) may be most likely to profit from all your long hours of toil. In my opinion, this is not just good common sense but a moral imperative.

D OES R ESEARCH H AVE E THICAL I MPLICATIONS ? I S Q UANTITATIVE A NALYSIS OF D ATA A M ORAL P ROCESS ?

You bet!

Think about the two scenarios at the beginning of this introduction. What if you, as superintendent, decide to bet your resources (and your students education) on laptops because some research showed that path to be of more benefit to the student. And what if, in reality, those results are the unintentional outcome of poor research practice and do not really reflect the true benefit of giving all students laptops? You just wasted millions of taxpayer dollars, and worse, the students get no benefit from that money, whereas if you had spent the money another way that is effective, they might have received significant benefit. Is that a moral or ethical issue? I think so.