Beth Crandall - Working Minds: A Practitioners Guide to Cognitive Task Analysis

Working Minds

A Practitioners Guide to Cognitive Task Analysis

Beth Crandall, Gary Klein, Robert R. Hoffman

A Bradford Book
The MIT Press
Cambridge, Massachusetts
London, England

2006 Massachusetts Institute of Technology

All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher.

MIT Press books may be purchased at special quantity discounts for business or sales promotional use. For information, please email special_sales@mitpress.mit.edu or write to Special Sales Department, The MIT Press, 55 Hayward Street, Cambridge, MA 02142.

This book was set in Stone Serif and Stone Sans on 3B2 by Asco Typesetters, Hong Kong and was printed and bound in the United States of America.

Library of Congress Cataloging-in-Publication Data

Crandall, Beth.

Working minds : a practitioners guide to cognitive task analysis / Beth Crandall, Gary Klein, Robert R. Hoffman.

p. cm.

A Bradford book.

Includes bibliographical references and index.

ISBN 10: 0-262-03351-8 (alk. paper)ISBN 10: 0-262-53281-6 (pbk. : alk. paper)
ISBN 13: 978-0-262-03351-0 (alk. paper)ISBN 13: 978-0-262-53281-5 (pbk : alk. paper)
1. Cognition. 2. Task analysis. I. Klein, Gary A. II. Hoffman, Robert R. III. Title.

BF311.C732 2006

152.4dc22

2005058030

10 9 8 7 6 5 4 3 2 1

Contents

Preface

Acknowledgments

1 Introduction

2 Overview of Cognitive Task Analysis Methods

I Tools for Exploring Cognition in Context

3 Preparation and Framing

4 Using Concept Maps for Knowledge Elicitation and Representation

5 Incident-Based CTA: Helping Practitioners Tell Stories

6 CTA Methods and Experiment-Like Tasks

7 Analysis and Representation

II Finding Cognition

8 Thinking About Cognition

9 Trends and Themes in the Development of Cognitive Task Analysis: The Rise of Modern Cognitive Psychology

10 Information Technology

III Putting CTA Findings to Use

11 The Role of Cognitive Requirements in System Development

12 Cognitive Training

13 Understanding How Consumers Make Decisions: Using Cognitive Task Analysis for Market Research

14 Cognitive Task Analysis for Measurement and Evaluation

15 Future Directions for Cognitive Task Analysis

Appendix: Guidance for Data Collection

Notes

References

Index

Preface

There are three senses to the title of this book, Working Minds. One is the notion that Cognitive Task Analysis is the study of cognition in real-world contexts and professional practice at work. A second is the sense that practitioners of CTA are themselves engaged in the work of studying the mind. The third is the sense of studying minds when they are engaged in successful accomplishmentwhen things work. Taken together, these three senses capture what this book is all about. Our reason for writing this book is to describe how to do CTA studies. Our motivation stems from our shared interest in Cognitive Task Analysis (CTA) and our shared experience with CTA methods. The chronology of our collaboration illustrates the way our interest in CTA grew over time.

In 1978, when Robert Hoffman was only two years out with his Ph.D. and still focused on the psycholingustics of figurative language, he received a request for reprints of articles from Gary Klein, who was trying to find ways to apply figurative language, in the form of analogical reasoning, to challenges such as reasoning with ill-defined goals and generating predictions in the face of uncertainty. Subsequent events and correspondence led to a realization that both researchers were following similar intellectual paths.

In the early 1980s, Hoffman began to turn from laboratory studies of sentence comprehension to applied studies, in particular the perceptual learning skills of thermographers and aerial photo interpreters. Those studies (Hoffman 1987) led Hoffman to discover how much he could learn about expert reasoning by confronting the practitioner with tough case scenarios.

Klein had also started to use tough cases in investigating the way firefighters made decisions (Klein, Calderwood, and Clinton-Cirocco 1986), using interviews about recent challenging incidents that revealed a great deal about expertise. At this point, however, the dialog between Hoffman and Klein centered on their interests in expertise rather than their enthusiasm for using tough cases to study expertise. In 1989 Hoffman invited Klein to participate in a small workshop on expertise (Hoffman 1992; Klein 1992). Then they collaborated on a chapter on the perceptual aspects of expertise (Klein and Hoffman 1993).

Gary Klein and Beth Crandall have worked together since 1986, when Crandall joined Klein Associates. Crandalls background in development psychology was a natural bridge to the work Klein was doing on expert-novice differences and the role of expertise in decision making. They have collaborated on many projects, co-authored articles and chapters, and co-presented workshops on Cognitive Task Analysis. They share an avid interest in research methodologies and how peoples lived experience, particularly their experiences in tough cases, can inform understanding of cognitive issues.

In 1995, while on a Fulbright in the United Kingdom, Hoffman was asked to conduct a review and analysis of all of the research that had been conducted up to that point by a company called Klein Associates. One thing that came from that review was a detailed report on the Critical Decision Method (CDM) which, it turned out, is also a method that focuses domain practitioners on the analysis of tough cases. Not wanting that report to collect dust, Hoffman approached Beth Crandall, bugging her mercilessly about publishing a review and analysis of the CDM (eventually, Hoffman, Crandall, and Shadbolt 1998).

We have been conducting applied studies (which today we call Cognitive Task Analysis) for about twenty years. A review of our files shows that, collectively, we have done CTA projects in more than a hundred distinct domains and have conducted over a thousand CTA interviews. We have personally interviewed and observed hundreds of domain practitioners in domains including weather forecasting, clinical nursing, fire-fighting, nuclear engineering, mathematics, and military command and control. The roster of participants includes individuals who are indubitably world-class and at the peak of their profession, and we feel fortunate to have had opportunities to work with the best of the best. We have also worked with individuals who are at journeyman and apprentice levels as well because a theory of cognition, and methods for studying it, must embrace phenomena that span the proficiency range.

We have each explored a variety of different CTA methods, so we have firsthand experience with many of the methods used in the field. We have also evaluated, refined, and extended our methods (repeatedly), and created new methods and new combinations of methods. Hoffman has even compared CTA methods empirically, evaluating them for their efficiencies and yield.

In addition, we have trained others to conduct CTA studies. Among the three of us we have conducted about three dozen CTA courses and workshops and trained hundreds of people. Our interest in helping others do CTA work has led us to prepare instructional, CD, and Web-based tools that we could leverage for this book. In preparing this book we have drawn on that experience in offering guidance about how to do CTA.

We want to show others how to do CTA studies because of our firm belief that CTA has a strong track record and is absolutely necessary in many modern applications of cognitive science. Researchers have relied on CTA methodology to fuel a new area in psychologythe field of naturalistic decision making (Flin et al. 1997;Hoffman in press;Klein et al. 1993; Salas and Klein 2001; Zsambok and Klein 1997). We also find an increasing demand for CTA researchers in applied communities that are developing complex systems in which teams of humans use information technologies to conduct cognitive work.