Sidney H. Irvine - Item Generation for Test Development

This book is based on a conference Generating Items for Cognitive Tests: Theory and Practice, held at the Chauncey Conference Center on the Educational Testing Service campus in Princeton, New Jersey, November 56, 1998. Funding for the conference and preparation of the book was provided by Educational Testing Services Research Division (Dr. Henry Braun), by the U.S. Air Force Office of Scientific Research (Dr. John Tangney), and by the Air Force Research Laboratorys Human Effectiveness Directorate (Drs. William Strickland, William Alley, and Bruce Gould). The support of these organizations and individuals has brought the project to fruition: and we extend to them the thanks of all who participated.

Educational Testing Services interest in the sharing and dissemination of knowledge was and continues to be due to the need for larger item pools as a result of the demands of computer adaptive testing, as is documented in several chapters and at various points in the discussions. In addition, the Air Force Research Laboratory had an interest in repeated-testing applications. Large item pool and repeated measure issues are the focus of much of the content of this volume. For the Air Force Office of Scientific Research, the current effort was part of a two conference-book project. The companion projectsupplementary to the current projects focus on methodology was devoted primarily to the content of assessments (Ackerman, Kyllonen, & Roberts [Eds.] [1999], Learning and Individual Differences: Process, Trait, and Content Determinants . Washington, DC: American Psychological Association).

The two-day conference was organized into four sessions, one each morning and afternoon, with three or four papers presented during each session, and a discussion of the papers at the end of each session. Many people made the discussions and debates lively and informative. Henry Brauns role was pivotal in introducing the conference and speakers, and leading the discussions. Howard Wainer was center stage for Session 3, adding his characteristic wit to the proceedings. Their skills were invaluable.

Apart from these major debts, we wish to thank all those who participated in the stimulating and lively discussions, including, in addition to the chapter authors and discussants, Chris Elshaw, a psychologist in the Centre for Human Sciences, at the United Kingdoms Ministry of Defence Evaluation and Research Agency (DERA); Stephen Luebke, a Senior Test Specialist at the Law School Admission Council in Newtown, Pennsylvania; and the following individuals from ETS: Mary Morley, a Research Scientist with a specialty in assessment design; Douglas Fiero, Group Leader, Arts and Literature, Assessment Division; and Ned Walthall, a content assessment specialist from the Assessment Division.

We also wish to thank Dianne Rein for her assistance on the manuscripts, Sumeet Dandeker, who transcribed the discussions in record time, and Catherine Hombo, who provided excellent technical reviews of some of the manuscripts. Above all, we wish to thank Kathleen Howell for her assistance in all phases of the conference and book planning and execution, including travel arrangements, manuscript collection and editing, subject and author indexes, and arrangements with the conference center. Like her late colleague, Ann Jungblut, she embodies that pursuit of excellence so closely associated with ETS conference organization and book production.

Finally, we note that Sam Messick, to whom this book is dedicated, played a critical role in the initiation of the conference, in developing its theme, and in selecting its speakers. Sam believed the time was right for the conference intellectually, and with characteristic magnanimity and charm he engineered its conception administratively. Without his vision and participation, the conference could not have taken place; and this book would not have been written. All of us were cognizant of his role and conscious of his influence as we prepared and assembled our talks and our chapters. His guiding hand is apparent throughout.

Abram, M.,

Ackerman, P. L.,

Advisory Group for Aerospace Research and Development (AGARD),

Akin, O.,

Albert, J. H.,

Almond, R. G.,

Anastasi, A.,

Anderson, J. D.,

Anderson, J. R.,

Armstrong, R. D.,

Atherton, R. M.,

Babkoff, H.,

Baddeley, A. D.,

Banerji, N.,

Bartram, D.,

Battig, W. F.,

Baxter, P.,

Bayliss, R.,

Bejar, I. I.,

Bennett, R. E.,

Berger, M. P. F.,

Beringer, J.,

Bevans, H. G.,

Bird, A. S.,

Birenbaum, M.,

Bittner, A. C., Jr.,

Blizzard, R. A.,

Bloom, B. S.,

Bock, R. D.,

Boer, L.,

Boies, S. J.,

Bongers, S. H.,

Bormuth, J. R.,

Bortkewitsch, L. von,

Bradlow, E. T.,

Braun, H. I.,

Bredenkamp, J.,

Brennan, R. L.,

Bridgeman, B.,

Brieman, L.,

Bunderson C V.,

Burke, E.,

Byrne, R. M.J.,

CTB/McGraw-Hill,

Cape, L. T.,

Carpenter, P. A.,

Carretta, T. R.,

Carroll, J. B.,

Carter, R. C,

Cattell, R. B.,

Chaffin, R.,

Chaiken, S. R.,

Chalifour, C.,

Chase, W. G.,

Chi, C. W.T.,

Chomsky, N.,

Christal, R. E.,

Clark, H. H.,

Clark, L. A.,

Collins, A. M.,

Collis, J. M.,

Cooper, L. A.,

Cronbach, L. J.,

Damos, D. L.,

Daneman, M.,

Dann, P. L.,

Davis, J. H.,

Dayton, C. M.,

De Boeck, P.,

Dempster, A. P.,

Diehl, K. A.,

Dresher, A.,

Dumais, S. T.,

Dunlap, W.,

Eifeldt, R,

Educational Testing Service,

Eichelman, W. H.,

Ekstrom, R. B.,

Elithorn, A.,

Elizur, D.,

Elliot, S. M.,

Els haw, C C,

Embretson, S. E.,

Engle, R. W.,

Englund, C. E.,

Enright, M. L.,

Ericsson, K. A.,

Estes, C. A.,

Estes, W. K.,

Etzel, S.,

Evans, J. St. B. T.,

Feynman, R. P.,

Finell, G.,

Fischer, G. H.,

Fitzpatrick, F.,

Fleck, D.,

Fleishman, E. A.,

Flynn, I. N.,

French, J. W,

Friedman, D.,

Friedman, J. H.,

Furneaux, W. D.,

Furnival, G. M.,

Gao, X.,

Gerritz, K.,

Gevins, J. S.,

Gitomer, D. G.,

Gitomer, D. H.,

Glaser, R.,

Gleser, G. C,

Goeters, K-M.,

Gopher, D.,

Greig,J. E.,

Grenzebach, A. P.,

Groen, G. J.,

Guttman, L.,

Habon, M. W.,

Haertel, E. H.,

Hambleton, R. K.,

Hamilton, P.,

Harbeson, M. M.,

Harris, R. I.,

Harsveld, M.,

Hayes, J. R.,

Hegge, F. W.,

Heineke, M.,

Hermans, P. H.,

Herskovits, E.,

Hetter, R. D.,

Hitch, G.,

Hively, W.,

Hockey, G. R.J.,

Hoffman, M.,

Holroyd, S. R.,

Holzman, G. B.,

Hombo, C. M.,

Hone, A.,

Hrmann, H. J.,

Horn, J. L.,

Hornke, L. F.,

Hough, P. V. C,

Houghton, A.,

Hunt, E.,

Huttenlocher, J.,

Hutwelker, R.,

Ippel, M. J.,

Irvine, C. D.,

Irvine, S. H.,

JAA-Joint Aviation Authorities,

Jacobs, N. R.,

Jacobs, P. J.,

Jacobson, R. L.,

Janssen, R.,

Jensen, A. R.,

Jex, H. R.,

Johnson, E. S.,

Johnson-Laird, P. N.,

Jones, A.,

Jones, D. H.,

Jones, S. R.,

Just, M. A.,