Conducting
Research
Surveys
via E-Mail
and the Web
Matthias Schonlau
Ronald D. Fricker, Jr.
Marc N. Elliott
RAND
Approved for public release; distribution unlimited
This study was conducted by RAND as part of its continuing program of self-sponsored research. The research described in this report was supported through the independent research and development provisions of RANDs contracts for the operation of its Department of Defense federally funded research and development centers: Project AIR FORCE (sponsored by the U.S. Air Force), the Arroyo Center (sponsored by the U.S. Army), and the National Defense Research Institute (sponsored by the Office of the Secretary of Defense, the Joint Staff, the unified commands, and the defense agencies). The RAND Statistics Group also provided additional funding for this research.
Library of Congress Cataloging-in-Publication Data
Schonlau, Matthias, 1967
Conducting research surveys via E-mail and the Web / Matthias Schonlau, Ronald D. Fricker, Jr., Marc N. Elliott.
p. cm.
MR-1480.
Includes bibliographical references.
ISBN 0-8330-3110-4
1. Social sciencesResearchMethodology. 2. Social sciencesData processing. 3. Social surveys. 4. Surveys. 5. Electronic mail systems. 6. Telephone surveys. 7. Internet. I. Fricker, Ronald D., 1960II. Elliott, Marc N., 1966-III. Title.
HA29.S366 2002
001.433dc21
2001048925
RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND is a registered trademark. RANDs publications do not necessarily reflect the opinions or policies of its research sponsors.
Cover design by Barbara Angell Caslon
Copyright 2002 RAND
All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from RAND.
Published 2002 by RAND
1700 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138
1200 South Hayes Street, Arlington, VA 22202-5050
201 North Craig Street, Suite 102, Pittsburgh, PA 15213
RAND URL: http://www.rand.org/
To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) 451-7002;
Fax: (310) 451-6915; Email:
PREFACE
Low-cost personal computers and the explosive growth of the Internet over the past decade have introduced new methods of conducting research surveys. It is now possible to conduct an entire survey solely through the World Wide Web or by e-mail. But, just as issues were raised about phone and mail surveys when they were first introduced, many researchers and practitioners are trying to determine the best way to conduct Internet surveys and questioning just how scientifically valid Internet-based surveys are.
This book provides practical information for researchers who are contemplating using the Internet in their survey activities. The authors examine the reported strengths and limitations of using the Internet to conduct research surveys and offer guidelines on survey design and implementation. This book should be of interest to social science and public policy researchers, although it is certainly applicable to any form of survey research, including that conducted within the Department of Defense and throughout the armed forces. It should also prove useful to principal investigators, survey coordinators, and survey programmers.
This study was conducted by RAND as part of its continuing program of self-sponsored research. Support was provided through the independent research and development provisions of RANDs contracts for the operation of its Department of Defense federally funded research and development centers: Project AIR FORCE (sponsored by the U.S. Air Force), the Arroyo Center (sponsored by the U.S. Army), and the National Defense Research Institute (sponsored by the Office of the Secretary of Defense, the Joint Staff, the unified commands, and the defense agencies). The RAND Statistics Group also provided additional funding for this research.
CONTENTS
Chapter One
INTRODUCTION
Chapter Two
BACKGROUND ON THE SURVEY PROCESS
Chapter Three
LITERATURE REVIEW OF WEB AND E-MAIL
SURVEYS
Chapter Four
CHOOSING AMONG THE VARIOUS TYPES OF
INTERNET SURVEYS
Chapter Five
GUIDELINES FOR DESIGNING AND IMPLEMENTING
INTERNET SURVEYS
Chapter Six
INTERNET SURVEY CASE STUDIES
Chapter Seven
CONCLUSIONS
C. HOW EFFECTIVE IS USING A CONVENIENCE SAMPLE
TO SUPPLEMENT A PROBABILITY SAMPLE?
FIGURES
TABLES
SUMMARY
The Internet is profoundly changing the way we communicate with one another. One of the most recent new uses of the World Wide Web is as a survey platform. Internet-based surveys, although still in their infancy, are becoming increasingly popular because they are believed to be faster, better, cheaper, and easier to conduct than surveys that use more-traditional telephone or postal mail methods. Based on the evidence in the literature and real-life case studies, this report examines the extent to which these claims hold true. Specifically, it analyzes the advantages and disadvantages of using the Internetboth e-mail and the Webto conduct research surveys.
This report provides practical information on design and implementation for researchers who are thinking about using the Internet in their survey activities or who are planning to conduct an Internet survey. In addition to our review of the literature on Internet surveys, we base our analysis on a number of other sources, including several institutions with experience in conducting surveys on the Internet, individual researchers who have relied on Web surveys in their studies, and our own personal experiences.
This report addresses three main questions that researchers face with regard to Internet surveys:
When should an Internet-based survey be considered?
What type of Internet survey is appropriate for a particular study?
How should an Internet survey be designed and implemented?
WHEN SHOULD AN INTERNET SURVEY BE CONSIDERED?
Internet surveys may be preferable to mail or telephone surveys in the following cases:
The survey can be conducted with a convenience sample. In a convenience sample, the probability with which a respondent is selected into the sample may not be known. Often, respondents self-select into the survey. For example, advertising on Web pages, in newsgroups, and in traditional media can be used to attract a large number of survey respondents.
The survey is being conducted in an organization that has a list of e-mail addresses for the target population. The benefits in terms of cost and timeliness are greatest when the target population can be contacted initially by e-mail. The U.S. Air Force, for example, maintains a standardized e-mail address system, as well as detailed information about individuals on their e-mail list. Such plusses make the Web a logical choice for a survey mode.
The target population represents a small slice of the total population. Contacting very small slices of the population via random digit dialing (RDD) phone surveys is very inefficient and therefore costly. Mail surveys in this case would be less costly but equally inefficient. In pre-recruited panels (in which potential survey respondents are recruited in advance for multiple surveys), the information obtained from panel participants allows for targeting subpopulations directly. The advantages in using prerecruited panels may make it more cost-efficient to contract with a commercial Web survey company that can target any subpopulation directly from its panel database.