John H. Aldrich - Improving Public Opinion Surveys: Interdisciplinary Innovation and the American National Election Studies

CHAPTER 1

Introduction to the Volume

John H. Aldrich and Kathleen M. McGraw

PUBLIC OPINION SURVEYS provide insights into a very large range of social, economic, and political phenomena. In this book, we look at the survey itself as the means by which the scholarly community seeks to understand those phenomena. A survey can be understood as a collection of ideas at the forefront both of what the scholarly community believes to be important for understanding opinion and behavior and of how that community thinks those ideas may best be measured and evaluated. In this volume, we have the opportunity to report the results of testing a large variety of the best new ideas that the scholarly community thinks should be considered for inclusion in such a survey. These ideas were proposed for addition to the American National Election Study (ANES) and tested in a pilot study conducted in 2006.

Why should the reader be interested in such reports? The answer is that these reports are a major part of why the ANES is considered the gold standard among public opinion surveys. These reports constitute The perhaps the most intellectually stimulating part of the gold standard, since they include the justification for the ideas that constitute the survey and, thus, the basis for the intellectual advances that the survey will offer. In addition, they reflect a return to a deep commitment to intellectual openness and an interdisciplinary search for the best new ideas and their translation into measurable projects. Survey research is in the midst of a particularly exciting time as new technologies are being added to these new ideas to create major ferment in the means and methods of survey research. Tat the ANES remains the gold standard is not inevitable, except by virtue of its commitment and resolve to be so. Let us begin by indicating how and why the ANES achieved its status and how this project seeks to extend that standard.

THE ANES

First used in 1948, the ANES has been in the field in every presidential election and nearly every congressional election since. It is supported by the National Science Foundation as one of its three big social science projects (the other two are the General Social Survey [GSS] and the Panel Study of Income Dynamics [PSID]). Each project is a national resource for the scholarly community, and each covers a large range of topics in the social and behavioral sciences. Each is, in its area, the gold standard for survey-based research.

The ANES has a special place in even this exalted category. Its longevity alone is a remarkable strength, especially when coupled with the serious attention given to maintaining as much continuity in measurement as the science makes possible. Sixty years of measuring public opinion and voting behavior has made possible the compilation of time-series analyses that are now starting to show real insights into, and to change how we view, campaigns and elections. The relatively recent decision by the NSF that the surveys will continue to be conducted face-to-face was based on its decision, reached after careful consultation with survey experts from across and even beyond the nation, that such a survey, expensive though it may be, is in fact the gold standard (a phrase that came out of this evaluation process) for doing a survey as well as it can be done (for transcripts of a 2003 workshop at the NSF that helped underpin these decisions, see http://www.nsf.gov/sbe/ses/polisci/reports/aebtrans21503.pdf).

There is one other reason why the ANES is an especially appropriate place to evaluate new ideas. The study of public opinion and presidential elections itself has several virtues. Voting is the one decision that the entire nation is asked to reach by the same day. All citizens are at least potentially exposed to the same phenomenathe same events of the last four years, the same candidates and parties standing for the presidency, many of the same campaigns, the same ads, the same media coverage, etc. Not only is the presidential election a common decision point, but presidential election campaigns are sufficiently complex and diverse enough to provide a rich tapestry on which to examine attitude formation, evaluation, decision making and choice, and many other topics of interest to the social and behavioral sciences. In this sense, the ANES survey instrument is genuinely the product of, and contributor to, interdisciplinary scholarship. The scholars contributing to this book come from nearly every social and behavioral science discipline and beyond themselves.

These strengths of the ANES survey instrument impose severe constraints as well. The instrument is limited by time, that is, the time that respondents are willing to give to responding to a survey. The survey's longevity and its importance to the scholarly community have made continuity of the sixty-year time series valuable, but have also limited the time available for innovations. And the centrality of national elections means that ideas that have relevance for understanding turnout and vote The choice are given priority over other equally interesting but less immediately relevant ideas.

The high value of space and time on the survey requires high hurdles for access. The ANES became a national resource supported by the NSF in the early 1970s. While the project's principal investigators (PIs) necessarily retain final authority, the Board of Overseers provides the basis for peer review of proposals for inclusion. From the 1970s through the 1990s, the board selected a series of topics and solicited the user community for proposals. Successful proposals were then included in pilot studiesthe ANES website reports on ten such studies conducted from 1979 to 1997 (see http://www.electionstudies.org/overview/rdd_tab1.htm). The board and PIs then selected new instrumentation based in part on the results of such proposals and their pilot-study testing. As part of the process for opening up the 2008 study to innovations, the NSF provided funding for a new mechanism for the input of ideas to test in a pilot study in 2006. The chapters that follow result from that new mechanism and provide the concrete evidence of the ANES as gold standard.

THE GOLD STANDARD IN SURVEY RESEARCH

What might being the gold standard in survey research entail? We (like others) have numerous criteria in mind. First, the basic notion of a survey is that it has high external validity. In particular, the advantage of surveys over other modes of empirical research is that one can make inferences about a (well-defined) universe from the observations were drawn. Tat means that the first great strength of a survey is that it is based on a probability sampling process. It is because of this grounding that one can make statistical inferences back to the universe one wants to study. This presumes, for example, that no matter what the other advantages of many online surveys may be, if they are not drawn from a universe with known probability, no matter how carefully one designs weights, it is extremely difficult for them to have a strong basis for inference to the population of interest.

A second great virtue of the procedures followed by the ANES is that they foster a high response rate. Many of the time-dependent surveys, such as those done by the media to provide a basis for their reporting of elections, have very low response rates (less than 20%). Indeed, declining response rates have become a major concern for all survey research, whether it is done by the government, the academy, the media, or campaigns. The ANES works very hard, innovatively, and indeed expensively to maintain a high response rate. Without a sufficiently high response rate, the advantages of probability-based sampling are lost.

The third component of the gold standard is to balance the openness of the instrument to the appropriate intellectual community with the maintenance of high standards of quality. This pairing often requires a balancing of objectives. The wiki approach, associated with the Internet, has the virtue of great openness, but as a result, it places a great premium on methods to maintain standards, even of truthfulness (something that