Goertz Gary - A tale of two cultures: qualitative and quantitative research in the social sciences

This appendix presents the results of a survey of methodological practices as carried out in substantive articles in political science and sociology. The survey tracks what scholars are actually doing when producing empirical work that is regarded by the disciplines as excellent, as defined by publication in major journals. While one can debate whether the practices typically used in this work represent best practices, they are the procedures that scholars employ to produce the best substantive work in these disciplines.

The results in the following tables derive from a stratified random sample of articles published in top journals in political science and sociology. The sampling frame is articles published from 2001 to 2010 in six leading journals: American Journal of Sociology, American Political Science Review, American Sociological Review, Comparative Politics, International Organization, and World Politics. The sample is stratified by journal and by time period (20012005 and 20062010). Forty articles were selected per strata, of which eighteen were coded, leading to a total of 216 articles. Review articles and non-empirical theory articles were excluded from the sample. The coding was performed by two Ph.D. candidates at Northwestern University, Khairunnisa Mohamedali and Christoph Nguyen. The spreadsheet with all the data is available upon request.

provides some basic statistics on our sample of articles. As one can see, quantitative methodologies make up the clear majority (72 percent versus 31 percent). Here it is important to keep in mind that we have two journals, American Political Science Review and American Sociological Review, that publish little qualitative work. The other four journals publish a significant proportion of qualitative work.

The data indicate that explicit multimethod work is almost never carried out in these journals. One of the challenges of multimethod research is conducting it within the confines of a journal length article. Yet in some subfields and for some journals, e.g., International Organization, we observe a tradition of publishing quantitative articles that include a couple of short case studies. Moreover, recent books on comparative politics and international relations are now explicitly cast as multimethod studies.

Table A.1
Methodologies Used

Note: Articles can use more than one methodology.

Table A.2
Qualitative Methodologies Used

Note: Articles can use more than one methodology.

presents basic information about various qualitative methodologies used in the articles in our survey. It is not surprising that over 90 percent of qualitative articles are individual case studies or small-N studies. Of these, there are many more small-N studies than individual case studies. The data also show that medium-N studies are not common; only 8 percent of the qualitative articles had 10 or more cases.

A fairly common component of qualitative work is the use of typologies, with about one quarter of all articles including an explicit typology. Since qualitative scholars are not constrained by needing data for lots of cases, they can more easily develop typologies for either descriptive or explanatory purposes. Often these typologies introduce new concepts, which is not uncommon in qualitative research (i.e., 31 percent of all qualitative articles introduce a new concept).

Table A.3
Quantitative Methodologies used

Note: Articles can use more than one methodology.

The percentages for the explicit discussion of counterfactuals and equifinality are lower (15 percent and 13 percent, respectively). As we have discussed, the use of counterfactuals in qualitative research is often implicit and not directly discussed as a method of inference. In turn, this is related to the nonsystematic way in which most qualitative researchers use methodological tools. Similarly, equifinality is often implicit in an analysis, though its limited usage is probably more related to the small N of much qualitative research. A case study almost by definition can only look at one path. Equifinality really comes into play when there are more than one or two cases.

Although process tracing is often used implicitly in qualitative research, it is usually not used explicitly. In our sample, only 22 percent of the articles explicitly used process tracing.

One of the big challenges for the field of qualitative methodology involves encouraging scholars to be more aware of methodological issues and more explicit about the procedures they use to make inferences. We hope that this book helps to promote greater methodological self-consciousness among qualitative researchers.

Turning to quantitative research, Bayesian techniques are on the rise, though we found few in our survey, which may be an artifact of the journals that we examined. Similarly, while the potential outcomes framework, aka Neyman-Rubin-Holland model, is very influential in methodological circles, it did not make an explicit appearance in any of the 216 articles that we surveyed. On the other hand, the use of instrumental variables and experiments could be seen as indicators of the potential outcomes framework, and they add up to 8 percent of all articles.

Cultures are always a mix of longstanding practices and rapidly changing ones. The qualitative and quantitative cultures and their subcultures are no exceptions. Our survey reflects research over the last decade, but if one looked over a longer period of time, many of the scores for the items coded would change significantly, and even more importantly some of the items would come and go. In addition to changing over time, cultures also vary across spacebe it subfields, disciplines, or geography. Multimethod analysis seems increasingly common and prestigious in comparative work and international relations, but much less so for work on American politics. In sociology, QCA commands more attention and respect than in political science. The same is true if one compares its standing in Europe to the United States.

Ultimately, then, our volume provides a snapshot of methodological practices at a certain time and place. The two cultures argument describes especially well the situation in the United States in the early twenty-first century. The differences between the two cultures are substantial enough that they will likely persist well into the future. Yet, we also think a growing group of scholars will cross boundaries and conduct research that innovatively combines ideas from both cultures. Work at this intersection focused on transcending differences may well represent the most exciting social science in the coming years.