Newhart Michelle - Understanding research methods: an overview of the essentials

The magnitude (i.e., size) of a difference when it is expressed on a standardized scale is referred to as the effect size . The statistic d is one of the most popular for describing the effect size of the difference between two means. To understand the need to consider the effect size of a difference, consider a practical problem in interpreting two sets of research findings that can be resolved using the statistic d .

Suppose that Experimenter A administered a new treatment for depression (Treatment X) to an experimental group, while the control group received a standard treatment. Furthermore, suppose that Experimenter A used a 20-item true-false depression scale (with possible raw scores from 0 to 20) and on the posttest obtained the results shown in Note that the difference between the two means is 5 raw-score points.

Now suppose that Experimenter B administered Treatment Y to an experimental group while treating the control group with the standard treatment. Furthermore, suppose Experimenter B used a 30-item scale with choices from strongly agree to strongly disagree (with possible scores from 0 to 120) and obtained the results in , which show a difference of 10 raw-score points in favor of the experimental group.

Which treatment is superior? Treatment X, which resulted in a 5-point raw-score difference between the two means, or Treatment Y, which resulted in a 10-point raw-score difference between the two means? The answer is not clear because the two experimenters used different measurement scales (0 to 20 versus 0 to 120).

Statistics Obtained in Experiment A (Treatment X)

Statistics Obtained in Experiment B (Treatment Y)

Differences Expressed in Raw Scores and Values of d

For the results of the two studies to be comparable, they need to be standardized so that both differences can be expressed on the same scale. The most straightforward way to do this is to express both differences in terms of standard-deviation units (instead of raw-score units). In Experiment A, one standard-deviation unit equals 4.00 raw-score points. The formula below shows how d is obtained. In this formula, m e stands for the mean of the experimental group, and m c stands for the mean of the control group. The difference between the means (5.00) is divided by the standard-deviation unit for Experiment A (4.00 points). This yields an answer of 1.25:

The result indicates that the experimental group exceeded the control group by 1.25 standard-deviation units. For all practical purposes, there are only three standard-deviation units on each side of the mean, or the center of the normal distribution graph. Thus, d is expressed in standard-deviation units and has an effective range from 0.00 (no difference between the means) to 3.00. For Experiment A, the experimental group is 1.25 standard deviations from the central value (0.00) on a standardized scale that ranges from 0.00 to 3.00. As you may recall, being between 1 and 2 standard deviations means that the value is not in the middle 68% but is between 68% and 95% of all values.

Using the formula for Experiment B, the difference in means (5.00) is divided by the standard deviation (10.00/14.00), yielding d = 0.71, which is almost three-quarters of the way above 0.00 on the three-point scale. Now we can compare the two experiments on a common (i.e., standardized) scale called d . Clearly, the difference in Experiment A (1.25) is greater than the difference in Experiment B (0.71).

summarizes the differences. Remember that the two raw-score differences are not directly comparable because different measurement scales were used (0 to 20 points versus 0 to 120 points). By examining the standardized values of d , a meaningful comparison of the results of the two experiments can be made.

Within each of the two examples in this topic, the two standard deviations are equal. When they are unequal, a special averaging procedure that results in a pooled standard deviation should be used. In the next topic, the interpretation of d is discussed in more detail. In , an alternative statistic for expressing effect size is described.