Peter Nardi - Critical Thinking: Tools for Evaluating Research

Good News! Drinking Wine Before Bedtime Can Help You Lose Weight. For several years, this headline and similar postings have appeared on Twitter, Facebook, and various blogs, excitedly spreading the results of a study supposedly supporting this claim (LaCapria, 2016). Despite secretly hoping that this might be true and blindly accepting it as fact, skeptical readers should instead dig deeper than the sensational headlines and critically ask important questions about the research design. Many media outlets touting the alleged findings cited only a tabloid newspaper article written in London about four women who told anecdotes about their wine consumption and weight loss. Yet, others also reported in a few sentences the results of an academic study that might have loosely been related to the claims. What questions would you need to ask about the study behind the headlines? What information do you want to know in order to decide how accurate the tweets, Facebook postings, blogs, and anecdotal stories are about the research?

Reading the original study you would see that all the women in the sample (men were not part of the research) actually gained weight over a period of time, but compared with nondrinkers, initially normal-weight women who consumed a light to moderate amount of alcohol gained less weight and had a lower risk of becoming overweight and/or obese during 12.9 years of follow-up (Wang et al., 2010: 453). In what ways does this conclusion differ from the headlines version of the research?

Notice that the sensational headline does not indicate that this study (1) was at least five years old; (2) consisted of a women-only sample; (3) looked at self-reported alcohol consumption of all types, not just wine, and at any time, not just at bedtime; and (4) most important, did not in any way demonstrate that drinking wine caused weight loss. So much for Wine as a Bedtime Snack Helps with Weight Loss, as a wine blog exclaimed (VinePair, 2015)!

Media reports of legitimate and scientific research often tell us what foods, diets, vitamins, and physical exercises are best for preventing illnesses and early death. To take one example, studies about the impact on our health of gluten in foods found evidence of sensitivity in people without celiac disease (an intestinal condition that is strongly controlled by gluten-free diets). However, a few years later, the same investigators in another scientific study concluded there was no specific response to a gluten diet among nonceliac gluten-sensitive people (Biesiekierski, Muir, and Gibson, 2013). Media reports of research like these studies often leave out important information about whom researchers sampled, how many participated in a study, and what valid and reliable research methods were used. How do we begin to get a handle on all of these sometimes-contradictory studies and determine the quality of their research methodologies?

Consider another kind of communication requiring critical thinking. Every year around the time for income-tax preparation, fake U.S. Internal Revenue Service (IRS) agents call people demanding back payments. It is estimated that thousands of people have fallen for the ruse and contributed over $20 million to these con artists. The IRS even warns that caller IDs are spoofed to look like theyre coming from a government-agency telephone. Furthermore, unlike the IRS, scammers talk over the phone about you owing more money, ask for immediate payment especially with prepaid debit cards or wire transfer, and issue threats to vulnerable and ill-informed citizens about losing a job, house, or car. What kinds of skills help you to decide whether such a communication is a scam or legitimate?

Each day we are also confronted with distorted and inaccurate information from politicians, media advertisers, pollsters, cable news experts, and friends passing along unfounded rumors. Websites during the 2016 U.S. presidential election sent out fake news reports, including that Pope Francis supported Donald Trump and that Hillary Clinton sold weapons to ISIS (Wingfield, Isaac, and Benner, 2016). With 44 percent of the general U.S. population getting their news from Facebook (Pew Research Center, 2016b), social media and search engines like Google have been pressured into controlling fake reports that appear in news feeds and searches.

An analysis of online news items in 2016 found that the 20 top-performing false election stories from hoax sites and hyperpartisan blogs generated 8,711,000 shares, reactions, and comments on Facebook. Within the same time period, the 20 best-performing election stories from 19 major news websites generated a total of 7,367,000 shares, reactions, and comments on Facebook (Silverman, 2016). Websites like PolitiFact and Snopes were created to monitor the truthfulness of many public figures proclamations, Internet rumors, and fake news scams. What impact do you think social media play in making false stories go viral and impacting public policy like elections?

Less benign are the urban legends that get shared on Facebook, retweeted through Twitter, and forwarded in emails. Although some of these posts are part of the social media game, and can be fun, many times scams following a disastrous flood or tornado, for example, take a psychological and financial toll on people and their families. What steps do you need to take in order to evaluate the accuracy of rumors and media statements from public figures? What tools help you become critical thinkers?