Coxall - Human manipulation: a handbook

1 Definition : Data dredging involves creating misleading relationships in a dataset. It's the equivalent of looking for an answer (any answer) before having phrased the question. It is a misuse of the techniques of data-mining and statistical analyses, such as regression analysis, with a manipulative intention.

Relationships found by dredging data might appear valid within the test set but they have no statistical signific ance in the wider population. It has become very popular since the advent of very large databases and the use of relational database technology.

We should note that data dredging can sometimes be a valid way of finding a possible hypothesis. But such a hypothesis must then be tested with data not in the original dredged dataset. It is misused when a hypothesis is stated as a fact without further validation and is only tested using data that actually originated the hypothesis in the first place .

Data dredging occurs when researchers browse data looking for relationships rather than forming a hypothesis before looking at the data. Another example is when subsets of data are deliberately chosen to create the illusion of significant patterns in deliberately narrowed down data sets.

In data dredging, large compilations of data are examined to find a correlation, without any pre-defined choice of a hypothesis to be tested. Since the required confidence interval to establish a relationship between two parameters is usually chosen to be 95% (meaning that there is a 95% chance that the relationship observed is not due to random chance), there is a thus a 5% chance of finding a correlation between any two sets of completely random variables.

Because data dredging exercises typically examine large datasets with many variables, it is almost certain that apparently statistically significant results will be found somewhere in the data, even though they are entirely spurious and coincidental.

This technique can be used in any field, but it is most often used in medical and other scientific research and the financial environment where interested parties fish the data for apparently interesting correlations and relationships.

For example, suppose that observers note that a particular town appears to have a cluster of cancers in their area , but the observers lack a firm hypothesis as to why this is . The researchers have access to a large amount of demographic data about the town and area, containing measurements for the area of hundreds of different mostly uncorrelated variables. Even if all these variables are independent of the cancer incidence rate, it is highly likely that at least one variable correlates significantly with the cancer rate. Whilst this may suggest a hypothesis, further testing using the same variables but with data from different locations is needed to confirm the hypothesis.

1.1 Traditional scientific methodology: For the lay reader it is important to understand the methodology used in conventional scientific research and how this provides safeguards against manipulative interference.

Conventional scientific method calls for a researcher to formulate a hypothesis, collect relevant data, use some method of statistical analysis to establish some form of correlation and then carry out a statistical significance test to see whether the results could be due to the effects of chance (the so-called null hypothesis). The results are then compared to the hypothesis to prove or disprove its truth.

A vital issue in proper statistical analysis is to test a hypothesis with data that was not used in constructing the hypothesis. This is central to the integrity of a scientific process , because every data set contains some patterns which are due entirely to chance.

If a hypothesis is not tested with a different dataset from the original study population then it is impossible to determine if the patterns found are chance patterns or whether they have some real significance. If we toss a coin 11 times and get heads 5 times and tails 6 times we could conclude a hypothesis that the coin favours tails between 6/11 - 7/11. However, testing this theory on the same data set will only confirm the theory and such confirmation will have no meaning. The statistical significance of the theory needs to be tested on a completely new , fresh dataset, using a new set of coin tossing results.