Hadley Wickham - R for Data Science: Import, Tidy, Transform, Visualize, and Model Data

• 2016-12-06: First Release

Data science is a huge field, and theres no way you can master it byreading a single book. The goal of this book is to give you a solidfoundation in the most important tools. Our model of the tools needed ina typical data science project looks something like this:

First you must import your data into R. This typically means that youtake data stored in a file, database, or web API, and load it into adata frame in R. If you cant get your data into R, you cant do datascience on it!

Once youve imported your data, it is a good idea to tidy it. Tidyingyour data means storing it in a consistent form that matches thesemantics of the dataset with the way it is stored. In brief, when yourdata is tidy, each column is a variable, and each row is an observation.Tidy data is important because the consistent structure lets you focusyour struggle on questions about the data, not fighting to get the datainto the right form for different functions.

Once you have tidy data, a common first step is to transform it.Transformation includes narrowing in on observations of interest (likeall people in one city, or all data from the last year), creating newvariables that are functions of existing variables (like computingvelocity from speed and time), and calculating a set of summarystatistics (like counts or means). Together, tidying and transformingare called wrangling, because getting your data in a form thatsnatural to work with often feels like a fight!

Once you have tidy data with the variables you need, there are two mainengines of knowledge generation: visualization and modeling. These havecomplementary strengths and weaknesses so any real analysis will iteratebetween them many times.

Visualization is a fundamentally human activity. A good visualizationwill show you things that you did not expect, or raise new questionsabout the data. A good visualization might also hint that youre askingthe wrong question, or you need to collect different data.Visualizations can surprise you, but dont scale particularly wellbecause they require a human to interpret them.

Models are complementary tools to visualization. Once you have madeyour questions sufficiently precise, you can use a model to answer them.Models are a fundamentally mathematical or computational tool, so theygenerally scale well. Even when they dont, its usually cheaper to buymore computers than it is to buy more brains! But every model makesassumptions, and by its very nature a model cannot question its ownassumptions. That means a model cannot fundamentally surprise you.

The last step of data science is communication, an absolutely criticalpart of any data analysis project. It doesnt matter how well yourmodels and visualization have led you to understand the data unless youcan also communicate your results to others.

Surrounding all these tools is programming. Programming is across-cutting tool that you use in every part of the project. You dontneed to be an expert programmer to be a data scientist, but learningmore about programming pays off because becoming a better programmerallows you to automate common tasks, and solve new problems with greaterease.

Youll use these tools in every data science project, but for mostprojects theyre not enough. Theres a rough 80-20 rule at play; you cantackle about 80% of every project using the tools that youll learn inthis book, but youll need other tools to tackle the remaining 20%.Throughout this book well point you to resources where you can learnmore.