Seppe vanden Broucke - Practical Web Scraping for Data Science: Best Practices and Examples with Python

KU Leuven, Leuven, Belgium

Dept of Decision Sci & Info Managem, KU Leuven Dept of Decision Sci & Info Managem, Leuven, Belgium

When surfing the web using a normal web browser, youve probably encountered multiple sites where you considered the possibility of gathering, storing, and analyzing the data presented on the sites pages. Especially for data scientists, whose raw material is data, the web exposes a lot of interesting opportunities :

The web contains lots of interesting data sources that provide a treasure trove for all sorts of interesting things. Sadly, the current unstructured nature of the web does not always make it easy to gather or export this data in an easy manner. Web browsers are very good at showing images, displaying animations, and laying out websites in a way that is visually appealing to humans, but they do not expose a simple way to export their data, at least not in most cases. Instead of viewing the web page by page through your web browsers window, wouldnt it be nice to be able to automatically gather a rich data set? This is exactly where web scraping enters the picture.

If you know your way around the web a bit, youll probably be wondering: Isnt this exactly what Application Programming Interface (APIs) are for? Indeed, many websites nowadays provide such an API that provides a means for the outside world to access their data repository in a structured way meant to be consumed and accessed by computer programs, not humans (although the programs are written by humans, of course). Twitter, Facebook, LinkedIn, and Google, for instance, all provide such APIs in order to search and post tweets, get a list of your friends and their likes, see who youre connected with, and so on. So why, then, would we still need web scraping? The point is that APIs are great means to access data sources, provided the website at hand provides one to begin with and that the API exposes the functionality you want. The general rule of thumb is to look for an API first and use that if you can, before setting off to build a web scraper to gather the data. For instance, you can easily use Twitters API to get a list of recent tweets, instead of reinventing the wheel yourself. Nevertheless, there are still various reasons why web scraping might be preferable over the use of an API:

In all of these cases, the usage of web scraping might come in handy. The fact remains that if you can view some data in your web browser, you will be able to access and retrieve it through a program. If you can access it through a program, the data can be stored, cleaned, and used in any way.

There are many practical applications of having access to and gathering data on the web, many of which fall in the realm of data science. The following list outlines some interesting real-life use cases :