Acodemy - Python: Learn Web Scraping with Python In A DAY! : The Ultimate Crash Course to Learning the Basics of Web Scraping with Python In No Time

The information provided in this book is designed to provide helpful information on the subjects discussed. The authors books are only meant to provide the reader with the basics knowledge of a certain topic, without any warranties regarding whether the student will, or will not, be able to incorporate and apply all the information provided. Although the writer will make his best effort share his insights, learning is a difficult task and each person needs a different timeframe to fully incorporate a new topic. This book, nor any of the authors books constitute a promise that the reader will learn a certain topic within a certain timeframe.

Table of Contents:

1. Introduction
1. What Is Web Scraping?
2. Why use Web Scraping?
3. Goal of This Book
4. Structure of This Book
2. Getting Started
1. Chapter Objectives
2. Why Use Python for Scraping?
3. Structuring a Python Project
4. Command Line Scripts
5. Python Modules
6. Managing Python Libraries
7. Chapter Summary
8. Lab Exercises
3. Reaching The Web
1. Chapter Objectives
2. Chapter Toolbox: Requests Library
3. Using the Requests Library
4. Simple Scraping using Regular Expressions
5. Chapter Summary
6. Lab Exercises
4. Parsing a Website
1. Chapter Objectives
2. Chapter Toolbox: BeautifulSoup
3. The Structure of an HTML document (DOM)
4. A (Very) Brief Guide to CSS Selectors
5. Using BeautifulSoup to Parse HTML
6. Writing Your First Real Scraper
7. Chapter Summary
8. Lab Exercise
5. Crawling the Web
1. Chapter Objectives
2. Chapter Toolbox: Scrapy
3. What is Crawling
4. Starting a Scrapy
5. Modeling Your Data
6. Building a Spider
7. Running Your Crawler
8. Chapter Summary
9. Lab Exercises
6. Appendix A: More on Regular Expressions
1. Metacharacters
2. Special Sequences
3. Groups
7. Appendix B: Solutions to Select Lab Exercises

Every serious programming project is centered on data. In fact, every programming project is centered on data; it just happens to be the case that some programming tools and some projects allow us to think in more abstract terms. But, for the vast majority of programming problems, a programmer will be required to do some data wrangling on their way to a solution. The first two questions which will need to be answered in order to make that data wrangling possible are where are you going to get the data that your program needs ? And how are you going to get that data from its source to your program ? If your answer to the first question was a website designed for humans, then this book will provide you with everything that you need to know in order to answer the second question.

Sometimes the answer to the where question will be simple and, sometimes, it may even be given a part of the problem statement. You might be given a data file or a database and then the answer to the how question becomes something as simple as using the data that you have been given. But sometimes you will need to find the data on your own. Fortunately, with the rise of the internet and our increasingly data-driven world, it is often the case that whatever data you need for your project is already available somewhere online. Unfortunately, the people who own that data did not have you in mind when they were deciding the best way to display it. How could they? More often than not, information which is visible on the web has been laid out so that, once it has been processed and displayed in a web browser, it can be easily read and understood by a human user. That is one of the primary reasons that the experience of using the web has improved so much over the past two decades, but these improvements in user experience have come at the cost of increased complexity in the structure and operation of webpages. Web browsers have improved alongside these pages and are very good at converting the instructions that they receive from a server into a coherent display of information. But browsers have the advantage of not needing to understand much, if anything, about the content of the pages that they display. Your program is not so lucky. In order to get at the data that it needs from a webpage, your program will need to not only understand the structure of the page, it will need to figure out enough about its content to determine what information on the page it is supposed to be reading. This is not always an easy task.

What is Web Scraping?

Web scraping is the process of using an automated system to extract useful information from a website which was not necessarily designed to be simple for machines to navigate. Any time that you copypaste text or numbers from a website into another program, you are acting as a sort of manual web scraping program. If you are dealing with a static set of data it might be possible to manually copy information from a site into a more accessible data format but, as you have probably experienced, doing this with anything more than around a hundred pieces of data can be an extremely arduous task and, beyond a certain point, it will take longer to transfer the data manually than it would to configure, or even build, a tool to perform the transfer for you, especially if you have the tools and the skills to do so quickly. In the case of a static set of data, the appropriate web scraping tool might be a standalone program which you point toward the data and run once. It will give you a file with the same data in a more useful form, and then you can move on with your project. Alternatively, you might be working with data which appears on a site which is regularly updated. In this case, it is would be less practical to retrieve the data manually because you would need to do so constantly. Even less practical, the choice of which data you scrape might need to be dictated by user input. If this is your situation, there is really no choice but to design a system which can grab useful data automatically.

If you are in the first situation, scraping static data, be aware that there are commercial products available which perform this type of task quite well. But, even if it is your intention to use only prebuilt tools to do all of your web scraping, an understanding of how those tools work will allow you to use them more effectively and to understand their strengths and limitations. In the latter two cases, which involved scraping dynamic data, your web scraper will likely need to incorporate as a part of the same system which will be using the data. If this is the case, commercial web scraping systems are still an option, but are a much less attractive one. In any event, having the knowledge and toolkit available to build, configure, and apply web scraping systems can be a great asset when working on any program which needs access to data from the web. Until the recent rise in the popularity of APIs, which are interfaces specifically designed to let programs and servers speak to each other more easily, web scraping was the basically the only way for a program to retrieve information from the internet. APIs are amazing tools and they should be used whenever they are available, but many useful resources do not have APIs available and, even when they do, the API that is available might not be capable of providing the specific information that you will need for your project. These days, it is important to be familiar both with how to use APIs and how to scrape data from the web when an API is not available or appropriate.