Kouzis-Loukas - Learning Scrapy: learn the art of efficient web scraping and crawling with Python

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at > for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

Let me take a wild guess. One of these two stories is curiously similar to yours:

Your first encounter with Scrapy was while searching the net for something along the lines of "web scraping Python". You had a quick look at it and thought, "This is too complex...I just need something simple." You went on and developed a Python script using requests, struggled a bit with beautiful soup, but finally made something cool. It was kind of slow, so you let it run overnight. You restarted it a few times, ignored some semi-broken links and non-English characters, and in the morning, most of the website was proudly on your hard disk. Sadly, for some unknown reason, you didn't want to see your code again. The next time you had to scrape something, you went directly to scrapy.org and this time the documentation made perfect sense. Scrapy now felt like it was elegantly and effortlessly solving all of the problems that you faced, and it even took care of problems you hadn't thought of yet. You never looked back.

Alternatively, your first encounter with Scrapy was while doing research for a web-scraping project. You needed something robust, fast, and enterprise-grade, so most of the fancy one-click web-scraping tools were out of question. You needed it to be simple but at the same time flexible enough to allow you to customize its behavior for different sources, provide different types of output feeds, and reliably run 24/7 in an automated manner. Companies that provided scraping as a service seemed too expensive and you were more comfortable using open source solutions than feeling locked on vendors. From the very beginning, Scrapy looked like a clear winner.

No matter how you got here, I'm glad to meet you on a book that is entirely devoted to Scrapy. Scrapy is the secret of web-scraping experts throughout the world. They know how to maneuver it to save them hours of work, deliver stellar performance, and keep their hosting bills to an absolute minimum. If you are less experienced and you want to achieve their results, unfortunately, Google will do you a disservice. The majority of Scrapy information on the Web is either simplistic and inefficient or complex. This book is an absolute necessity for everyone who wants accurate, accessible, and well-organized information on how to make the most out of Scrapy. It is my hope that it will help the Scrapy community grow even further and give it the wide adoption that it rightfully deserves.