Learning Scrapy: Learn the art of effi cient web scraping and crawling with Python - Softcover

Kouzis-Loukas, Dimitrios

Softcover

ISBN 10: 1784399787 ISBN 13: 9781784399788

Publisher: Packt Publishing, 2016

Synopsis

Learn the art of efficient web scraping and crawling with Python

About This Book

• Extract data from any source to perform real time analytics.
• Full of techniques and examples to help you crawl websites and extract data within hours.
• A hands-on guide to web scraping and crawling with real-life problems and solutions

Who This Book Is For

If you are a software developer, data scientist, NLP or machine-learning enthusiast or just need to migrate your company's wiki from a legacy platform, then this book is for you. It is perfect for someone , who needs instant access to large amounts of semi-structured data effortlessly.

What You Will Learn

• Understand HTML pages and write XPath to extract the data you need
• Write Scrapy spiders with simple Python and do web crawls
• Push your data into any database, search engine or analytics system
• Configure your spider to download files, images and use proxies
• Create efficient pipelines that shape data in precisely the form you want
• Use Twisted Asynchronous API to process hundreds of items concurrently
• Make your crawler super-fast by learning how to tune Scrapy's performance
• Perform large scale distributed crawls with scrapyd and scrapinghub

In Detail

Scrapy is a python based, open source framework used mainly for web crawling. It provides a code re-use functionality to build and scale robust crawling projects. Scrapy can also be used to extract data using various APIs and to perform real time analytics on the data. It has a very large community and has become one of the most popular web crawler in the past few years.

This book covers the long awaited Scrapy v 1.0 that empowers you to extract useful data from virtually any source with very little effort. It starts off by explaining the fundamentals of Scrapy framework, followed by a through description of how to extract data from any source, clean it up, shape it as per your requirement using Python and 3rd party APIs. Next you will be familiarised with the process of storing the scrapped data in databases as well as search engines and performing real time analytics on them with Spark Streaming. By the end of this book, you would have learned enough to scrap any data for your application with ease.

About this title

About the Author

Dimitrios Kouzis-Loukas has over fifteen years experience as a topnotch software developer. He uses his acquired knowledge and expertise to teach a wide range of audiences how to write great software, as well. He studied and mastered several disciplines, including mathematics, physics, and microelectronics. His thorough understanding of these subjects helped him raise his standards beyond the scope of "pragmatic solutions." He knows that true solutions should be as certain as the laws of physics, as robust as ECC memories, and as universal as mathematics. Dimitrios now develops distributed, low-latency, highly-availability systems using the latest datacenter technologies. He is language agnostic, yet has a slight preference for Python, C++, and Java. A firm believer in open source software and hardware, he hopes that his contributions will benefit individual communities as well as all of humanity.

Edition details

• Publisher: Packt Publishing
• Publication date: 2016
• Language: English
• ISBN 10: 1784399787
• ISBN 13: 9781784399788
• Binding: Paperback
• Number of pages: 270