Scrapy Tutorial: How To Make A Web-Crawler Using Scrapy?
Web scraping is an effective way of gathering data from the webpages, it has become an effective tool in data science. With various python libraries present for web scraping like beautifulsoup, a data scientist’s work becomes optimal. Scrapy is a powerful web framework used for extracting, processing and storing data. In this article, we will learn how we can make a web crawler using scrapy, following are the topics discussed in this blog:
- What is Scrapy?
- What is A Web Crawler?
- How to Install Scrapy?
- Starting Your First Scrapy Project
- Making Your First Spider
- Extracting Data
- Storing the Extracted Data
What is Scrapy?
Scrapy is a free and open-source web crawling framework written in python. It was originally designed to perform web scraping, but can also be used for extracting data using APIs. It is maintained by Scrapinghub ltd.
Scrapy is a complete package when it comes to downloading the webpages, processing and storing the data on the databases.
It is like a powerhouse when it comes to web scraping with multiple ways to scrape a website. Scrapy handles bigger tasks with ease, scraping multiple pages or a group of URLs in less than a minute. It uses a twister that works asynchronously to achieve concurrency.
It provides spider contracts that allow us to create generic as well as deep crawlers. Scrapy also provides item pipelines to create functions in a spider that can perform various operations like replacing values in data etc.
What is A Web-Crawler?
A web-crawler is a program that searches for documents on the web automatically. They are primarily programmed for repetitive action for automated browsing.
How does it work?
A web-crawler is quite similar to a librarian. It looks for the information on the web, categorizes the information and then indexes and catalogs the information for the crawled information to be retrieved and stored accordingly.
The operations that will be performed by the crawler are created beforehand, then the crawler performs all those operations automatically which will create an index. These indexes can be accessed by an output software.
Let’s take a look at various applications a web-crawler can be used for:
- Price comparison portals search for specific product details to make a comparison of prices on different platforms using a web-crawler.
- A web-crawler plays a very important role in the field of data mining for the retrieval of information.
- Data analysis tools use web-crawlers to calculate the data for page views, inbound and outbound links as well.
- Crawlers also serve to information hubs to collect data such as news portals.
How To Install Scrapy?
To install scrapy on your system, it is recommended to install it on a dedicated virtualenv. Installation works pretty similarly to any other package in python, if you are using conda environment, use the following command to install scrapy:
conda install -c conda-forge scrapyyou can also use the pip environment to install scrapy,
pip install scrapyThere might be a few compilation dependencies depending on your operating system. Scrapy is written in pure python and may depend on a few python packages like:
- lxml — It is an efficient XML and HTML parser.
- parcel — An HTML/XML extraction library written on top on lxml
- W3lib — It is a multi-purpose helper for dealing with URLs and webpage encodings
- twisted — An asynchronous networking framework
- cryptography — It helps in various network-level security needs
Starting Your First Scrapy Project
To start your first scrapy project, go to the directory or location where you want to save your files and execute the following command
scrapy startproject projectnameAfter you execute this command, you will get the following directories created on that location.
- projectname/
scrapy.cfg: it deploys configuration file
- projectname/
__init__.py: projects’s python module
items.py: project items definition file
middlewares.py: project middlewares file
pipelines.py: project pipelines file
settings.py: project settings file
- spiders/
__init__.py: a directory where later you will put your spiders
Making Your First Spider
Spiders are classes that we define and scrapy uses to gather information from the web. You must subclass scrapy.Spider and define the initial requests to make.
You write the code for your spider in a separate python file and save it in the projectname/spiders directory in your project.
quotes_spider.py
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
def start_request(self):
urls = [ '<a href="http://quotes.toscrape.com/page/1/">http://quotes.toscrape.com/page/1/</a>',
http://quotes.toscrape.com/page/2/,
]
for url in urls:
yield scrapy.Request(url=url , callback= self.parse)
def parse(self, response):
page = response.url.split("/")[-2]
filename = 'quotes-%s.html' % page
with open(filename, 'wb') as f:
f.write(response.body)
self.log('saved file %s' % filename)As you can see, we have defined various functions in our spiders,
- name: It identifies the spider, it has to be unique throughout the project.
- start_requests(): Must return an iterable of requests which the spider will begin to crawl with.
- parse(): It is a method that will be called to handle the response downloaded with each request.
Extracting Data
Until now the spider does not extract any data, it just saved the whole HTML file. A scrapy spider typically generates many dictionaries containing the data extracted from the page. We use the yield keyword in python in the callback to extract the data.
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [ http://quotes.toscrape.com/page/1/',
http://quotes.toscrape.com/page/2/,
]
def parse(self, response):
for quote in response.css('div.quote'):
yield {
'text': quote.css(span.text::text').get(),
'author': quote.css(small.author::text')get(),
'tags': quote.css(div.tags a.tag::text').getall()
}When you run this spider, it will output the extracted data with the log.
Storing the Data
The simplest way to store the extracted data is by using feed exports, use the following command to store your data.
scrapy crawl quotes -o quotes.jsonThis command will generate a quotes.json file containing all the scraped items, serialized in JSON.
This brings us to the end of this article where we have learned how we can make a web-crawler using scrapy in python to scrape a website and extract the data into a JSON file. I hope you are clear with all that has been shared with you in this tutorial.
If you wish to check out more articles on the market’s most trending technologies like Artificial Intelligence, DevOps, Ethical Hacking, then you can refer to Edureka’s official site.
Do look out for other articles in this series which will explain the various other aspects of Python and Data Science.
15. Python vs C++
17. Python SciPy
20. Python Basics
23. Python Decorator
32. Python 3.8
34. Python Tutorial
Originally published at https://www.edureka.co on September 6, 2019.
