tutorial on installing Python's scrapy Framework on Linux systems

This is an open source tool for extracting web site data. The Scrapy framework, developed with Python, makes crawling fast, simple, and extensible. We have created a virtual machine (VM) and installed Ubuntu 14.04 LTS on it in virtual box.
Installing Scrapy
Scrapy relies on Python, the Development library, and PIP. The latest version of Python has been preinstalled on Ubuntu. So we just need to install the PIP and Python development libraries before installing scrapy.

Pip is a replacement for the Python package indexer easy_install for installing and managing Python packages. The installation of the PIP package is visible in Figure 1.

  sudo apt-get install Python-pip

Figure: 1 PIP installation

We have to install the Python Development Library with the following command. If the package is not installed then it will report an error about the Python.h header file when installing the Scrapy framework.

  sudo apt-get install Python-dev

Figure: 2 Python Development Library

The Scrapy framework can be installed either from the Deb package or from the source. In Figure 3 we installed the Deb package with the PIP (Python Package Manager).

Figure: 3 Scrapy Installation

The successful installation of scrapy in Figure 4 takes some time.

Figure: 4 Successful installation of the Scrapy framework
Extracting data using the Scrapy framework
Basic Tutorials

We will use Scrapy to extract the store name from the fatwallet.com (Sell card shop). First, we use the following command to create a new Scrapy project "store Name", as shown in Figure 5.

$sudo scrapy Startproject Store_name

Figure: 5 Scrapy frame New Project

The above command creates a "store_name" directory in the current path. The files/folders contained in the project's home directory are shown in Figure 6.

$sudo LS–LR Store_name

Figure: 6 contents of the Store_name project

A summary of each file/folder is as follows:

• Scrapy.cfg is a project configuration file
• store_name/another folder under the home directory. This directory contains the Python code for the project.
• store_name/items.py contains items that will be crawled by spiders
• Store_name/pipelines.py is a pipeline file
• store_name/settings.py is the configuration file for the project
• store_name/spiders/, which contains spiders for crawling.

Since we want to fatwallet.com from the name of the store, so we modify the document as follows (LCTT: here does not indicate which document, the translator thinks it should be items.py).

  Import Scrapy  class Storenameitem (scrapy. Item):    name = Scrapy. Field ()  # Remove the name of the card store

After that we will write a new spider under the project's store_name/spiders/folder. A spider is a Python class that contains the following required properties:

• Spider name (name)
• Crawl start URL (start_urls)
• Contains a parsing method that extracts the corresponding regular expression from the response. The Analytic method is very important to the crawler.

We created the "storename.py" crawler under the storename/spiders/directory and added the following code to extract the store name from the fatwallet.com. The output of the crawler is written to the file (StoreName.txt), as shown in Figure 7.

  From Scrapy.selector import selector from Scrapy.spider import basespider from scrapy.http import Request from Scrapy . HTTP Import formrequest Import re Class Storenameitem (basespider): name = "Storename" allowed_domains = ["fatwallet.c Om "] start_urls = [" http://fatwallet.com/cash-back-shopping/"] def parse (self,response): output = open (' StoreName.txt ' , ' W ') resp = Selector (response) tags = resp.xpath ('//tr[@class = "Storelistrow"]|\//tr[@class = "Storelistrow even"]| \//tr[@class = "Storelistrow even last"]|\//tr[@class = "Storelistrow Last"]. Extract () for i in tags:i = i. Encode (' utf-8 ', ' ignore '). Strip () Store_name = "If Re.search" (r "class=\" Storeliststorename\ ">.*?<", I,re. I|re. S): Store_name = Re.search (r "class=\" Storeliststorename\ ">.*?<", I,re. I|re. S). Group () Store_name = Re.search (r ">.*?<", Store_name,re. I|re. S). Group () Store_name = Re.sub (R ' > ', "", Re.sub (R ' < ', "", Store_name,re. I)) Store_name = Re.sub (R ' & ', "&", Re.sub(R ' & ', "&", Store_name,re. I)) #print store_name output.write (store_name+ "" + "\ n")

