MAC OS Installation Scrapy

Individuals find it useful to master simple crawler knowledge, especially if they want to automatically download some resources from certain websites or to count some data. For product managers, if you want to understand the price of competitive products, alternative products, you can write a crawler script from the major e-commerce sites to crawl the relevant information, to avoid the trouble of manual processing.

The Python crawler can start with a simple scrapy framework. For Mac OS systems, packages that are installed by default cannot be deleted because the system itself refers to the library of the python2.x that comes with it. If you use python3.x can be installed using homebrew, but if you are accustomed to using python2.x, then installing directly on the default path will make an error.

Found existing installation: six 1.4.1DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.Uninstalling six-1.4.1:Exception:Traceback (most recent call last):

The reason is simple: Scrapy relies on the six library, but the system's six library is older, installation scrapy need to uninstall after installing a new. But Mac OS itself relies on six, causing it to fail to be deleted, so there is no way to install scrapy.

The workaround is to install using VIRTUALENV.

1. sudo pip install virtualenv
2. virtualenv scrapyenv
3. cd scrapyenv
4. source bin/activate
5. pip install Scrapy

After the installation is complete, install the following package:

Well, you can use to scrapy startproject yourproject start your reptile journey.

MAC OS Installation Scrapy