Installation of the CentOs system and the Python and scrapy framework

1: Install the CentOS system first:

Why choose a CentOS system instead of Ubuntu?

I tried on Ubuntu three times to install Python and scrapy, the results are not successful, in OPS Wang's proposal to use CentOS;

(I can't help but spit, Python installation package depends on package installation is really laborious!)

2:centos installation Process

The overall installation process is relatively smooth, more simple than Ubuntu; I chose a mini iOS after the installation also no desktop environment, very good, meet the requirements.

After the installation of the interface: command-line environment, very good.

3: Use the Mobaxterm SSH tool to link CentOs

Need to know the IP address of the CentOS system:

Then go check the ifconfig-a results. This command is not, can understand because it is installed mini OS Some do not understand, not on their own to install Bai;

CentOS 7 Minimized installation can't find ' ifconfig ' command--fix tips CentOS7 install software with Yum cannot find a valid baseurl for repobase7x86_64

Both of the above have been tested and can be used;

As shown in the following link:

4 Next, let's see if it's okay to install Python.

Before installing Python, take a look at whether the system has its own python

It seems that his own python is 2.7; What I need here is Python 3.6.

Then install the python3.6 yourself and set the system default

The following methods can only be used for CENTOS7 or EL7 cores
Yum-y Update
Yum-y Install https://centos7.iuscommunity.org/ius-release.rpm

Install the latest version python3.6.x
Yum-y Install ncurses-devel gdbm-devel xz-devel tk-devel
Yum-y install gcc OPENSSL-DEVL mysql-devl ZLIB-DEVL
Yum-y Install python36u python36u-libs python36u-pip python36u-devel

Create a soft link
Ln-s/usr/bin/python3.6/usr/bin/python3
Ln-s/USR/BIN/PIP3.6/USR/BIN/PIP3

PIP3 Install--upgrade pip

In addition Python 3.6 can create virtual environments directly using Python3-m venv Venv_name

The following is the first to create a virtual environment

Python3-m venv scrapy_venv

Activating a virtual environment

Source/opt/python_venv/scrapy_venv/bin/activate

Pip Install-r requirement.txt

Following file: Requirement.txt

asn1crypto==0.24.0Attrs==17.4.0Automat==0.6.0Certifi==2018.1. -Cffi==1.11.5Chardet==3.0.4constantly==15.1.0Cryptography==2.1.4Cssselect==1.0.3Hyperlink==18.0.0Idna==2.6Incremental==17.5.0lxml==4.1.1NumPy==1.14.1OpenCV-python==3.4.0.12Parsel==1.4.0Pillow==5.0.0PYASN1==0.4.2PYASN1-modules==0.2.1Pycparser==2.18Pydispatcher==2.0.5Pymysql==0.8.0Pyopenssl==17.5.0pytesseract==0.2.0python-logstash==0.4.6Qiniu==7.2.0Queuelib==1.4.2Redis==2.10.6Requests==2.18.4scrapy==1.5.0scrapy-splash==0.7.2Scrapyd-client==1.1.0Service-identity==17.0.0Six==1.11.0Twisted==17.9.0URLLIB3==1.22W3lib==1.19.0Zope.Interface==4.4.3

This is basically done.

Installation of the CentOs system and the Python and scrapy framework