Python crawler (1), Python crawler (

Python crawler (1), Python crawler (

It took four days to write a simple crawler program in python. The entire process is divided into two parts: tool installation and program implementation.

This article does not talk about the specific implementation of the program, but is about some preparations

Part 1 (Tool installation)

Download and install the development tool MySql + MySQLWorkbench + PyCharm (refer)

Part 2 (Program Implementation)

Install the MySQLdb module in python on mac OS x

Before the entire program starts, the Python module required by the crawler is the MySQLdb module.

It also took some time to install this module. The installation process is as follows:

1. Download The mysqldb module at the following URL:

Http://sourceforge.net/projects/mysql-python/

2. in mac OS x, double-click to decompress the package, enter the installation directory on the terminal, and execute python setup. py build

If an error such as sh: mysql_config: command not found occurs, you can solve the problem as follows:

When MySql for Mac is installed by default, the mysql_config file is generated in the/usr/local/mysql/bin directory.

Therefore, the solution is to find the site. cfg file in the MySQL-python installation package, open it, and find the following content:

# The path to mysql_config.

# Only use this if mysql_config is not on your PATH, or you have some weird

# Setup that requires it.

# Mysql_config =/usr/local/bin/mysql_config

Remove the first well number of the last sentence and modify it:

Mysql_config =/usr/local/mysql/bin/mysql_config

　　

Then execute:

$ Python setup. py install

The error "Reason: image not found" may occur. The error "resolving Reason: image not found" is as follows:

The solution is to run the following command on the terminal:

$ Sudo ln-s/usr/local/mysql/lib/libmysqlclient.18.dylib/usr/lib/libmysqlclient.18.dylib

$ Sudo ln-s/usr/local/mysql/lib/mysql

Sudo ln-s/usr/local/mysql/lib/libmysqlclient.18.dylib/usr/lib/libmysqlclient.18.dylib

Note: When installing these things on a mac, you need to know how to use vim when modifying the configuration file.

Installing the MySQLdb module in python on mac OS x is a great deal of effort. At last, this module can be applied in both the terminal and Pycharm environment, and the module can be linked to the operation database.

Specific implementation of the program to be continued Python crawler (2)

Refer to blog:

Http://www.cnblogs.com/ifantastic/archive/2013/04/13/3017677.html

Http://www.codeif.com/post/1073/

Teach a small python Crawler

Link: pan.baidu.com/s/1qWsE43q password: sorn
Let's take a look at this. I hope it will be helpful for you to write a python crawler video tutorial.
 
I need a python web crawler index.

1. What does "Web Crawler Index" mean?

2. I have prepared a tutorial for you on crawlers.
There are too many posts, so I won't post them any more. All of them are here:
How to capture static Web pages using Python, C #, and other languages to simulate website login

There are some, principles, logic, sample code, including C # And Python.

(No post address is provided here. Please search for the title on Google to find the post address)