Python3 get a lot of movie information

Lab this time to collect the movie information, give a very large data set, the dataset contains more than 4,000 movie names, I need to write a crawler to crawl the movie name corresponding movie information.

In fact, in the actual operation, there is no need for reptiles, just a little bit of a simple python base on it.

Front-facing requirements:

Python3 Grammar Basics

HTTP Network Basics

The first step is to determine the provider of the API. IMDB is the largest movie database, and its relative, there is a Omdb website that provides APIs for use. The API of this website is very friendly and easy to use.

http://www.omdbapi.com/

The second step is to determine the format of the URL.

The third step is to understand how the basic requests library is used.

http://cn.python-requests.org/zh_CN/latest/

Why should I use requests and not use Urllib.request?

Because this library of Python is prone to a variety of wonderful problems, I have had enough ...

Fourth step, write the Python code.

What I want to do is read the file row by line, then use the name of the line to get the movie information. Because the source file is large, readlines () cannot fully read all movie names, so we read them line by row.

Import Requestsfor Line in open ("Movies.txt"):    s=line.split ('%20\n ')    urll= ' http://www.omdbapi.com/?t= ' +s[0 ]    result=requests.get (URLL)    if result:        json=result.text        Print (JSON)        p=open (' Result0.json ', ' A ')        P.write (JSON)        p.write (' \ n ')        P.close ()

I pre-formatted the movie name file all over again, replacing all the blanks with "%20", which makes it easy to use the API (otherwise it will be an error). This feature can be done with visual Studio code.

Note that when encoding, choose GBK encoding, otherwise the following error will occur:

1 unicodedecodeerror: ' GBK ' codec can ' t decode byte 0xff in position 0:illegal multibyte sequence

Fifth step, do optimization and exception handling.

Mainly do three things, the first thing, control API speed, to prevent being blocked by the server;

Second thing, get API key (even using multiple keys)

The third thing: exception handling.

Import requests 3 key=["]for line in open (" Movies.txt "):    try:        # ...    Except Timeouterror:        continue    except Unicodeencodeerror:        continue    except Connectionerror:        Continue

The complete code is posted below:

#-*-Coding:utf-8-*-import requestsimport timekey=[' xxxxx ', ' yyyyy ', zzzzz ', ' aaaaa ', ' bbbbb ']i=0for line in open (" Movies.txt "):    try:        i= (i+1)%5        s=line.split ('%20\n ')        urll= ' http://www.omdbapi.com/?t= ' +s[0]+ ' &apikey= ' +key[i]        result=requests.get (URLL)        if result:            json=result.text            print (JSON)            p=open (' Result0.json ', ' a ')            P.write (JSON)            p.write (' \ n ')            p.close ()            time.sleep (1)    Except Timeouterror:        continue    except Unicodeencodeerror:        continue    except Connectionerror:        continue

Let's have a cup of tea and see how your program runs!

