HTTP protocol and using Python to get data and write to MySQL

First, the HTTP protocol

Second, HTTPS protocol

Third, use Python to get the data

(1) Urlib

(2) Get request

(3) Post request

Four, crawl the Watercress movie actual combat

1. Ideas

(1) Enter Https://movie.douban.com/j/search_tags?type=movie in the browser to get the category label of the movie that is displayed, take "hot" as an example

{"Tags": ["Hot", "latest", "classic", "playable", "Watercress High score", "unpopular good film", "Mandarin", "Europe and America", "Korea", "Japan", "Action", "comedy", "Love", "sci-fi", "Suspense", "horror", "growth")

(2) Enter https://movie.douban.com/into the homepage of the browser, then pull down to "recent popular movies", then click "More", the URL shown in the browser is https://movie.douban.com/explore#! Type=movie&tag= popular &sort=recommend&page_limit=20&page_start=0, which means 20 movies on the first page of a popular movie.

(3) pull down to the bottom, click "Show More", then the URL becomes https://movie.douban.com/explore#!type=movie&tag= popular &sort=recommend& Page_limit=20&page_start=20, it also represents the 20 films on the second page.

(4) Enter the following URL in a browser to get 20 JSON-formatted strings: https://movie.douban.com/j/search_subjects?type=movie&tag= Popular &sort= Recommend&page_limit=20&page_start=0.

(5) For each URL, if there is data in the returned result, increase the page_start by 20 to continue executing the GET request until the data is no longer returned.

2. Code implementation

Import urllib.requestfrom urllib import parseimport json# get all movie label URLs = ' Https://movie.douban.com/j/search_tags? Type=movie ' # need to convert Chinese characters into 16 binary form, otherwise it will be reported encoding error print (Parse.quote (' hot ')) request = Urllib.request.Request (url=url) response = Urllib.request.urlopen (Request, timeout=20) # Gets the JSON form of the string result = Response.read () print (Result) # Parses a string in JSON form to a dictionary result = json.loads (result) print (result) # The Label field that takes the dictionary is stored in the list tags = result[' tags ']print (tags) #    Define a list to store the basic information of a movie movies = []# respectively handles each tagfor tag in tags:print (tag) tag = parse.quote (tag) print (tag) start = 0               # keep asking until the return result is empty while True: # stitching requires URL URL for request = ' https://movie.douban.com/j/search_subjects? ' ' Type=movie&tag= ' + tag + '&sort=recommend&page_limit=20&page_start= ' + str (start) print (URL) request = Urllib.request.Request (url=url) response = Urllib.request.ur Lopen (Request, timeout=20) # Gets the JSON form of the string result = Response.read () print (Result) # string in JSON form        Parse to dictionary result = json.loads (result) print (Result) # stores the Label field from the dictionary to the list result = Result[' subjects '] Print (Result) # loop bounce condition If Len (result) ==0:break # adds each record to the movies list for I TEM in Result:movies.append (item) # Modifies the starting position, equivalent to clicking "Show more" start + = 20print (len (Movies))

HTTP protocol and using Python to get data and write to MySQL