Python crawler (2): submit data to web pages, python Crawler

Python crawler (2): submit data to web pages, python Crawler
Python crawler (2): submit data to webpages

Recall that when we were looking at some websites, whether we met some information in some websites and started to display some information. Then, when we pulled the mouse wheel down, some information is displayed. This is asynchronous loading. In my previous article, all titles crawled by the python crawler Baidu Post Bar title data are loaded on the page. But how should we crawl data that is not loaded at the beginning?

Next, we will introduce some concepts:

• Asynchronous loading: to give a simple example, if a teacher judges a assignment, there are two situations: the first one is that no matter which student writes the assignment first, all the students are waiting to finish the assignment, the teacher collects all assignments and starts to judge assignments. The second case is when a student finishes his homework, and the teacher judges one. This saves some time and does not affect the progress. The second method is asynchronous loading.
• The Get method obtains data from the server.
• Post is to send data to the server
• Get functions are implemented by constructing parameters in the URL.
• Post is to put the data in the header and submit the data

Submit data to webpage

• Core Method: requests. post
• Core steps: Construct a form-submit a form-obtain the returned information

First, let's look at the website we want to crawl:

Now let's open the Chrome review element, how to show

In the source code of the webpage, we can find that the titles of each card are stored in the code in this format.

<div class="card-title">Titomirov Vodka LLC</div>

OK, we have found the rule, so we can build our program according to this rule:

title ＝ re.findall('"card-title">(.*?)</div>',post_html.text,re.S)

The above Code does not understand what it means. You can view the title data of the python crawler Baidu Post Bar.

We have finished searching for the title, and then we will focus on it.

This occurs when we slide down and down to the bottom:

This is the asynchronous loading of data. How can we obtain the asynchronous loading of thin data?

Use our review elements and click "NetWork ,:

Now all the data is empty. At this time, we slide down our mouse wheel and you will see a lot of data,

Click the first name in the NetWork. The following information is displayed:

Now let's analyze:

View

Remote Address:50.18.112.181:443Request URL:https://www.crowdfunder.com/deals&template=false&random_seed=1Request Method:POSTStatus Code:200 OK

Request Method: POST. This indicates that we have submitted data to the webpage.

Submit address: https://www.crowdfunder.com/deals&template=false&random_seed=1

Next we will analyze the price increase data, go down, and find:

form Dataentities_only:truepage:1

The above is the submission information, and the page is our page number according to the English meaning. Based on the above information, we can build our form:

# Note that the numbers following the page must be placed in quotation marks. Post_data = {'entities _ only': 'true', 'page': '1 '}

After submitting this form, we can obtain the returned information and apply a regular expression to the returned information to extract the content we are interested in.

The complete code is as follows:

#-*-Coding: utf8-*-import requestsimport re # url = 'https: // www.crowdfunder.com/browse/deals'url = 'https: // region = {'entities _ only ': 'true', 'page': '1'} # submit and obtain the returned data. post_html = requests. post (url, data = post_data) # analyze the returned data titles = re. findall ('"card-title"> (. *?) </Div> ', post_html.text, re. S) for title in titles: print title

When you change 'page': '1' to 'page': '2', different data will be obtained.

The above is all content. If you have any questions or deficiencies, please feel free to discuss them.

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.