Python simple Crawler 2. python Crawler

Source: Internet
Author: User

Python simple Crawler 2. python Crawler
Next, the last crawler, we continue to study BeautifulSoup.

Python simple Crawler 1

The last time we crawler we successfully crawled the source code of the web page, we will continue to write how to capture the specific elements we want.

First, review the following basic structure of BeautifulSoup:

#! /Usr/bin/env python #-*-coding: UTF-8-*-from bs4 import BeautifulSoupimport requestsheaders = {'user-agent': 'mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 ',} url = "crawling webpage address" web_data = requests. get (url, headers = headers) soup = BeautifulSoup (web_data.text, "lxml ")

The important thing is to emphasize that this is a basic structure for us to crawl web pages, just like building a foundation.

First, import the required tool, that is, the module method from import

In disguise, the browser headers sends a request to the browser requests. get to get the returned result. In BeautifulSoup, the webpage code is parsed in lxml format.

Then, we start to fetch the desired data sample slices on the webpage (usually placed in the src under the label img ), title (usually named after title or put in h3 or a label), where the content is also put in the same as the title

The CSS naming structure is generally used to distinguish between labels.

1. capture the most intuitive image address first:

Right-click the operation, which is also known as the Anti-key-click Next to review the element data to find the element address you selected:

Whether the image address and label img src = "the connection here is the image address" is displayed"

Then we write the following code to capture the image address Tag:

from bs4 import BeautifulSoupimport requestsheaders = {    'User-Agent':'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36',}url = "http://www.lotour.com/guoneiyou/"web_data = requests.get(url,headers=headers)soup = BeautifulSoup(web_data.text,"lxml")imgs = soup.select("img")for img in imgs:    print img

In fact, you only need to change the url address in the structure code, and then add imgs = soup. select ("img"), which is equivalent to the variable name = value. What type is the value?

Select is a method to obtain the selection. If you have used mysql select * from table, can you view the table content with the same effect ("img"), which tag is obtained, which is similar to the where position selected by mysql?

Since select is a one-time query of the returned list, it seems very difficult to use the for loop to remove each element again so that we can see multiple rows of data as follows:

There are several ways to remove the intermediate src method, such as regular expression matching. However, we have a simple built-in method to listen to how the results are decomposed next time.

No matter how easy the tool is, you must use it more to become proficient. Now we can continue to grasp the title or the content. Of course, we are still step-by-step skilled in operations,

Right-click the title or content-click the review element and the effect is as follows:

Here, we correspond to the title of the webpage and the location of the source code and its tag. Finally, we will find the html body div # container .... div. the auther p a line will certainly doubt what it is.

In fact, here is the hierarchical relationship of web pages and the positioning of content. If you want to know more about the basic structure and tag of HTML, you will understand it.

In this case, we still Add the following code to the basic structure to capture the title of the webpage, reminding that other content on the webpage is the same as this principle. We will soon get started by thinking more and summarizing it.

titles = soup.select(" div.auther > p > a")for title in titles:    print title

The old rule uses select to select the obtained content ("div. auther> p> a ") This is the position of the webpage positioning title. It is used in the middle of the div tag CSS. point connection under the p label under the label

Because the labels have A hierarchical structure, we want to tell the Code where to find something. For example, if the BOSS wants you to go to the data room, the information in the third line of the cabinet will be the same.

I suggest you try to practice more and catch other elements multiple times. You may have your own understanding. The effect is as follows:

The next step is the event that everyone is willing to take advantage of. It is the girl who crawls major shopping websites or the movie name of major movie websites. As long as you practice more, you will be proficient, the last thing to note is that many websites will be crawled out.

Garbled code is not a program problem. The website requires Decoding Based on different encoding structures. We recommend that you do not study decoding. We still read the basics to learn about webpage encoding.

Or right-click the review element (some websites will block this function and view the source code) view charset = "encoding format" on the webpage. If it is standard UTF-8, it is okay. If it is gb2312 or other information that may be captured, garbled characters will appear.

Therefore, if you encounter this internal problem, please make a detour, because if you do not learn more, it is difficult to explain it, and there is a basic Baidu solution, I will also write a special title about the program code in the future (please do your best)

Finally, you can only say thank you for watching the next meeting ..!

 

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.