Python crawler Learning Record [code, detailed steps], python Crawler

Python crawler Learning Record [code, detailed steps], python Crawler

Introduction:

I learned "Python web crawler practice" in NetEase cloud class yesterday. The video link instructor gave me a clear explanation and followed the practice to master the crawler basics. I strongly recommend this!

In addition, you can view the course records prepared by a student on the Internet. The course records are very detailed and can be prioritized for reference. Portal: Click

This article is a recording of Learning Videos by yourself. Please read ~~~

 

Experiment: Sina news homepage crawler practices

Http://news.sina.com.cn/china/

I. Preparation

• Built-in developer tools in the browser (taking Chrome as an example)

• Python3 requests Library

• Python3 BeautifulSoup4 Library (note that BeautifulSoup4 and BeautifulSoup are different)

• Jupyter notebook

2. analysis before capturing

Taking Chrome as an example, capture the previous analysis steps

3. Start to write the first Web Crawler

Requests Library

• Network Resource Retrieval suite
• Improve the shortcomings of Urllib2 and allow users to obtain Network Resources in the simplest way
• You can use REST to access network resources.

Jupyter

Use jupyter to capture and print the webpage in the browser, and then pressCtrl-FSearch for the corresponding content to confirm that the content we want to crawl is on this page.

Test example:

1 import requests2 res = requests.get('http://www.sina.com.cn/')3 res.encoding = 'utf-8'4 print(res.text)

4. Use BeautifulSoup4 to analyze webpage Elements

Test example:

 1 from bs4 import BeautifulSoup 2 html_sample = ' \ 3 
5. Basic BeautifulSoup operations
Use select to find the elements containing the h1 tag
soup = BeautifulSoup(html_sample)header = soup.select('h1')print(header)print(header[0])print(header[0].text)
Use select to find the tag containing
soup = BeautifulSoup(html_sample, 'lxml')alink = soup.select('a')print(alink)for link in alink:    print(link)    print(link.txt)
Use select to find all the elements whose id is title (# Must be added before the id #)
alink = soup.select('#title')print(alink)
Use select to find all elements whose class is link (you need to add .)
soup = BeautifulSoup(html_sample)for link in soup.select('.link'):    print(link)
Use select to find the href link of all a tags
Alinks = soup. select ('A') for link in alinks: print (link ['href ']) # principle: the attributes of tags are packaged into dictionaries.
6. Observe how to capture Sina news
The key lies in searching for CSS positioning.

 

 
  
Chrome developer tool (after entering the developer tool, click element observation in the upper left corner to see it)
Chromefind element positioning .png
 
  
Firefox Developer Tools
 
  
InfoLite (FQ required)

 
7. Create a Sina news Web Crawler
Capture Time, title, content
import requestsfrom bs4 import BeautifulSoupres = requests.get('http://news.sina.com.cn/china')res.encoding = 'utf-8'soup = BeautifulSoup(res.text, 'lxml')for news in soup.select('.news-item'):    if (len(news.select('h2')) > 0):        h2 = news.select('h2')[0].text        time = news.select('.time')[0].text        a = news.select('a')[0]['href']        print(time, h2, a)
Capture news internal pages
News site: http://news.sina.com.cn/o/2017-12-06/doc-ifypnyqi1126795.shtml

Description of internal information .png

Obtain news subject title, time, and source

It involves time and String Conversion.
From datetime import datetime // String Conversion time --- strptimedt = datetime. strptime (timesource, '% Y % m month % d % H: % m') // convert the time string to strftimedt. strftime (% Y-% m-% d)
Organize news documents and obtain edit names

Steps for organizing news:
1. capture;
2. Get a paragraph;
3. Remove the editer information of the last row;
4. Remove spaces;
5. Replace spaces\nHere, you can replace it with various other forms;
It is simply a sentence.
Number of news comments captured

Explanation:
Comments are transmitted through JS Code. Since Javascript is used, the probability of passing comments through AJAX is very high.XHRBut no total number of comments in Response2;Then you can onlyJSInside, the blanket search shows the total number of comments in the Response.2And finally found.
 
Find links and request methods
Supplemented today. The number of comments increases in real time. Please do not think it strange ^_^
Then you can use the verification code.

 
Explanation:
var data={......}It looks likejsonString, removevar data=To change itjsonString.
As you can see,jdThe string contains the comment information.

Return to the Chrome development tool and view the number of comments.

Obtain the news identifier (News ID)
Method 1: Cutting Method
# Obtain the news number newsurl = 'HTTP: // response = newsurl. split ('/'{}-1}.rstrip('.shtml'). lstrip ('doc-I ') newsid
Method 2: Regular Expression
import rem = re.search('doc-i(.*).shtml', newsurl)newsid = m.group(1)newsid
8. Create a function to retrieve comments
Make a summary and organize the method for obtaining the comment into a function. After a news page Link is lost, you can use this function to retrieve its total number of comments.

 
9. Establish a function for extracting news internal information


10. retrieve each news content from the List link
IfDocThere are no things we want to find below, so there is reason to suspect that this webpage produces materials through non-synchronous means. ThereforeXHRAndJSFind it below.
Sometimes you will find non-synchronous dataXHRBut inJSBelow. This is because these materials will beJSAs Chrome's developer tool thinks this is a JS fileJSBelow.
InJSFind the information we are interested in, and then clickPreviewPreview. If you are sure you are looking for it, you can goHeadersViewRequest URLAndRequest Method.
AverageJSThe first one is probably what we are looking for. Pay special attention to the first one.

1. Select the Network tag
2. Click JS
3,Find the page Link page = 2
Process paging links

Pay attention to the head and tail, and turn them into standard ones.jsonFormat.

11. Create an analytic List link function
Sort the preceding steps and encapsulate them into a function.
def parseListLinks(url):    newsdetails = []    res = requests.get(url)    jd = json.loads(res.text.lstrip('newsloadercallback()').rstrip(');'))    for ent in jd['result']['data']:        newsdetails.append(getNewsDetail(ent['url']))    return newsdetails
12. Use the for loop to generate multi-page links


13. Capture each page of news in batches


14. Use pandas to organize data
Python for Data Analysis

 

 
  
Originated from R
 
  
Table-Like format
 
  
Provides a high-performance and easy-to-use Data Frame format for users to quickly operate and analyze Data.

 

15. Save data to the database

 
 

The first web crawler was finally completed. Looking at the final result, I have a sense of accomplishment! Pai_^
If you are interested, try it. Welcome to the discussion ~~~
If the article is useful, please like it. Thank you for your support!
Special gift: GitHub code Portal
Thank you for your patience. If you can help me a little bit, please light up my GitHub star. Thank you ~~~