Python crawler notes, python Crawler

Python crawler notes, python Crawler

When I was a beginner in python crawlers, I felt that I could not learn anything. I had a hard time crawling Baidu Post Bar novels.

I feel that it is not very practical, but I still write it down and forget it. It is also a souvenir.

In Sina sae blog address: http://r1c3.sinaapp.com/welcome to visit

#! /Usr/bin/env/python 27 # coding: gbkimport urllib2import redef findurl (I): pattern = re. compile ('/p/\ d {10}') Match = re. search (pattern, I ). group () url = 'HTTP: // tieba.baidu.com '+ Match + '? See_lz = 1 'Return urldef findtitle (I): pattern = re. compile (U' \ xb5 \ xda. + \ xd5 \ xc2. + \ xa1 \ xbf ') title = re. search (pattern, I ). group () return titledef main (): name = raw_input ('Enter the post name: ') fow.open(name1_'.txt', 'W + ') name = urllib2.quote (name) url = 'HTTP: // tieba.baidu.com/f/good? Kw = '+ name +' & cid = 0 & pn = 'for index in xrange (550,-1,-50 ): page = urllib2.urlopen (url + str (index )). read () pattern = re. compile (U' <a href = "/p/\ d {10}" title = "\ xb5 \ xda. + \ xd5 \ xc2. + "t') result = re. findall (pattern, page) for each in reversed (result): # fo. writelines (findtitle (each) + '\ n') article = urllib2.urlopen (findurl (each )). read () pattern = re. compile ('Id = "post_content. *?> (.*?) </Div> ') content = re. search (pattern, article ). group () pattern = re. compile ('. +> ') stripl = re. search (pattern, content ). group () result = content. replace ('<br>', '\ n '). rstrip ('</div> '). lstrip (stripl) fo. writelines (result + '\ n') fo. close () print 'done! 'If _ name __= = '_ main _': main ()

　　

How familiar can python be to write crawler code?

You are familiar with crawlers. I have written it in java before, and I will learn python later. I will be able to write the Python version after reading the api in two days.

If you do not know the crawler principle at all, it will be slow. If you do not have any other programming language experience, it will be even slower.

For beginners of python, what modules are required for learning python crawlers?

Urllib, urllib2, urlparse, BeautifulSoup, mechanic, cookielib, etc.
It is not difficult to master these databases. What is difficult for Web Crawlers is that you need to design your own pressure control algorithms, your resolution algorithms, and graph Traversal Algorithms.