Python Learning notes (i)

Because of the work needs, often need to sina a blog to find information, in the Post directory of a page to the naked eye search, see the right title and then click on the link to see the content, know the appropriate copy down. It's annoying. So always have an idea, learn reptiles.

Take the unit issued by the book card to buy the book "Python programming from the beginning to practice", with a bit of programming base, three hours to read the basic section, and then install Python began to collect a variety of Web text tutorials, began to practice.

(a) Regular title extract the Post text

Sina blog post a sentence can be placed in a few tags inside, feel very wonderful (perhaps I am ignorant). Because of the mess, Sir studied the re-regular expression module and started

#导入request方法from urllib Import request# Importing the regular Expression module imports re# open the Web page to get the response text Response=request.urlopen ('/HTTP/ Blog.sina.com.cn/s/blog_xxxxxxx.html ') #读取网页源码 don't know how to express this thing page=response.read () #转码js = Page.decode (' Utf-8 ') # Compilation matching mode pat=re.compile (R ' ((?<=>) [^>]+? (? =</font>) | (?<=>) [^>]+? (?=</span>)) #匹配网页内容match =re.findall (PAT,JS) #若匹配则输出if match:    n=0    for Each_match in match:        n+=1        print (N, ":" +each _match + ' \ n ')

Regular matching pattern is not perfect to match a very small number of unrelated content, plus a variety of web tutorials to understand the various, so that the Sir lost the direction of efforts.

(b) Download the picture according to the URL in Excel

After a lapse of one months, suddenly had the opportunity to collect pictures, and let Sir have a little power. Because the URL has been collected by other crawlers in the Excel workbook, so the Sir learned a third-party package OPENPYXL, and then started again.

From urllib import request                    #导入request函数from OPENPYXL import load_workbook #导入load_workbook函数 # Custom one based on URL and picture file name called parameter , the function of auto download picture def downloadimage (imageurl,imagepath):    response=request.urlopen (IMAGEURL) #访问图片地址, get response    Imagecontents=response.read () #获取图片内容    f=open (ImagePath, ' WB ') #打开文件    f.write (imagecontents) #写入内容    f.close# Close File # Open Excel Workbookwb=load_workbook (' image.xlsx ') #指定工作表ws =wb.active
#获取单元格内容 start from the second line to the end of the blank cell I=2while Ws.cell (row=i,column=1). Value!=none:   #生成文件名 A column file name     path= str (i-1) + ' _ ' + Ws.cell (row=i,column=1). Value + '. jpg '   #获取URL地址 B is listed as the URL address   Url=ws.cell (row=i,column=2). Value   #调用函数下载图片   downloadimage (imageurl=url,imagepath=path) #循环下载图片   print (' Downloaded ' +str (i-1) + ' HD picture >>> ')   I +=1       

Because the function downloadimage inside did not add exception handling, so when the request failed, the script will stop moving. After adding exception handling, Sir has noticed that the download speed is really slow. So the next day to help a acquaintance.

(iii) Multi-threaded Download pictures

The friend sent me a code to learn. After the study, Sir found that the specific things to do, put in the multi-threaded download that class inside, you can use multi-threaded download. The speed flies quickly. 30 seconds gave me more than 2000 pictures, up to 1G, fortunately I terminated the script in time.

At the same time, Sir also learned how to write the exception, that is, in the JSON to find the image address the beginning of the puzzle, it took a while to understand how.

Import Requestsimport Jsonimport Threadingdefault_header = {#具体请求头自己去弄}_session=requests.session () _ Session.headers.update (Default_header) #多线程下载class MyThread (threading. Thread): Def __init__ (self,imgurl,fname): Threading. Thread.__init__ (self) self.imgurl=imgurl self.fname=fname def run (self): print ("Downloading", self.    Imgurl) Download (self.imgurl,self.fname) def download (fileid,type): Img_url= "http://img.hb.aicdn.com/" +fileid Imgresp=requests.get (img_url) byte_img = imgresp.content try:out = open (type, ' WB ') Out.write (byte_im g) Out.flush () out.close () except Exception as E:print (e) if __name__ = = "__main__": R =_sessio N.get (' http://huaban.com/pins/873774526/?xxxxxx ') url=json.loads (r.text) urllist=url[' pin ' [' board '] [' pins '] for I in urllist:key=i[' file ' [' Key '] print (key) #download (key,key+ '. jpg ') myThread (key,key+ '. jpg '). Start ()

　　

(iv) Multi-thread acquisition and transformation

　

From urllib import request from BS4 import BeautifulSoup from OPENPYXL import Workbook import Threadingimport Reimp Ort time# function def get_title (URL): Global n #变量全局化 Global pageindex #变量全局化 #global ws #为什么声明与不声明都一    Sample? Try: #打开网页, save quiz title and link R=request.urlopen (url,timeout=10) #打开网页, set timeout response S=r.read () #读取网页源 Code js=s.decode (' Utf-8 ') #网页转码 soup=beautifulsoup (JS, ' lxml ') #网页解析 #正则查找子节点 for a in soup.find_a LL (' A ', Href=re.compile (R ' Http://blog.sina.com.cn/s/blog ')): N+=1 #计数 Ws.cell (row=n+1,column=1) . Value=n #写入序号 Ws.cell (row=n+1,column=2). value=a.string #写入标题 Ws.cell (row=n+1,colum n=3). Value=a.get (' href ') #写入链接 print ("Download succeed >>>>" +url+ ' \ n ') #正常输出 except EXCEP     tion as E:print ("Download failed >>>>" +url+ ' \ n ') #异常输出 if __name__ = = ' __main__ ': #主线程 Start=time.timE () #开始时间 filename= ' multi-threaded crawl Enlightenment Hui want the question title and link. xlsx ' #Excel文件名 threads=[] #新建线程列表 wb=workbook () #新建工作簿 Ws=wb.active #活动工作表 n=0 #初始化计数 #输入表头 Ws.cell (row=n+1,column=1). value= ' Serial number ' Ws.cell ( row=n+1,column=2). value= ' title ' Ws.cell (row=n+1,column=3). value= ' link ' #设置起止页码 pageindex=1 pagecount=10 #filenam E=str (PageCount) +filename while Pageindex<=pagecount:url=u ' http://blog.sina.com.cn/s/articlelist_xxxxxxx Xx_0_ ' +str (pageindex) + '. html ' t=threading. Thread (target=get_title,args= (URL,)) #添加子线程 threads.append (t) #加入线程列表 pageindex+=1 #下一页 for T in  Threads:t.setdaemon (True) #守护线程 must T.start () #开始线程 #若t before Start (). Join () put here becomes a single-threaded for-T in Threads:t.join () #线程等待 The main thread must be Wb.save (filename=filename) after the execution of the child thread is completed #保存工作 Book End=time.time () #结束时间 print (' usedtime%f seconds! '% (end-start) + ' \ n ') #输出耗时!

　　

　　

　　

　　

　　

　　

Python Learning notes (i)