1. Why learn pythonin terms of programming language, the undergraduate has been used in C + + for several years, because the postgraduate direction and machine learning related, so most of the recent Learning machine learning, read the "machine learning Combat" This book, the examples are in Python to write, and at present, The more supported language for machine learning algorithms is the python,matlab/octave of course, which is also well suited for machine learning, but the academic tools, the speed and so on are certainly not as good as Python, industrial development or Python, C + +.
in short, for learning machine learning, Python and NumPy library to be familiar.
so this two days decided to learn python, looked for a good review of the public class for two days, while looking at the code, feeling that Python is really a very simple language, as long as a little C + +, C, Java or other language Foundation, a day or two can be fully introductory python. Of course the introduction is simple, proficient difficult, or to rely on more practice.
The following simple small reptile program is after watching the video after writing, is considered the first small program2, Reptile small programbecause I just want to see Andrew Ng's machine learning course, I used this crawler to crawl the video on the webpage, the Web address:Http://v.163.com/special/opencourse/machinelearning.html
Right-click on the source code and discover that the video format that provides the download is ". mp4" suffix:
The video downloaded on the Web page is in the source code:href= 'http://mov.bn.netease.com/mobilev/2011/9/8/V/S7CTIQ98V.mp4 ' you can then write the regular expression you want to match: R=r "href= ' (http.*\.mp4) '"
the next task is to get the Web page source code, and then find all the strings that match the regular r in the source code. Grab source code can take advantage of urllib in the Urlopen () method: Page=urllib.urlopen (URL), return is a page of the object pages, through the Html=page.read () can save the page source code into the HTML variable.
after the source code is captured, it is necessary to find and obtain all of the inside:href= 'Http://mov.bn.netease.com/mobilev/2011/9/8/V/S7CTIQ98V.mp4'can be obtained through regular R, and the FindAll method in the regular module re: Mp4list=re.findall (re_mp4,html)FindAll Returns the list, the element in the table is the address of the video, such as the following is a video address: Http://mov.bn.netease.com/mobilev/2011/9/8/V/S7CTIQ98V.mp4
after capturing the video address, use the Urlretrieve () method in the module urllib to download the video through the video address: Urllib.urlretrieve (mp4url),Mp4url is the element in Mp4list. In addition, you can also give the downloaded video name:Urllib.urlretrieve (Mp4url, "%s.mp4"%filename), this filename is a variable, and when a video is downloaded, it adds 1, so all videos are named 1.mp4,2.mp4,3.mp4 ...........
to make it easier to see the download progress, add a sentence after Urllib.urlretrieve (Mp4url, "%s.mp4"%filename):print ' file '%s.mp4 "Done '%filename,This will output a line of hints after downloading a video
The results are as follows:
The code is as follows: (python2.6)
#!/usr/bin/pythonimport re import urllibdef gethtml (URL):p age=urllib.urlopen (URL) html=page.read () return htmldef GETMP4 (HTML): R=r "href= ' (http.*\.mp4) '" Re_mp4=re.compile (R) mp4list=re.findall (re_mp4,html) filename=1for Mp4url in Mp4List:urllib.urlretrieve (Mp4url, "%s.mp4"%filename) print ' file '%s.mp4 ' done '%filenamefilename+=1url=raw_ Input ("Please input the source URL:") html=gethtml (URL) getMp4 (HTML)
Python crawler captures video on a Web page in bulk
