Python Crawl site URL gadget

1. Install the Python requests module (via PIP):

The environment has been built!

2, test the process of crawling URLs:

Crawled out of the URL has JavaScript code, the regular on still need to be more perfect, interested can study next ~!

Tool Source code:

#coding: Utf-8

Import Sys

Import re

Import requests

#获取输入URL and get the page text

input = raw_input ("Please input URL, format like this (http://www.baidu.com):")

print ' Input:%s '% input

R = requests.get (input)

data = R.text

#利用正则查找所有URL

Link_list =re.findall (r "<=href=\"). +? =\")| (? <=href=\ '). +? (? =\ ') ", data)

Count = 0

For URL in link_list:

File = Open ("C:\\Test.txt", "a")

File.write (url+ "\ n")

Count = Count + 1

Print URL

print ' \ n '

print ' Total URL is: ' + str (count)

print ' \ n '

print ' crawling achieve ... '

File.close ()

Python Crawl site URL gadget