Ruby-Get product links on the website

The scripts written in Ruby run faster, and you decided to use ruby.

Compared with my e-commerce website, I have the opportunity to use it: Get all URLs on a page.

Save the following script as URLScan. RB. Usage: run the CMD command to enter the script path and run Ruby URLScan. rb url.

In the script, Data. Scan () can be placed into other regular expressions you want to search for on the corresponding page, such as data. Scan (/<a href = "(.*?) ">/), That is, all URLs

Obtain all product links in the following example.

URLScan. Rb

Require 'net/HTTP 'filename = file. new ('C: \ 1.txt ', "W +") if $ * [0] = nilputs "not URL, please for: Ruby URLScan. RB www.gaopeng.com "elseh = net: HTTP. new ($ * [0], 80) resp, Data = H. post ('HTTP: // www.gaopeng.com/', nil) # obtain the tag of the page for obtaining all URLs, for example:/index.html if resp. message = "OK" data. scan (/<a target = "_ blank" href = "(. *?) "/). Each do | x | # Get all URLs marked as href # <a target =" _ blank "href =" http://www.gaopeng.com/deals/beijing/ftuan/1431948203 "> puts xfilename. Puts xendendend

Here, the POST method in HTTP. Rb is used. If an error occurs in the get method, no get () is used ().