PowerShell Crawl Web Table

Source: Internet
Author: User

Today, I inadvertently saw a missionary wrote a blog post http://www.cnblogs.com/piapia/p/5367556.html (two crawlers in PowerShell), very inspired, he tried to grab a bit, successfully crawled the Web page form. Because I am the English version of the system, the interface of the Chinese system into a string is garbled, so the test is on the English page operation.


PowerShell 5 has a new function called convertfrom-string, whose function is to convert a string into an object. One of the parameters is to match the corresponding part of the string to generate the object according to the specified template, and we can use this function to crawl the table in the page.

Detailed Help documentation links

https://technet.microsoft.com/library/dn807178 (v=wps.640). aspx


First look at a basic example

$a =@ ' 1 2 3 6 7 co2:2 2 2 3 ' @ $t =@ ' {co1*:1} {} {Co3:3} {Co4:4}{co1*:5} 6 7 8 ' @ $c = $a | Convertfrom-string-delimiter "\ r \ n" $d = $a | Convertfrom-string-templatecontent $t


The same string, the first I use a delimiter to return a newline to generate an object, and the second I use a custom template format to match. Note that the format of the attribute definition is delimited by {}, and then the first one requires {attribute name *:}, which does not need to be added * at least to match 2 rows of data.

You can see that the first object has 3 properties, P1 is 1 2 3 4,P2 is 4 5 6 7, P3 is 9 2 2 3;

The second object is automatically matched according to each column (there is already a template that matches the first 2 rows)

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/82/21/wKioL1dMMXmT517MAAC13raKm50625.png "style=" float: none; "title=" 1.PNG "alt=" Wkiol1dmmxmt517maac13rakm50625.png "/>



Let's take a look at 2 examples.

The first example is this page, which has a list of Australia proxy servers, as shown below, I want to catch

Http://www.proxylisty.com/country/Australia-ip-list


650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M00/82/21/wKioL1dMMcPhUrWlAADhnEA-kQ4504.png "style=" float: none; "title=" 5.PNG "alt=" Wkiol1dmmcphurwlaadhnea-kq4504.png "/>


Basic idea: Invoke-restmethod directly to the entire Web page, automatically converted to a string object.

Then design the corresponding template. Because it is an HTML file, the HTML code that is converted to string will be inside. So the key is how to get these table templates with HTML code out.

Very simple, the Web page can view the source code of HTML, the following a large section of the HTML code can be copied directly from the page to paste the corresponding 2-Line table code can be slightly modified to add the property name is OK.

Then the corresponding Table object is automatically generated based on the template matching.

$web  =  ' http://www.proxylisty.com/country/Australia-ip-list ' $template  = @ ' <tr><td >{ip*:203.56.188.145}</td><td><a href= ' Http://www.proxylisty.com/port/8080-ip-list '  title= ' Port 8080 proxy list ' >{Port:8080}</a></td><td>HTTP</td> <td><a style= ' color:red; '  href= ' Http://www.proxylisty.com/anonymity/High anonymous / Elite proxy-ip-list '  title= ' High anonymous / elite proxy proxy list ' >High anonymous  / elite proxy</a></td><td>no</td><td><a href= '/HTTP Www.proxylisty.com/country/Australia-ip-list '  title= ' australia ip proxy list ' >Australia</a></td><td>13 months</td><td>2.699 sec</td><td><div  Id= "Progress-bar"  class= "all-rounded" ><div title= ' 50% '  id= "Progress-bar-percentage"   class= "all-rounded"  style= "width: 50%" >{reliability:50%}</div></div></td></ Tr><tr><td>{ip*:103.25.182.1}</td><td><a href= ' http://www.proxylisty.com/ Port/8081-ip-list '  title= ' port 8081 proxy list ' >{port:8081}</a></td><td >http</td><td><a style= ' color:red; '  href= ' http://www.proxylisty.com/anonymity/Anonymous proxy-ip-list '  title= ' anonymous proxy  proxy list ' >anonymous proxy</a></td><td>no</td><td><a  href= ' http://www.proxylisty.com/country/Australia-ip-list '  title= ' australia ip proxy  list ' >australia</a></td><td>15 months</td><td>7.242 sec</td><td ><div id= "Progress-bar"  class= "all-rounded" ><div title= ' 55% '  id= " Progress-bar-percentage " class=" all-rounded " style=" width: 55% ">{reliability:55%}</div ></div></td></tr> ' @ $temp =invoke-restmethod  -uri  $web   $result  =  ConvertFrom-String -TemplateContent  $template    -InputObject   $temp   $result   | sort reliability


Successful crawl

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/82/21/wKioL1dMMaWCCivbAAHWHcOlVps447.png "title=" 2.PNG " Style= "Float:none;" alt= "Wkiol1dmmawccivbaahwhcolvps447.png"/>


Similarly, beans have recently been more concerned about healthy food and I want to see what low GI foods are

Http://ultimatepaleoguide.com/glycemic-index-food-list

I need to get this form out of here.

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M01/82/22/wKiom1dMML-y5oUKAAHGJCyOPPg023.png "title=" 4.PNG " Style= "Float:none;" alt= "Wkiom1dmml-y5oukaahgjcyoppg023.png"/>


[Email protected] ' <tr><td valign= "top" >{food*:banana cake, made with sugar}</td><td valign= "Top" >{GI:47}</TD><TD valign= "Top" >{size:60}</td></tr><tr><td valign= "Top" >{ Food*:banana cake, made without sugar}</td><td valign= "top" >{gi:55}</td><td valign= "Top" >{ Size:60}</td></tr> ' @ $web 2= ' http://ultimatepaleoguide.com/glycemic-index-food-list/' $temp = Invoke-restmethod-uri $web 2 $result 1 = convertfrom-string-templatecontent $t 2-inputobject $temp $result 1 | Out-gridview


Success!

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/82/22/wKiom1dMMLPi0kjjAAHT85uJ0gM653.png "title=" 3.PNG " Style= "Float:none;" alt= "Wkiom1dmmlpi0kjjaaht85uj0gm653.png"/>


This is useful, especially when you need to get some information about a Web page, and of course, if the Web page itself provides a restful interface, it's even easier to get the JSON-formatted content directly.

This article is from the "Mapo Tofu" blog, please be sure to keep this source http://beanxyz.blog.51cto.com/5570417/1784596

PowerShell Crawl Web Table

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.