Today, I inadvertently saw a missionary wrote a blog post http://www.cnblogs.com/piapia/p/5367556.html (two crawlers in PowerShell), very inspired, he tried to grab a bit, successfully crawled the Web page form. Because I am the English version of the system, the interface of the Chinese system into a string is garbled, so the test is on the English page operation.
PowerShell 5 has a new function called convertfrom-string, whose function is to convert a string into an object. One of the parameters is to match the corresponding part of the string to generate the object according to the specified template, and we can use this function to crawl the table in the page.
Detailed Help documentation links
https://technet.microsoft.com/library/dn807178 (v=wps.640). aspx
First look at a basic example
$a =@ ' 1 2 3 6 7 co2:2 2 2 3 ' @ $t =@ ' {co1*:1} {} {Co3:3} {Co4:4}{co1*:5} 6 7 8 ' @ $c = $a | Convertfrom-string-delimiter "\ r \ n" $d = $a | Convertfrom-string-templatecontent $t
The same string, the first I use a delimiter to return a newline to generate an object, and the second I use a custom template format to match. Note that the format of the attribute definition is delimited by {}, and then the first one requires {attribute name *:}, which does not need to be added * at least to match 2 rows of data.
You can see that the first object has 3 properties, P1 is 1 2 3 4,P2 is 4 5 6 7, P3 is 9 2 2 3;
The second object is automatically matched according to each column (there is already a template that matches the first 2 rows)
650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/82/21/wKioL1dMMXmT517MAAC13raKm50625.png "style=" float: none; "title=" 1.PNG "alt=" Wkiol1dmmxmt517maac13rakm50625.png "/>
Let's take a look at 2 examples.
The first example is this page, which has a list of Australia proxy servers, as shown below, I want to catch
Http://www.proxylisty.com/country/Australia-ip-list
650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M00/82/21/wKioL1dMMcPhUrWlAADhnEA-kQ4504.png "style=" float: none; "title=" 5.PNG "alt=" Wkiol1dmmcphurwlaadhnea-kq4504.png "/>
Basic idea: Invoke-restmethod directly to the entire Web page, automatically converted to a string object.
Then design the corresponding template. Because it is an HTML file, the HTML code that is converted to string will be inside. So the key is how to get these table templates with HTML code out.
Very simple, the Web page can view the source code of HTML, the following a large section of the HTML code can be copied directly from the page to paste the corresponding 2-Line table code can be slightly modified to add the property name is OK.
Then the corresponding Table object is automatically generated based on the template matching.
$web = ' http://www.proxylisty.com/country/Australia-ip-list ' $template = @ ' <tr><td >{ip*:203.56.188.145}</td><td><a href= ' Http://www.proxylisty.com/port/8080-ip-list ' title= ' Port 8080 proxy list ' >{Port:8080}</a></td><td>HTTP</td> <td><a style= ' color:red; ' href= ' Http://www.proxylisty.com/anonymity/High anonymous / Elite proxy-ip-list ' title= ' High anonymous / elite proxy proxy list ' >High anonymous / elite proxy</a></td><td>no</td><td><a href= '/HTTP Www.proxylisty.com/country/Australia-ip-list ' title= ' australia ip proxy list ' >Australia</a></td><td>13 months</td><td>2.699 sec</td><td><div Id= "Progress-bar" class= "all-rounded" ><div title= ' 50% ' id= "Progress-bar-percentage" class= "all-rounded" style= "width: 50%" >{reliability:50%}</div></div></td></ Tr><tr><td>{ip*:103.25.182.1}</td><td><a href= ' http://www.proxylisty.com/ Port/8081-ip-list ' title= ' port 8081 proxy list ' >{port:8081}</a></td><td >http</td><td><a style= ' color:red; ' href= ' http://www.proxylisty.com/anonymity/Anonymous proxy-ip-list ' title= ' anonymous proxy proxy list ' >anonymous proxy</a></td><td>no</td><td><a href= ' http://www.proxylisty.com/country/Australia-ip-list ' title= ' australia ip proxy list ' >australia</a></td><td>15 months</td><td>7.242 sec</td><td ><div id= "Progress-bar" class= "all-rounded" ><div title= ' 55% ' id= " Progress-bar-percentage " class=" all-rounded " style=" width: 55% ">{reliability:55%}</div ></div></td></tr> ' @ $temp =invoke-restmethod -uri $web $result = ConvertFrom-String -TemplateContent $template -InputObject $temp $result | sort reliability
Successful crawl
650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/82/21/wKioL1dMMaWCCivbAAHWHcOlVps447.png "title=" 2.PNG " Style= "Float:none;" alt= "Wkiol1dmmawccivbaahwhcolvps447.png"/>
Similarly, beans have recently been more concerned about healthy food and I want to see what low GI foods are
Http://ultimatepaleoguide.com/glycemic-index-food-list
I need to get this form out of here.
650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M01/82/22/wKiom1dMML-y5oUKAAHGJCyOPPg023.png "title=" 4.PNG " Style= "Float:none;" alt= "Wkiom1dmml-y5oukaahgjcyoppg023.png"/>
[Email protected] ' <tr><td valign= "top" >{food*:banana cake, made with sugar}</td><td valign= "Top" >{GI:47}</TD><TD valign= "Top" >{size:60}</td></tr><tr><td valign= "Top" >{ Food*:banana cake, made without sugar}</td><td valign= "top" >{gi:55}</td><td valign= "Top" >{ Size:60}</td></tr> ' @ $web 2= ' http://ultimatepaleoguide.com/glycemic-index-food-list/' $temp = Invoke-restmethod-uri $web 2 $result 1 = convertfrom-string-templatecontent $t 2-inputobject $temp $result 1 | Out-gridview
Success!
650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/82/22/wKiom1dMMLPi0kjjAAHT85uJ0gM653.png "title=" 3.PNG " Style= "Float:none;" alt= "Wkiom1dmmlpi0kjjaaht85uj0gm653.png"/>
This is useful, especially when you need to get some information about a Web page, and of course, if the Web page itself provides a restful interface, it's even easier to get the JSON-formatted content directly.
This article is from the "Mapo Tofu" blog, please be sure to keep this source http://beanxyz.blog.51cto.com/5570417/1784596
PowerShell Crawl Web Table