Simple implementation of Web content parsing

• Overview

In the daily development work, sometimes we need to go to some Web site crawl data, to crawl data, we must first understand the structure of the Web page, according to the specific structure of the page, write the corresponding program to collect data. There's just a need to update the shipping address recently. Because the system's existing delivery address is a long-ago data, the user in the process of use reflects the location of the user can not find the address information, so the update of the existing address data is also put on the agenda.

Through the search, finally found the National Bureau of Statistics of the People's Republic of China online address data needed, official channels, the accuracy of data, integrity are guaranteed. This article uses the statistics published by the National Bureau of Statistics (currently the latest data) as an example to illustrate the analysis of the page structure I do not say much here (develop basic skills), directly on the code bar, and attach the full demo.

• Need to parse page effects

• POM configuration file

<Projectxmlns= "http://maven.apache.org/POM/4.0.0"Xmlns:xsi= "Http://www.w3.org/2001/XMLSchema-instance"xsi:schemalocation= "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">  <modelversion>4.0.0</modelversion>  <groupId>Crawler</groupId>  <Artifactid>Crawler</Artifactid>  <version>0.0.1-snapshot</version>  <Build>    <sourcedirectory>Src</sourcedirectory>    <Plugins>      <plugin>        <Artifactid>Maven-compiler-plugin</Artifactid>        <version>3.5.1</version>        <Configuration>          <Source>1.6</Source>          <Target>1.6</Target>        </Configuration>      </plugin>    </Plugins>  </Build>  <Dependencies>      <Dependency>        <groupId>Org.jsoup</groupId>        <Artifactid>Jsoup</Artifactid>        <version>1.7.2</version>    </Dependency>    <Dependency>        <groupId>Net.sf.json-lib</groupId>        <Artifactid>Json-lib</Artifactid>        <version>2.4</version>        <classifier>Jdk13</classifier><!--Specify JDK version -    </Dependency>  </Dependencies></Project>

• Address node location file

 Packagecrawler;Importjava.util.List; Public classLocation {String code;    String name; List<Location>children;  PublicLocation () {} PublicLocation (string code, string name) { This. Code =Code;  This. Name =name; }     PublicString GetCode () {returnCode; }     Public voidSetcode (String code) { This. Code =Code; }     PublicString GetName () {returnname; }     Public voidsetName (String name) { This. Name =name; }     PublicList<location>GetChildren () {returnchildren; }     Public voidSetchildren (list<location>children) {         This. Children =children; }}

• Test class Testmain

 Packagecrawler;Importjava.io.IOException;Importjava.util.ArrayList;Importjava.util.List;ImportJava.util.regex.Matcher;ImportJava.util.regex.Pattern;ImportOrg.jsoup.Jsoup;Importorg.jsoup.nodes.Document;Importorg.jsoup.nodes.Element;Importorg.jsoup.select.Elements;ImportNet.sf.json.JSONObject; Public classTestmain { Public Static voidMain (string[] args)throwsIOException {Document doc= Jsoup.connect ("http://www.stats.gov.cn/tjsj/tjbz/xzqhdm/201703/t20170310_1471429.html"). get (); Element Masthead= Doc.select ("Div.xilan_con"). First (). Getelementsbyclass ("Trs_editor"). First (). Getelementsbyclass ("Trs_preappend"). First (); Elements allelements= Masthead.getelementsbytag ("P"); List<Location> provincelist =NewArraylist<location>(); Location Province=NULL; Location City=NULL;  for(Element element:allelements) {String HTML= Element.select ("Span[lang]"). First (). HTML (); String LocationCode=Testmain.getlocationcode (HTML); String LocationName= Element.select ("Span[style]"). Last (). HTML (); if(Locationcode.endswith ("0000")) {//Province or municipalityProvince =NewLocation (LocationCode, LocationName); Province.setchildren (NewArraylist<location>());            Provincelist.add (province); }Else if(Locationcode.endswith ("00")) {//CityCity =NewLocation (LocationCode, LocationName); City.setchildren (NewArraylist<location>());            Province.getchildren (). Add (city); }Else{//County or districtLocation County =NewLocation (LocationCode, LocationName);            City.getchildren (). Add (county); }} location Root=NewLocation ("0", "root");        Root.setchildren (provincelist); Jsonobject Jsonobj=Jsonobject.fromobject (root);    System.out.println (Jsonobj.tostring ()); }     Public Staticstring Getlocationcode (string html) {string regEx= "[^0-9]"; Pattern P=Pattern.compile (regEx); Matcher m=p.matcher (HTML); returnM.replaceall (""). Trim (); }}

• Results (data download)

Welcome reprint, reprint must indicate the source

Simple implementation of Web content parsing