JAVA analysis html algorithm (JAVA Web page spider algorithm)

 

Recently, some friends are working on Spider algorithms or in-depth data mining on webpages. However, when encountering complicated and complicated html pages, everyone is discouraged. It is difficult to obtain the corresponding data.

The oldest way is to use regular expressions to estimate the cost of tedious things and waste our precious time.

The second method is to use the open-source htmlparser package. This is an old project, but the effect is not estimated to be very good. It seems that html cannot be analyzed in depth and only 5 levels of structure can be analyzed;

I have the source code of htmlparser. You can obtain the source code of all hyperlinks.

               Parser parser =  Parser("http://www.lovezan.com"= parser.extractAllNodesThatMatch(                  (node  LinkTag)               ( i = 0; i < nodeList.size(); i++=                 (n.extractLink().equals("http://www.zuzwn.com"

The third method is also a method I have been using. First, I want to clear html as xml, and then Parse xml to get data using java. Now I want to upload the source code of a java clean html file:

/* * To change this template, choose Tools | Templates * and open the template in the editor. */package exec;import java.io.File;import java.io.IOException;import org.htmlcleaner.CleanerProperties;import org.htmlcleaner.HtmlCleaner;import org.htmlcleaner.PrettyXmlSerializer;import org.htmlcleaner.TagNode;/** * */public class HtmlClean {    public void cleanHtml(String htmlurl, String xmlurl) {        try {            long start = System.currentTimeMillis();            HtmlCleaner cleaner = new HtmlCleaner();            CleanerProperties props = cleaner.getProperties();            props.setUseCdataForScriptAndStyle(true);            props.setRecognizeUnicodeChars(true);            props.setUseEmptyElementTags(true);            props.setAdvancedXmlEscape(true);            props.setTranslateSpecialEntities(true);            props.setBooleanAttributeValues("empty");            TagNode node = cleaner.clean(new File(htmlurl));            System.out.println("vreme:" + (System.currentTimeMillis() - start));            new PrettyXmlSerializer(props).writeXmlToFile(node, xmlurl);            System.out.println("vreme:" + (System.currentTimeMillis() - start));        } catch (IOException e) {            e.printStackTrace();        }    }}