Java Parsing HTML algorithm (Java Web spider algorithm example) _java

It is daunting to encounter complex and cumbersome HTML pages. Because it is difficult to obtain the appropriate data.

The oldest way is to try to use regular expressions to estimate that the cumbersome things outweigh the gains and waste our precious time.

The second method uses the open source organization Htmlparser The package, this is an older project, but the effect estimate is not very good, as if cannot analyze the HTML deeply, can only analyze the 5 level structure;

I have a Htmlparser source code to get all the hyperlinks

Copy Code code as follows:

/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
Package test;

Import Java.util.HashMap;
Import Java.util.Map;

Import Org.htmlparser.Node;
Import Org.htmlparser.NodeFilter;
Import Org.htmlparser.Parser;
Import Org.htmlparser.tags.LinkTag;
Import org.htmlparser.util.NodeList;

public class Getlinktest {

public static void Main (string[] args) {

try {
Filter out <A> labels through filters
Parser Parser = new Parser ("http://www.jb51.net");
NodeList nodelist = Parser.extractallnodesthatmatch (new Nodefilter () {
Implement this method to filter labels
public Boolean Accept (node node) {
if (node instanceof Linktag)//Tag
{
return true;
}
return false;
}
});
Print
for (int i = 0; i < nodelist.size (); i++) {
Linktag n = (linktag) nodelist.elementat (i);
System.out.print (N.getstringtext () + "==>>");
System.out.println (N.extractlink ());
try {
if (N.extractlink (). Equals ("Http://www.jb51.net")) {
System.out.println (N.extractlink ());
}
catch (Exception e) {
}
}
catch (Exception e) {
E.printstacktrace ();
}

}
}

The third option, which I've been working on, is to first clean HTML into XML, then get the data in Java parsing XML, and now upload the source code for Java clear HTML:

Copy Code code as follows:

/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
Package exec;

Import Java.io.File;
Import java.io.IOException;
Import org.htmlcleaner.CleanerProperties;
Import Org.htmlcleaner.HtmlCleaner;
Import Org.htmlcleaner.PrettyXmlSerializer;
Import Org.htmlcleaner.TagNode;

/**
*
*/
public class Htmlclean {

public void cleanhtml (string htmlurl, String xmlurl) {
try {
Long start = System.currenttimemillis ();

Htmlcleaner cleaner = new Htmlcleaner ();
Cleanerproperties props = Cleaner.getproperties ();
Props.setusecdataforscriptandstyle (TRUE);
Props.setrecognizeunicodechars (TRUE);
Props.setuseemptyelementtags (TRUE);
Props.setadvancedxmlescape (TRUE);
Props.settranslatespecialentities (TRUE);
Props.setbooleanattributevalues ("Empty");

Tagnode node = Cleaner.clean (new File (Htmlurl));

System.out.println ("Vreme:" + (System.currenttimemillis ()-start));

New Prettyxmlserializer (props). Writexmltofile (node, xmlurl);

System.out.println ("Vreme:" + (System.currenttimemillis ()-start));
catch (IOException e) {
E.printstacktrace ();
}
}
}