Java Jsoup Library: The basic use of web crawler

The following http://news.csdn.net/news.html for the Crawler sample site analysis, F12 view page layout, the effect is as follows: analysis, the individual data to be grilled to the unit as a tag, that is, a unit tag corresponding to an object. The following defines the Jsoup helper class, Getrootelements () returns the root tag of the object that needs to be parsed, which needs to be returned Document.getelementsbyclass ("unit"). The Anaylizerootelement (Element rootelemen) method resolves the specified rootelemen to the desired object.

 PackageCom.coca.android_unity_lab.joke;

ImportCom.coca.unity_dev_utils.android.log.UtilsLog;
ImportCom.coca.unity_dev_utils.android.log.UtilsLogFactory;
ImportCom.coca.unity_dev_utils.java.UtilsCollections;

ImportOrg.jsoup.nodes.Document;
ImportOrg.jsoup.nodes.Element;
ImportOrg.jsoup.select.Elements;

ImportJava.util.List;

/**
* Created by Administrator on 2016/6/16.
*/
Public abstract classJsouphelper {
    private static FinalUtilslogLG= Utilslogfactory.GetLogger(Jsouphelper.class);
PrivateDocumentDocument;

 PublicJsouphelpersetdocument(Document Document) {
         This.Document= document;
return this;
    }

    Public voidStartanaylizebyjsoup() {
Elements rootelements = getrootelements (Document);
        LG. E ("Get started with Jsoup analysis data: Analizejsoup, total data volume:"+ rootelements.size ());
 for(Element rootelement:rootelements) {
Anaylizerootelement (rootelement);
        }
}

    /**
* Gets the parsed root collection
     *
     * @paramDocument
     * @return
     */
    Public AbstractElementsgetrootelements(Document Document);

    /**
* Generate corresponding Java objects based on each root layout
     *
     * @paramrootelement
     * @return
     */
    Public abstract voidanaylizerootelement(Element rootelement);
}

Here's how it works:

Jsouphelper=NewJsouphelper () {
    @Override
     PublicElementsgetrootelements(Document Document) {
        returnDocument.getelementsbyclass ("unit");
    }

    @Override
    Public voidanaylizerootelement(Element rootelement) {
jokeadapterentity entity =NewJokeadapterentity ();
        Element contentelement = Jsouphelper.paraseelement(rootelement, Utilscollections.createlistthroughmulitparamters("H1", "a"));
        Entity.setcontent (Contentelement.text ());

        Element imageelement = Jsouphelper.paraseelement(rootelement, Utilscollections.createlistthroughmulitparamters("DL", "DT", "a", "img"));
if(Imageelement! =NULL) {
            LG. E ("Captured data:"+ imageelement.attr ("src"));
            Entity.setimgurl (Imageelement.attr ("src"));
        }
        Adapter. Adddataresource (0, Entity;
    }
};

Call the following method,

jsouphelper. Setdocument (Jsoup.  Parse(response)). Startanaylizebyjsoup ();

Here's how to recursively search for a specified label:

/**
* Recursive parsing tags
 * @paramelement
 * @paramTags tags search rules in sequence
 * @return
 */
Public StaticElementparaseelement(element element, List<string> tags) {
    if(Utilscollections.Iscollectionnotempty(tags)) {
String Parsetag = Tags.get (0);
        Elements Elements = Element.getelementsbytag (Parsetag);

BooleanIselementsnotempty = elements! =NULL&& elements.size () >0;
        LG. E ("Parse tag:"+ Parsetag +", Size is"+ (Iselementsnotempty? Elements.size ():0));
if(Iselementsnotempty) {
            returnparaseelement(Elements.first (), Tags.sublist (1, Tags.size ()));
        }Else{
            LG. E ("element set under this tab is empty, return null");
return null;
        }
}Else{
        LG. E ("Find specified element");
returnElement;
    }
}

From for notes (Wiz)

Java Jsoup Library: The basic use of web crawler