Notes for using Jsoup and Jsoup

Notes for using Jsoup and Jsoup

During this time, my work was relatively idle. I was interested in reading the data when I went online to learn it.

It is really convenient to use jsoup to capture data. The only drawback is that the official API is in English. It is really troublesome for programmers who have poor English skills and can only practice it a little bit.

So with this notebook, you can study it later.

Jsoup's series of html parsing operations are based on the Document instance object. There are many instantiation methods, through URLs, files, and strings.

Here I use the url method to parse some content on a website.

Eg:

Document doc = Jsoup. connect (url). get ();

This is an officially provided instantiation method. However, when I use it, the console reports an error.

Org. jsoup. HttpStatusException: HTTP error fetching URL. Status = 403, URL = XXX

Then I found the reason on the Internet:

When the server of some websites responds to http requests, the information submitted by the client is relatively complete. In the Connection class of Jsoup, this Header is used to improve the request information.

When requesting a webpage, our browser sends some data in the header of the request, such as the browser type, version, and language. When we use Jsoup to complete the request webpage, it is best to complete the request header information.HeaderMethod.

Add

Document doc = Jsoup. connect (url ). userAgent ("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31 "). get ();

This error can be solved.

Most of the doc operations are similar to those in JavaScript.

GetElementsByTag (): Get a node

GetElementsByClass (): Get the node in a class

GetElementById (): Get the node in an id

And so on.

These methods return the Element Object.

You can continue to operate on the Element object.

Children (): Get the subnode. I tried to obtain the first subnode.

Text (): Get text

Attr (): gets the attribute value of the node.

Put a piece of code:

Document docu = Jsoup. connect (url ). userAgent ("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31 "). get ();
Elements ele = docu. getElementsByClass ("text"); // obtain the content of a node whose class is text.
For (Element e: ele) {// Loop
Elements uurl = e. children ();
String t = uurl. text (); // obtain the title
String hr = uurl. attr ("href"); // get the link
Document docum = Jsoup. connect (hr ). userAgent ("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31 "). get (); // according to the link obtained in the previous step, execute the html
Element elem = docum. getElementById ("contentText ");
Elements elems = elem. getElementsByTag ("p ");
String text = elems. text (); // get the content
}

Code snippets are used to capture data of a website.