Record a simple httpclient crawl page content

Today's network era, the HTTP protocol is so important, with the development of Java, more and more people use Java directly through the HTTP protocol access to network resources, Although java.net provides basic functionality to access the HTTP protocol, there are still many features that are not flexible to most applications, HttpClient is a subproject under Apache Jakarta Common, A Java toolkit that provides access to the HTTP protocol, provides more, faster, and richer methods, httpclient the main common functions are: Implement all HTTP methods (Get,post,put,head,delete, etc.); support redirection; Support HTTPS Support Proxy Server. , so for httpclient to know some of the necessary, the previous company used it to dynamically log on to the other system and crawling data, thanks to a small example, is the following program,

simply get the content of a Web page:

HttpClient client =NewHttpClient (); Try{GetMethodGet=NewGetMethod ("http://www.baidu.com"); System. out. println ("Executing request"+Get. GetURI ()); Client.executemethod (Get); intStatusCode =Get. Getstatuscode (); System. out. println ("Status:"+statuscode); if(StatusCode = =HTTPSTATUS.SC_OK) {HTML=Get. getresponsebodyasstring (); } System. out. println (HTML); Get. Releaseconnection (); } Catch(Exception e) {//Todo:handle Exception       }

GetMethod () can be used in two ways, the first of which is directly like the above example passing through the entire URL address of new GetMethod ("http://www.baidu.com"), and the other can set the host and port,postmethod post = New Postmethod ("/jsp/login.jsp");

Client.gethostconfiguration (). Sethost (host, Port);

Then call HttpClient's Executemethod execution. Getstatecode Get Status Code

You can also use the Post method to achieve background simulation login:

HttpClient client =NewHttpClient (); Try{String host= "localhost"; intPort = 8080; Postmethod Post=NewPostmethod ("/jsp/login.jsp");                     Client.gethostconfiguration (). Sethost (host, Port); System.out.println ("Executing request" +Post.geturi ()); Namevaluepair name1=NewNamevaluepair ("username", "000001"); Namevaluepair name2=NewNamevaluepair ("Password", "888888"); Post.setrequestbody (Newnamevaluepair[]{name1,name2}); Try{Client.executemethod (POST); } Catch(Exception e) {e.printstacktrace (); }                     intStatusCode =Post.getstatuscode (); System.out.println ("Status:" +statuscode); if(StatusCode = =HTTPSTATUS.SC_OK) {HTML=post.getresponsebodyasstring (); } Else if((statuscode = = httpstatus.sc_moved_temporarily) | |(StatusCode= = httpstatus.sc_moved_permanently) | |(StatusCode= = Httpstatus.sc_see_other) | | (StatusCode = =httpstatus.sc_temporary_redirect)) {                    //read the new URL addressHeader Header= Post.getresponseheader ("Location"); if(Header! =NULL) {String Newuri=Header.getvalue (); System.out.println ("Newuri:" +Newuri); if(Newuri = =NULL) || (Newuri.equals ("")) {Newuri= "/"; } GetMethod Get=NewGetMethod (Newuri); Try{Client.executemethod (GET); } Catch(Exception e) {e.printstacktrace (); }                        byte[] Readbody =Get.getresponsebody (); String Response=NewString (readbody); System.out.println ("=================================================");                        SYSTEM.OUT.PRINTLN (response); System.out.println ("=================================================");                    Get.releaseconnection ();           }} System.out.println (HTML);       Post.releaseconnection (); } Catch(Exception e) {//Todo:handle Exception       }       returnhtml

when using Namevaluepair to encapsulate data, using the Tongue setrequestbody () to send data, if you want to get the transmitted data on the front page, it can be obtained by Request.getparameter () method.

Record a simple httpclient crawl page content