Use HttpClient login to get back page information

Introduction

HttpClient is the next Client Programming toolkit in the Java language that supports the HTTP protocol, which implements all the methods of the HTTP protocol, but does not support JS rendering. We are doing some small things, it may be necessary to login to some websites to obtain information, then HttpClient is your good helper, nonsense not to say, into the actual combat.

The actual meaning of a login

In the HTTP rampage today, we have to log on to some websites every day, so what is the meaning of login? First, you must have a certain understanding of cookies. Cookies are small files that are stored locally, which are sent by the server and are read and written locally by the browser. When visiting certain websites, the browser checks to see if there is any cookie information on the site, and if so, when the request is sent, the server can read the cookie information in the browser's request and can write the cookie information in response to the request. The cookie information includes the key value, content, expiration time, and the website to which it belongs.

When it comes to the fact that the cookie is almost over, what's going on? Login is the server to your browser to write cookies, if only to write cookies on your computer, then people with ulterior motives to forge a cookie will also have the opportunity to log into the site, so the server will keep a copy of the same information in memory, this process is called the session. If you click the Exit button on the website, the server will erase the in-memory cookie and clear the browser's cookie about the login. Knowing this, we can get started.

Two Find login key cookie

Here we can use Wireshark to grasp the packet analysis. Open the home, open Wireshark, start listening to the port, enter the user name and password, click Login, view Wireshark caught bag. As follows:

The first picture is the local post submission data.

The second picture is the information submitted, including the _xsrf,password,remember_me,email, note that the information submitted includes COOKIE,_XSRF can be obtained from the home page.

The third picture is the information returned by the server, note that its status is 200, and the description is successful.

The fourth chapter is the data returned by the server, note that it has three cookie settings, and information with a login success or not.

What can we know through the above steps? First, send the login request with the cookie, and the format of the post data, and then we can get the login cookie information (fourth picture).

Three use httpclient to construct login information

How does the httpclient simulate a browser? First you need to create a httpclient, which is used to emulate a browser. Next, construct a POST request, add post data information, and cookies. The detailed code is as follows:

Import Org.apache.http.*;import Org.apache.http.client.cookiestore;import Org.apache.http.client.config.cookiespecs;import Org.apache.http.client.config.requestconfig;import Org.apache.http.client.entity.urlencodedformentity;import Org.apache.http.client.methods.CloseableHttpResponse; Import Org.apache.http.client.methods.httpget;import Org.apache.http.client.methods.httppost;import Org.apache.http.client.protocol.httpclientcontext;import Org.apache.http.config.lookup;import Org.apache.http.config.registrybuilder;import Org.apache.http.cookie.cookiespecprovider;import Org.apache.http.impl.client.basiccookiestore;import Org.apache.http.impl.client.closeablehttpclient;import Org.apache.http.impl.client.httpclients;import Org.apache.http.impl.cookie.basicclientcookie;import Org.apache.http.impl.cookie.defaultcookiespecprovider;import Org.apache.http.message.basicnamevaluepair;import Org.apache.http.util.entityutils;import Java.io.ioexception;import Java.util.hashmap;import java.util.LinkedList;import Java.util.list;import Java.util.Map; /** * Created by Gavin on 15-7-23. */public class Httpclienttest {public static void main (string[] args) {//Create a httpclient requestconf        IG Requestconfig = Requestconfig.custom (). Setcookiespec (cookiespecs.standard_strict). build ();        Closeablehttpclient httpClient = Httpclients.custom (). Setdefaultrequestconfig (Requestconfig). build ();            try {//Create a GET request to receive _XSRF information httpget get = new HttpGet ("http://www.zhihu.com/");            Get _xsrf closeablehttpresponse response = Httpclient.execute (Get,context);            Setcookie (response);            String responsehtml = entityutils.tostring (Response.getentity ());            String Xsrfvalue = Responsehtml.split ("<input type=\" hidden\ "name=\" _xsrf\ "value=\" ") [1].split (" \ "/>") [0];            System.out.println ("Xsrfvalue:" + xsrfvalue);                         Response.close (); Constructing post Data List<namevaluepair> valuepairs = new linkedlist<namevaluepair> ();            Valuepairs.add (New Basicnamevaluepair ("_xsrf", Xsrfvalue));            Valuepairs.add (New Basicnamevaluepair ("email", "[email protected]"));            Valuepairs.add (New Basicnamevaluepair ("Password", "xxxxx"));            Valuepairs.add (New Basicnamevaluepair ("Remember_me", "true"));                         urlencodedformentity entity = new Urlencodedformentity (Valuepairs, consts.utf_8);            Create a POST request HttpPost post = new HttpPost ("Http://www.zhihu.com/login/email"); Post.setheader ("Cookie", "cap_id=\" yja5mje0yzyyngq2ndy5nwjhmmfhn2yyy2ewodiwzjq=|1437610072| E7cc307c0d2fe2ee84fd3ceb7f83d298156e37e0\ ";             ");            Inject post data post.setentity (entity);            HttpResponse HttpResponse = Httpclient.execute (POST);             Print Login Success Information Printresponse (HttpResponse); Constructs a GET request to test whether the login cookie gets httpget g = new HttpGet ("Http://www.zhihu.Com/question/following ");            Get the cookie information returned by the POST request String C = Setcookie (HttpResponse);            Inject the cookie into the GET request header G.setheader ("Cookie", c);            Closeablehttpresponse r = Httpclient.execute (g);            String content = entityutils.tostring (r.getentity ());            SYSTEM.OUT.PRINTLN (content);        R.close ();        } catch (IOException e) {e.printstacktrace ();            } finally {try {httpclient.close ();            } catch (IOException e) {e.printstacktrace (); }}} public static void Printresponse (HttpResponse httpresponse) throws ParseException, ioexcept        Ion {//Get response message entities httpentity entity = httpresponse.getentity ();        Response Status System.out.println ("Status:" + Httpresponse.getstatusline ());        System.out.println ("headers:");        Headeriterator iterator = Httpresponse.headeriterator (); while (Iterator.hasnext ()) {System.out.println ("\ T" + iterator.next ());            }//Determine if the response entity is empty if (entity! = null) {String responsestring = entityutils.tostring (entity);            SYSTEM.OUT.PRINTLN ("Response length:" + responsestring.length ());        SYSTEM.OUT.PRINTLN ("Response content:" + responsestring.replace ("\ r \ n", ""));    }} public static map<string,string> Cookiemap = new hashmap<string, string> (64); Gets the cookie from the response message public static String Setcookie (HttpResponse httpresponse) {System.out.println ("----setcooki        EStore ");        Header headers[] = httpresponse.getheaders ("Set-cookie");            if (headers = = NULL | | headers.length==0) {SYSTEM.OUT.PRINTLN ("----There are no cookies");        return null;        } String cookie = "";            for (int i = 0; i < headers.length; i++) {Cookie + = Headers[i].getvalue (); if (i! = headers.length-1) {               Cookie + = ";";        }} "String cookies[] = Cookie.split ("; ");            for (String c:cookies) {c = C.trim ();            if (Cookiemap.containskey (c.split ("=") [0])) {Cookiemap.remove (c.split ("=") [0]); } cookiemap.put (C.split ("=") [0], c.split ("="). Length = = 1?        "":(c.split ("="). Length ==2?c.split ("=") [1]:c.split ("=", 2) [1]);        } System.out.println ("----setcookiestore success");        String cookiestmp = "";         For (String Key:cookieMap.keySet ()) {cookiestmp +=key+ "=" +cookiemap.get (key) + ";";}    Return cookiestmp.substring (0,cookiestmp.length ()-2); }}

The process of the code is:

1. Get XSRF information from the homepage.

2. The cookie information is required in the POST request, but we do not get the cookie in the first step, please find the cookie in the browser and add it, the cookie above is what I found.

3. Submit a POST request and get a login cookie

4. Just find a sub-page that needs to be logged in, write the resulting cookie to the request header, submit the request, and see if the login was successful.

Four-result verification

The first image shows that you have a cookie and signed in successfully

The second picture shows that you have entered the interface that needs to be logged in

Summarize

When we need to log in to an interface to get information, we need to know what the login actually does, that is, read and write cookie,post data.

When a cookie is obtained, it needs to be fetched from the response header, which needs to be written in a timely manner when the server sends new cookie information.

When we can login to a website, how to operate its content, here is recommended Jsoup, conscience library, imitation jquery operation mode.

Excerpt from Open source China Community: http://my.oschina.net/jiangmitiao/blog/483092

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Use HttpClient login to get back page information