Crawl data after logging in

This time, a customer needs to crawl another site's data, including data submissions. These actions need to be completed after logging in. There are no technical difficulties. The key is to use Fiddler to find the parameters and URLs.

Remember login status

HttpClient can remember the login status, log-in can be said HttpClient saved up.

 PrivateHttpClient _client;  PublicHttpClient HttpClient {Get            {                if(_client = =NULL)                {                    if(session["Client"]!=NULL) {_client= session["Client"] asHttpClient; }                    Else                    {                        varHandler =NewHttpclienthandler {automaticdecompression=Decompressionmethods.gzip, UseCookies=true, Proxy=NewWebProxy ("http://ip:8080/",true,NULL,                                    NewNetworkCredential ("Username",the PWD",the domain"))};//Agent _client=NewHttpClient (handler); _client. Defaultrequestheaders.add ("user-agent","mozilla/5.0 (Windows NT 6.1; WOW64) applewebkit/537.36 (khtml, like Gecko) chrome/36.0.1985.143 safari/537.36"); Clientlogin (NewClientlogomodel ()); session["Client"] =_client; }                                  }                return_client; }        }

Because the target site is a JSON-passed parameter. is also a parameter returned with JSON. is not a form submission format. So the parameters are also converted to JSON before the post.

  Public ObjectClientlogin (Clientlogomodel logomodel) {if(Logomodel = =NULL) {Logomodel=NewClientlogomodel (); }            vardata = JsonConvert.            SerializeObject (Logomodel); varLogoparams =Newlist<keyvaluepair<string,string>>(); Logoparams.add (Newkeyvaluepair<string,string> ("Data", data)); varResponse = _client. Postasync (NewUri (Logonurl),New formurlencodedcontent (logoparams)).            Result; varresult =Response. Content.readasstringasync ().            Result; returnresult; }

Return Data Conversion

The URL is obtained from the left side of fiddler, and the right TextView is the parameter format, and the data format is returned below.

Every time you convert, write generics.

     PublicT gettlist<t> (ObjectObjstringURL) {            vardata =jsonconvert.serializeobject (obj);; varParamlist =NewList<keyvaluepair<string, string>> {Newkeyvaluepair<string,string> ("Data", data)}; varResponse = Httpclient.postasync (NewUri (URL),Newformurlencodedcontent (paramlist)).            Result; varresult =Response. Content.readasstringasync ().            Result; returnJsonconvert.deserializeobject<t>(result); }

Call:

     Public ActionResult tradepage (tradequeryparm param)        {       var data = Gettlist<traderequstresult >(obj, tradelisturl);             return Partialview (data);        }

The front end then passes the parameters over.

  $.post ("/trade/tradepage", {                    agentname:agentname, shortname:shortname,                    startdate:startdate, EndDate: EndDate, page:cpage                function  (data) {                    $ ("#mtable"). HTML (data);  }

Crawl data after logging in