Crawling to the urltask of millions of information

The function of this module is to get a list of followers from the Nexturl queue by getting the URL of the user's watchlist. JSON data returned by the server

Encapsulates a class of serialization and deserialization of an object

 Public   classSerializehelper {/// <summary>        ///Serialization of Data/// </summary>        /// <param name= "value" ></param>        /// <returns></returns>         Public Static stringSerializetostring (Objectvalue) {            returnJsonconvert.serializeobject (value); }        /// <summary>        ///deserialization Operation/// </summary>        /// <typeparam name= "T" ></typeparam>        /// <param name= "str" ></param>        /// <returns></returns>         Public StaticT deserializetoobject<t> (stringstr) {                      returnJsonconvert.deserializeobject<t>(str); }}

Package Urltask Class

 Public class Urltask    {         Private  string Get Set ; }         Private string Get Set ; }          Public Urltask (string  _url)        {            = _url;          }}

Add a method to get resources

Private BOOL gethtml ()        {            jsonstring= httphelp.downloadstring (URL);            Console.WriteLine ("json download complete ");             return ! string . IsNullOrEmpty (jsonstring);        }
Parsing JSON methods

  Public  voidanalyse () {Try            {                if(gethtml ()) {Stopwatch Watch=NewStopwatch (); Watch.                                   Start (); Followerresult result= serializehelper.deserializetoobject<followerresult>(jsonstring); if(!result.paging.is_end) {Rediscore.pushintolist (1,"Nexturl", Result.paging.next); }                                               foreach(varIteminchresult.data) {intType=math.abs (item. GetHashCode ())%3+3; if(Rediscore.insetintohash (Type,"Urltokenhash", Item.url_token,"exist") {rediscore.pushintolist (1,"Urltoken", Item.url_token); }} watch.                    Stop (); Console.WriteLine ("parsing json with {0} milliseconds", watch.                Elapsedmilliseconds.tostring ()); }            }            Catch(Exception ex) {Console.WriteLine (ex).            ToString ()); }                  }

Parsing: If Result.paging.is_end is true, then this is the last page of the user's attention list, then its nexturl should be added to the queue, not to join, for the following user array, because the information does not go all, do not have the ID to go to the home page for more information.

Crawling to the urltask of millions of information