爬去知乎百萬使用者資訊之UserTask

標籤：pack   style   catch   url   工具   ber   連結   empty   car   

UserTask是擷取使用者資訊的爬蟲模組

public   class UserManage    {        private string html;        private string url_token;     }

建構函式

 

使用者首頁的uRL格式為"https://www.zhihu.com/people/"+url_token+"/following";

 public UserManage(string urltoken)         {             url_token = urltoken;         }

先封裝一個擷取html頁面的方法

 

 private bool GetHtml()        {                            string url="https://www.zhihu.com/people/"+url_token+"/following";            html = HttpHelp.DownLoadString(url);            return  !string.IsNullOrEmpty(html);        }

拿到了html頁面，接下來是剝取頁面中的JSON，藉助HtmlAgilityPack

public  void  analyse()        {                if (GetHtml())                {                    try                    {                        Stopwatch watch = new Stopwatch();                        watch.Start();                        HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();                        doc.LoadHtml(html);                        HtmlNode node = doc.GetElementbyId("data");                        StringBuilder stringbuilder =new StringBuilder(node.GetAttributeValue("data-state", ""));                        stringbuilder.Replace("&quot;", "‘");                                   stringbuilder.Replace("&lt;", "<");                        stringbuilder.Replace("&gt;", ">");                                             watch.Stop();                       Console.WriteLine("分析Html用了{0}毫秒", watch.ElapsedMilliseconds.ToString());                                           }                    catch (Exception ex)                    {                        Console.WriteLine(ex.ToString());                    }                }                        }    

添加使用者的關注列表的連結

 private void  GetUserFlowerandNext(string json)        {                 string foollowed = "https://www.zhihu.com/api/v4/members/" + url_token + "/followers?include=data%5B*%5D.answer_count%2Carticles_count%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge%5B%3F(type%3Dbest_answerer)%5D.topics&offset=0&limit=20";                 string following = "https://www.zhihu.com/api/v4/members/" + url_token + "/followees?include=data%5B%2A%5D.answer_count%2Carticles_count%2Cfollower_count%2Cis_followed%2Cis_following%2Cbadge%5B%3F%28type%3Dbest_answerer%29%5D.topics&limit=20&offset=0";                 RedisCore.PushIntoList(1, "nexturl", following);                 RedisCore.PushIntoList(1, "nexturl", foollowed);        }

 

對json資料進一步剝取，只要使用者的資訊，藉助JSON解析工具Newtonsoft.Json

 

private void  GetUserInformation(string json)        {                  JObject obj = JObject.Parse(json);                string xpath = "[‘" + url_token + "‘]";                JToken tocken = obj.SelectToken("[‘entities‘]").SelectToken("[‘users‘]").SelectToken(xpath);                RedisCore.PushIntoList(2, "User", tocken.ToString());                      }  

現在來完成下analyse函數

  public  void  analyse()        {                if (GetHtml())                {                    try                    {                        Stopwatch watch = new Stopwatch();                        watch.Start();                        HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();                        doc.LoadHtml(html);                        HtmlNode node = doc.GetElementbyId("data");                        StringBuilder stringbuilder =new StringBuilder(node.GetAttributeValue("data-state", ""));                        stringbuilder.Replace("&quot;", "‘");                                   stringbuilder.Replace("&lt;", "<");                        stringbuilder.Replace("&gt;", ">");                        GetUserInformation(stringbuilder.ToString());                        GetUserFlowerandNext(stringbuilder.ToString());                        watch.Stop();                        Console.WriteLine("分析Html用了{0}毫秒", watch.ElapsedMilliseconds.ToString());                                           }                    catch (Exception ex)                    {                        Console.WriteLine(ex.ToString());                    }                }                        }            }    

 

 

爬去知乎百萬使用者資訊之UserTask