httpWebRequest擷取流和WebClient的檔案抓取

標籤：referer   抓取資料   line   出錯   .com   file   儲存檔案   down   win   

httpWebRequest擷取流和WebClient的檔案抓取

昨天寫一個抓取，遇到了一個坑，就是在擷取網路流的時候，人為的使用了stream.Length來擷取流的長度，擷取的時候會拋出錯誤，查了查文檔，原因是某些流是無法擷取到資料的長度的，所以不能直接得到。如果是常和stream打交道就能避免這個問題。其實直接使用do-while來擷取就行了，代碼如下：

int i=0;do{    byte[] buffer = new byte[1024];    i = stream.Read(buffer, 0, 1024);    fs.Write(buffer, 0, i);} while (i >0);

其中while後只能寫i>0；而不能寫成i>=1024;原因可以看MSDN中的一段解釋：msdn

僅當流中沒有更多資料且預期不會有更多資料（如通訊端已關閉或位於檔案結尾）時，Read 才返回 0。 即使尚未到達流的末尾，實現仍可以隨意返回少於所請求的位元組。

一下是httpwebrequest和webClient抓取資料的簡短代碼：

httpWebRequest

/// <summary>/// /// </summary>/// <param name="url">抓取url</param>/// <param name="filePath">儲存檔案名稱</param>/// <param name="oldurl">來源路徑</param>/// <returns></returns>public static bool HttpDown(string url, string filePath, string oldurl){    try    {        HttpWebRequest req = WebRequest.Create(url) as HttpWebRequest;        req.Accept = @"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8    ";        req.Referer = oldurl;        req.UserAgent = @" Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36    ";        req.ContentType = "application/octet-stream";        HttpWebResponse response = req.GetResponse() as HttpWebResponse;        Stream stream = response.GetResponseStream();       // StreamReader readStream=new StreamReader         FileStream fs = File.Create(filePath);        long length = response.ContentLength;        int i=0;        do        {            byte[] buffer = new byte[1024];            i = stream.Read(buffer, 0, 1024);            fs.Write(buffer, 0, i);        } while (i >0);                 fs.Close();        return true;    }    catch (Exception ex)     {         return false;    }}

WebClient

public static bool Down(string url, string desc,string oldurl){    try    {        WebClient wc = new WebClient();        wc.Headers.Add(HttpRequestHeader.Accept, @"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");        wc.Headers.Add(HttpRequestHeader.Referer, oldurl);        wc.Headers.Add(HttpRequestHeader.UserAgent, @" Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.154 Safari/537.36");        wc.Headers.Add(HttpRequestHeader.ContentType, "application/octet-stream");        wc.DownloadFile(new Uri(url), desc);        Console.WriteLine(url);        Console.WriteLine("    "+desc + "   yes!");        return true;    }    catch (Exception ex)    {        return false;    }}

httpWebRequest擷取流和WebClient的檔案抓取