How C # crawls Web content using HttpWebRequest, HttpWebResponse emulation browser

 Public stringGethtml (stringURL, Encoding ed) {            stringHtml =string. Empty;//Initializes a new WebrequstHttpWebRequest Request =(HttpWebRequest) webrequest.create (URL); Request.keepalive=true; Request.protocolversion=httpversion.version11; Request.method="GET"; Request.accept="*/* "; Request.useragent="mozilla/5.0 (Windows NT 6.1) applewebkit/536.5 (khtml, like Gecko) chrome/19.0.1084.56 safari/536.5"; Request.referer=URL; HttpWebResponse Htmlresponse=(HttpWebResponse) request.getresponse (); //returning data streams from Internet resourcesStream Htmlstream =Htmlresponse.getresponsestream (); //Read Data streamStreamReader Weatherstreamreader =NewStreamReader (Htmlstream, ed); //reading DataHtml =Weatherstreamreader.readtoend ();            Weatherstreamreader.close ();            Htmlstream.close ();            Htmlresponse.close (); //View HTML source files for different Web sites            returnHtml; }         Public stringGethtml (stringURL) {            returngethtml (URL, Encoding.UTF8); }

How C # crawls Web content using HttpWebRequest, HttpWebResponse emulation browser