Decrypt the Gzip compressed Web page data stream (GO)

The page data cannot be retrieved because a webpage is experiencing problems.

After a few troubleshooting, found that the site will check the client header information, if you encounter an ambiguous header information is directly negative, return 0 data.

If the header information is correct, the gzip-compressed data is returned, so the idea of getting the Web data directly is dashed.

As a matter of fact. NET has encapsulated the GZIP and deflate-decompression algorithm classes for us, all in the System.IO.Compression namespace.

Let me show you how to parse and extract the gzip-compressed web data stream.

Imports System.Net
Imports System.IO
Imports System.IO.Compression
Imports System.Text



Function gzip2html (ByVal url asstring) asstring

' HTTP request
Dim req as HttpWebRequest = httpwebrequest.create (URL)
Req. Method = "Get" ' Get ' or POST
Req. Accept = "*/*"
Req. useragent = "mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;) "' Here simulates the request field of the XP system IE7
Req. Referer = "Source Address"

' A simulation of some header information
Req. Headers ("accept-language") = "ZH-CN"
Req. Headers ("ua-cpu") = "x86"
Req. Headers ("accept-encoding") = "gzip, deflate"

' HTTP GET
Dim res as HttpWebResponse = req. GetResponse
' Get Data text encoding
' Dim enc as Encoding = encoding.getencoding (res. CharacterSet) ' If there is garbled, please change the following way
Dim enc as Encoding = Encoding.default

' Create a gzip decompression stream
Dim GZ asnew GZipStream (res. GetResponseStream, Compressionmode.decompress)
' Use a temporary memory stream to save the extracted data
Dim Ms Asnew MemoryStream
' Buffered data
Dim buf (asbyte), I asinteger=0
' Constantly extracting data from the stream
Whiletrue
i = GZ. Read (buf, 0, 100)
If I =0thenexitwhile
Ms. Write (buf, 0, i)
Endwhile
' Convert data to Characters
DIM ret asstring= Enc. GetString (Ms. ToArray)

' Close all the streams
Gz. Close ()
Ms. Close ()
Res. Close ()

Return ret
End Function

I test the decompression data successfully, other environment is not tested, please modify it yourself.

Decrypt the Gzip compressed Web page data stream (GO)