Compression and decompression during transmission of large data volumes for WebService

When accessing webserivice, if the data volume is large, the data transmission will be slow. To increase the speed, we will think of compressing the data. First, let's analyze it.

When data is transmitted in webserice, dataset is generally used for data transmission. The execution process is to first convert dataset to XML for transmission. The format of dataset to XML is as follows:

<DataSetName>  <DataTableName>    <Column1Name>.......</Column1Name>    <Column2Name>.......</Column2Name>    <Column3Name>.......</Column3Name>  </DataTableName>.........<DataSetName>

Obviously, datase adds a large amount of XML format data when T is converted to XML, which increases the transmission volume.

After analysis, we can find two ways to solve the problem of large data transmission volume:

1. Do not directly use dataset to transmit data, and avoid additional data added when the data is converted to XML. Therefore, we can convert dataset into datasetsurrogate objects to be serialized using binary and transmit data using binary data. Of course, you can also use other better methods. In short, it is to reduce the additional data added for transmission.

2. compress the data before transmitting it. There are many compression methods. For details, refer to my article. Net's research on compression and decompression.

 

Refer to the following code (gzip of. NET is used for compression, and the compression efficiency may not be very good ):

// ================================================ ==================================================================/// Class Name: datasetzip // <summary> /// when the data volume in dataset is large, network data transmission will be slow. /// This class converts dataset to datasetsurrogate object and uses binary for serialization. /// then compress the object and transmits it, finally, extract the data. /// </Summary> /// <remarks> // convert or restore the data in the able in the dataset. /// </remarks>/* = ========================================================== ============================== change record No. Update date developer change content 001 2008/7/22 new sheets ======================================== =========================================== */public class datasetzip {// Message ID private const string msg_err_internal = "mfwe00016 "; /// <summary> // get and convert dataset to datasetsurrogate object. Use binary for serialization, decompressed binary array /// </Summary> /// <Param name = "dsdata"> dataset data to be compressed </param> /// <returns> post-binary array </returns> Public static byte [] getdatasetzipbytes (Dataset dsdata) {try {datasetsurrogate DSS = new datasetsurrogate (dsdata); binaryformatter SER = new binaryformatter (); memorystream MS = new memorystream (); Ser. serialized (MS, DSS); byte [] buffer = Ms. toarray (); byte [] zipbuffer = compress (buffer); Return zipbuffer;} catch (exception ex) {Throw new datasetconverterexception (msg_err_internal, new string [] {"datasetzip ", "getdatasetzipbytes"}, ex, null) ;}/// <summary>.. Net built-in gzip to compress binary arrays, the compression ratio may not be too good /// </Summary> /// <Param name = "data"> binary array </param> /// <returns> the compressed binary array </returns> Public static byte [] compress (byte [] data) {memorystream MS = new memorystream (); stream zipstream = NULL; zipstream = new gzipstream (MS, compressionmode. compress, true); zipstream. write (data, 0, Data. length); zipstream. close (); Ms. position = 0; byte [] compressed_data = new byte [Ms. length]; Ms. read (compressed_data, 0, Int. parse (Ms. length. tostring (); Return compressed_data ;} /// <summary> /// decompress the binary array /// </Summary> /// <Param name = "data"> binary array </param> // /<returns> decompressed dataset </returns> Public static dataset decompress (byte [] data) {try {byte [] buffer = NULL; memorystream zipms = new memorystream (data); buffer = etractbytesformstream (zipms, Data. length); binaryformatter SER = new binaryformatter (); datasetsurrogate DSS = Ser. deserialize (New memorystream (buffer) as datasetsurrogate; dataset dsdata = DSS. converttodataset (); Return dsdata;} catch (exception ex) {Throw new datasetconverterexception (msg_err_internal, new string [] {"datasetzip", "decompress"}, ex, null) ;}/// <summary> // use. the built-in gzip of net decompress the data stream /// </Summary> /// <Param name = "zipms"> data stream </param> /// <Param name = "datablock "> Data Length </param> // <returns> decompressed binary array </returns> Public static byte [] etractbytesformstream (memorystream zipms, int datablock) {byte [] DATA = NULL; int totalbytesread = 0; stream zipstream = NULL; zipstream = new gzipstream (zipms, compressionmode. decompress); While (true) {array. resize (ref data, totalbytesread + datablock + 1); int bytesread = zipstream. read (data, totalbytesread, datablock); If (bytesread = 0) {break;} totalbytesread ++ = bytesread;} array. resize (ref data, totalbytesread); return data ;}}