Sharpziplib Data Compression

After reading a post on Silverlight forum to Improve the Performance of WCF, I think it is quite useful. Obviously for distributed applicationsProgram, It seems that nothing is more important than improving the client response speed. After that, I searched for the relevant materials and made some exercises. Of course, I also saw the PDF of Li Bo about improving web service performance.

1. datasetsurrogate accelerates WebService: For datasetsurrogate, I checked msdn to display the namespace of this class in Microsoft. synchronization. data, but in. net Framework 3.5sp1 this namespace does not have this class. ms in. the datasetsurrogate class from net 1.1 seems to have been abandoned. however, you can still download datasetsurrogate from the MS website.Code, You need to compile and generate the DLL yourself.

2. When using Web Services, you are often used to returning dataset. However, dataset has a remotingformat attribute, which can be set to serializationformat. XML and serializationformat. the difference between binary and binary is the serialization depth. I understand that the XML format is redundant in the morning. The binary format is highly efficient in transmission, but it will increase the data volume, this affects performance, but data can be encrypted and transmitted when converted to binary. therefore, compression is very important to take advantage of binary transmission. the test results are as follows:

In this test, two million pieces of data are read from the database and the following measures are taken:

Getcompressbytes: reads data to dataset. After binary serialization, Gzip compression is used to return

Getdatasetbytes: reads data to dataset and serializes data in binary format (instead of setting the value of remotingformat ).

Getdatasetbytes: reads data to dataset and returns

From the above, it seems that the results are not as mentioned above. My explanation is that the test time starts from the call of getcompressbytes to the end. The intermediate time includes binary serialization and compression, what I care about is the transmission process. compression is a time-consuming process.

3. sharpziplib

Gzip compression is a compression in the. NET system. Io namespace.AlgorithmHowever, I am more concerned about the sharpziplib open-source compression framework, which supports zip, Gzip, tar, and bzip2. let's take a look at the compression ratio of sharpziplib and gzip and the duration of the entire process:

Getcompressbytes: compress legth: 751893, time: 00: 00: 02: 1001201 (. NET built-in gzip) compression rate 22%

Sharpziplib: compressbytes: 583020 uncompress Bytes: 3339243 time: 00: 00: 00: 6660381 (maximum compression rate) 17%

Obviously, the size of the 2w data records in the same dataset is the same as that in binary format. The compression ratio is not much different, but the transmission time is almost three times. Obviously, the above results are only for the text type, and the compression of PDF files is not obvious. For RAR files, zip files cannot be compressed, this is because either compression method is based on the Harman algorithm. (Of course, this statement is not very strict. Below are some codes

[Webmethod (description = "3. Use the binary sequence of the DataSet object, compress the object with compress (compression inherent in IIS), and return the result")]
Public byte [] getcompressbytes ()
{
Dataset DS = gettestdata ();
// Byte [] buffer = gettestdata (filename );
/*----*/
Binaryformatter fmater = new binaryformatter ();
Memorystream MS = new memorystream ();
Fmater. serialize (MS, DS );
/*----*/
Byte [] buffer = Ms. toarray ();
Return dataformatter. getbinaryformatdatacompress (buffer );
}

Public static byte [] getbinaryformatdatacompress (byte [] binarydataresult)

{
Return compress (binarydataresult );
}
/// <Summary>
/// Compress data
/// </Summary>
/// <Param name = "data"> </param>
/// <Returns> </returns>
Public static byte [] compress (byte [] data)
{
Byte [] bdata;
Memorystream MS = new memorystream ();
Gzipstream stream = new gzipstream (MS, compressionmode. Compress, true );
Stream. Write (data, 0, Data. Length );
Stream. Close ();
Stream. Dispose ();
// You must disable the stream to return Ms streaming data. Otherwise, the data is incomplete.
// And the decompression method stream. Read (buffer, 0, buffer. Length) will return 0
Bdata = Ms. toarray ();
Ms. Close ();
Ms. Dispose ();
Return bdata;
}

Sharpziplib compression:

[Webmethod (description = "4. Use the binary sequence of the list object and compress it with icsharpcode. sharpziplib.")]
Public byte [] getdatalistsharpzipcompressbytes ()
{
// Byte [] buffer = gettestdata (filename );
Ilist <users> List = gettestdatalist ();
/*----*/
Binaryformatter SER = new binaryformatter ();
Memorystream MS = new memorystream ();
Ser. serialize (MS, list );
Byte [] buffer = Ms. toarray ();
/*----*/
Byte [] zipbuffer = new compressionhelper (compressionlevel. bestspeed). compresstobytes (buffer );
// Ms. toarray ();
Return zipbuffer;
}

Public byte [] compresstobytes (byte [] bytestocompress)
{
Memorystream MS = new memorystream ();
Stream S = getoutputstream (MS );
S. Write (bytestocompress, 0, bytestocompress. Length );
S. Close ();
Return Ms. toarray ();
}

Of course, in actual applications, after we compress the data, we usually need to decompress the data on the client, deserialize it, and most of the time the data is transmitted instead of dataset, but list <t> or a single object, or some value types.

Buffer = client5.getdatalistsharpzipcompressbytes (); // obtain the compressed list through WCF
Buffer = new compressionhelper (compressionlevel. bestcompression). decompresstobytes (buffer); // Extract
Ilist <webservicetest. Users> List = dataformatter. retrievedatalist (buffer) as ilist <webservicetest. Users>; // convert it to the list actually used <users>

I talked to a senior colleague about the WCF Data Compression topic. He raised a question: in practical applications, for compressing blob (Big Data) it is worth considering that it is actually a very CPU resource-consuming task and does not support multi-core optimization. That is to say, no matter how many cores your server is, it is still a single core, therefore, the server memory and CPU are both a great test. However, it is easy to understand that data compression consumes CPU resources. I read a MB PDF file and use sharpziplib1 to compress (ZIP) the CPU. The memory usage is soaring, which is the same as using WinRAR directly, however, the CPU is not a single-core operation as described by colleagues.