Node. js-based compression and decompression methods, node. js compression and decompression

Node. js-based compression and decompression methods, node. js compression and decompression

Compression format

Zip and gzip are two of the most common compression formats. Of course, gzip is rarely used in Windows.

Tar is an archive format, which is not compressed by default. You need to combine gzip to compress the final tar file into a tar.gz file in gzip format, which is usually abbreviated as tgz.

Why not mention rar? Because it is a patented algorithm, you can obtain the decompression tool for free, and the compression tool is charged. Therefore, in general application scenarios, rar compressed files are rarely provided.

This article will introduce how gzip, tar, tgz, and zip are compressed and decompressed under Node. js.

Uncompressed file library

The uncompressed file library used in this article comes from urllib. You need to clone it to the specified directory first.

Copy codeThe Code is as follows:
Git clone https://github.com/node-modules/urllib.git nodejs-compressing-demo

Gzip

In the Linux world, each tool has a pure and single responsibility. For example, gzip only compresses files. It does not matter how To Compress folders, that is what tar is responsible.

Gzip command line to compress a file

For example, if you want to compressing the nodejs-compressing-demo/lib/urllib. js file to gzip, you will get a urllib.js.gz file, and the source file will be deleted.

$ Ls-l nodejs-compressing-demo/lib/urllib. js-rw-r -- 1 a 31318 Feb 12 11: 27 nodejs-compressing-demo/lib/urllib. js $ gzip nodejs-compressing-demo/lib/urllib. js $ ls-l nodejs-compressing-demo/lib/urllib.js.gz-rw-r -- 1 a 8909 Feb 12 11: 27 nodejs-compressing-demo/lib/urllib.js.gz # restore Compression file $ gunzip nodejs-compressing-demo/lib/urllib.js.gz

The file size is reduced from 31318 bytes to 8909 bytes, and the compression effect exceeds 3.5 times.

You can also use pipe and cat commands to compress and save the file as any file:

$ ls -l nodejs-compressing-demo/README.md-rw-r--r-- 1 a a 13747 Feb 12 11:27 nodejs-compressing-demo/README.md$ cat nodejs-compressing-demo/README.md | gzip > README.md.gz$ ls -l README.md.gz-rw-r--r-- 1 a a 4903 Feb 12 11:50 README.md.gz

Node. js implements gzip

Of course, we will not really implement a gzip algorithm and tool from scratch. In the Node. js world, someone has already prepared these basic libraries for you, and we only need to use them out of the box.

This article uses the compressing module to compress and decompress all the code.

Why is compressing selected? Because it has sufficient code quality and unit test guarantee, and is in active maintenance status, the API is very friendly and supports stream interfaces.

Promise Interface

Const compressing = require ('compressing '); // select the gzip format, and then call compressFile compressing.gzip. compressFile ('nodejs-compressing-demo/lib/urllib. js', 'nodejs-compressing-demo/lib/urllib.js.gz '). then () => {console. log ('success ');}). catch (err => {console. error (err) ;}); // the decompression process is the response process. All interfaces are uncompresscompressing.gzip. uncompress ('nodejs-compressing-demo/lib/urllib.js.gz ', 'nodejs-compressing-demo/lib/urllib. js2 '). then () => {console. log ('success ');}). catch (err => {console. error (err );});

Combined with the programming model of async/await, code writing is a normal asynchronous io operation.

Const compressing = require ('compressing '); async function main () {try {await compressing.gzip. compressFile ('nodejs-compressing-demo/lib/urllib. js', 'nodejs-compressing-demo/lib/urllib.js.gz '); console. log ('success');} catch (err) {console. error (err);} // extract try {await compressing.gzip. uncompress ('nodejs-compressing-demo/lib/urllib.js.gz ', 'nodejs-compressing-demo/lib/urllib. js2 '); console. log ('success');} catch (err) {console. error (err) ;}} main ();

Stream interface

Note that when programming in Stream mode, you must handle the error events of each stream and manually destroy all streams.

Fs. createReadStream ('nodejs-compressing-demo/lib/urllib. js '). on ('error', handleError ). pipe (new compressing.gzip. fileStream () // It's a transform stream. on ('error', handleError ). pipe (fs. createWriteStream ('nodejs-compressing-demo/lib/urllib.js.gz 2 ')). on ('error', handleError); // extract, that is, the direction of pipe is reversed to fs. createReadStream ('nodejs-compressing-demo/lib/urllib.js.gz 2 '). on ('error', handleError ). pipe (new compressing.gzip. uncompressStream () // It's a transform stream. on ('error', handleError ). pipe (fs. createWriteStream ('nodejs-compressing-demo/lib/urllib. js3 ')). on ('error', handleError );

According to the official recommendation of Backpressuring in Streams, we should use the pump module to work with Stream mode programming to complete the cleaning of these Streams.

Const pump = require ('pump '); const source = fs. createReadStream ('nodejs-compressing-demo/lib/urllib. js'); const target = fs. createWriteStream ('nodejs-compressing-demo/lib/urllib.js.gz 2 '); pump (source, new compressing.gzip. fileStream (), target, err => {if (err) {console. error (err);} else {console. log ('success') ;}}); // decompress pump (fs. createReadStream ('nodejs-compressing-demo/lib/urllib.js.gz 2 '), new compressing.gzip. fileStream (), fs. createWriteStream ('nodejs-compressing-demo/lib/urllib. js3 '), err => {if (err) {console. error (err);} else {console. log ('success ');}});

Advantages of Stream APIs

The Stream interface seems much more complicated than the Promise interface. Why is there such an application scenario?

In fact, in the HTTP service field, the Stream model has a greater advantage, because the HTTP Request itself is a Request Stream. to compress and return an uploaded file with gzip, using the Stream interface, you do not need to save the uploaded file to a local disk, but directly consume the file Stream.

Using the sample code uploaded from the egg file, we can perform gzip compression and then return the code.

Const pump = require ('pump '); class UploadFormController extends Controller {//... other codes async upload () {const stream = await this. ctx. getFileStream (); // directly assign the compressed stream to ctx. body to achieve streaming response returned by side compression. this. ctx. body = pump (stream, new compressing.gzip. fileStream ());}}

Tar | gzip> tgz

You can see in gzip in advance that tar is responsible for packaging folders: package.

For example, to package the entire nodejs-compressing-dem o folder into a file and send it to others, you can run the tar command.

$ tar -c -f nodejs-compressing-demo.tar nodejs-compressing-demo/$ ls -l nodejs-compressing-demo.tar-rw-r--r-- 1 a a 206336 Feb 12 14:01 nodejs-compressing-demo.tar

As you can see, the files packaged by tar are generally large because they are uncompressed and the size is close to the total size of the actual folder. Therefore, we all compress the package at the same time.

$ tar -c -z -f nodejs-compressing-demo.tgz nodejs-compressing-demo/$ ls -l nodejs-compressing-demo.tgz-rw-r--r-- 1 a a 39808 Feb 12 14:07 nodejs-compressing-demo.tgz

The difference between tar and tgz exceeds 5 times, which can greatly reduce the network transmission bandwidth.

Node. js implements tgz

Promise Interface

First use compressing.tar. compressDir (sourceDir, targetFile) to package a folder into a tar file, and then use the gzip compression method above to compress the tar file into a tgz file.

Const compressing = require('compressing'your compressing.tar. compressDir ('nodejs-compressing-demo', 'nodejs-compressing-demo.tar '). then () => {return compressing.gzip.compressFile('nodejs-compressing-demo.tar ', 'nodejs-compressing-demo.tgz ');});. then () => {console. log ('success ');}). catch (err => {console. error (err) ;}); // decompress compressing.gzip. uncompress ('nodejs-compressing-demo.tgz ', 'nodejs-compressing-demo.tar '). then () => {return compressing.tar.uncompress('nodejs-compressing-demo.tar ', 'nodejs-compressing-demo2 ');});. then () => {console. log ('success ');}). catch (err => {console. error (err );});

Combined with the programming model of async/await, the code is easier to read:

Const compressing = require ('compressing '); async function main () {try {await compressing.tar. compressDir ('nodejs-compressing-demo', 'nodejs-compressing-demo.tar '); await it', 'nodejs-compressing-demo.tgz '); console. log ('success');} catch (err) {console. error (err);} // extract try {await compressing.gzip. uncompress ('nodejs-compressing-demo.tgz ', 'nodejs-compressing-demo.tar'); await compressing.tar.uncompress('nodejs-compressing-demo.tar ', 'nodejs-compressing-demo2'); console. log ('success');} catch (err) {console. error (err) ;}} main ();

Stream interface

Through the compressing.tar. Stream class, you can dynamically add any files and folders to a tar stream object, which is very flexible.

const tarStream = new compressing.tar.Stream();// dirtarStream.addEntry('dir/path/to/compress');// filetarStream.addEntry('file/path/to/compress');// buffertarStream.addEntry(buffer);// streamtarStream.addEntry(stream);const destStream = fs.createWriteStream('path/to/destination.tgz');pump(tarStream, new compressing.gzip.FileStream(), destStream, err => { if (err) {  console.error(err); } else {  console.log('success'); }});

Zip

Zip can actually be seen as a "commercial" combination of tar + gzip. It makes it unnecessary for users to distinguish whether it is a compressed file or a compressed folder. I will use zip anyway.

Example of compressing a folder using the zip command line tool:

$ zip -r nodejs-compressing-demo.zip nodejs-compressing-demo/ adding: nodejs-compressing-demo/ (stored 0%) adding: nodejs-compressing-demo/test/ (stored 0%) ... adding: nodejs-compressing-demo/.travis.yml (deflated 36%)$ ls -l nodejs-compressing-demo.*-rw-r--r-- 1 a a 206336 Feb 12 14:06 nodejs-compressing-demo.tar-rw-r--r-- 1 a a  39808 Feb 12 14:07 nodejs-compressing-demo.tgz-rw-r--r-- 1 a a  55484 Feb 12 14:34 nodejs-compressing-demo.zip

By comparing the size of the tgz and zip files, we can see that gzip is better than zip under the default compression parameter.

Node. js implementation zip

The implementation code is similar to tar, but it is compressed by default and does not need to be added to gzip.

Const compressing = require('compressing'your compressing.zip. compressDir ('nodejs-compressing-demo', 'nodejs-compressing-demo.zip '). then () => {console. log ('success ');}). catch (err => {console. error (err);}); // decompress compressing.zip.uncompress('nodejs-compressing-demo.zip ', 'nodejs-compressing-demo3 '). then () => {console. log ('success ');}). catch (err => {console. error (err );});

Summary

Is compression and decompression Based on Node. js easier than imagined? Thanks to the npm giant, we can have a simple experience with command line tools for programming.

Whether it is the Promise interface or the Stream interface, it has the most suitable scenario. will you choose?

So far, what services and functions can you provide with the compression and decompression capabilities?

The above is all the content of this article. I hope it will be helpful for your learning and support for helping customers.