Mongospy, Mongowatch and MongoDB data compression

From an article in the Openmymind blog, the author introduces two MongoDB monitoring gadgets, Mongospy and Mongowatch, which they write with Node.js, and then puts forward the idea of using compression to save space when storing text in MongoDB.

These two gadgets are not very powerful and simple to implement, and if you can manipulate MongoDB in any language, believe you can write a similar thing.

Mongospy: A small tool for monitoring mongodb slow queries, Use Node.js to read the latest data from MongoDB's System.profile collection and display it on a Web page so you can see MongoDB's slow queries in real time. Of course, the premise is that you have to configure the DB profile level. Concrete visible: "Mongo Database Profiler"

Mongowatch: This script is timed to get MongoDB state information and is displayed in chronological order on the Web page so you can see directly the changes in the status metrics you MongoDB over time.

Above said two gadgets, if you are interested you can take a look at the specific usage, the following section proposes the idea of compressing MongoDB text data using compression method.

MongoDB is a document-type database with data that uses Bson structure storage, Bson supports text-type data storage, and also supports binary data storage. Imagine that if the Bson structure would compress strings and store them in binary form, that would save a lot of space.

But there is a premise that you will not be able to query the compressed fields indexed. In fact, this can be guaranteed in many cases, such as we use MONGODB Store blog information, is not to do query operations.

Unfortunately, MongoDB does not provide such functionality, although this feature has long been proposed (SERVER-164).

Since MongoDB is not supported, then we can consider the implementation at the client, the current compression algorithm is countless, depending on your use of language support. such as Google's snappy or messagepack and so on. In fact, this idea is not new, and in some storage that does not provide a level two index, such as memcached, we have already used a similar approach to conserve memory.

For example, you can compress some plain text data into binary, and then save a compressed type of field for identification, as follows:

The top three of the above data are compressed using a Type 1 compression algorithm to compress data in binary data, and fourth is not compressed. The specific situation can be compressed by the compression algorithm in different lengths of data compression rate to adjust, such as only two characters of the value, you may not need to compress, this time you set the type to 0 means not to do compression.

(Responsible editor: Lu Guang)