Msgpack [C ++] usage notes and msgpack/cpickle Performance Comparison

Install msgpack in python:

Wget http://pypi.python.org/packages/source/m/msgpack-python/msgpack-python-0.1.9.tar.gz

Python2.x setup. py install -- prefix =/usr/local/similarlib/

The msgpack version of python is usually easy to use, and the speed is faster than the built-in pickle and cpickle of python. The use of C ++ version is troublesome. Below is a demo of my learning, parse a complex Dictionary of python-msgpack dump.

#include <msgpack.hpp>#include <fstream>#include <iostream>using namespace std;template <class T>void msgunpack(const char* binary_file, T& t, char* buff, uint32_t max){msgpack::unpacked msg;ifstream tf_file(binary_file,ios::in|ios::binary|ios::ate);uint32_t size = tf_file.tellg();tf_file.seekg(0, ios::beg);tf_file.read(buff, size);tf_file.close();msgpack::unpack(&msg, buff, size);msg.get().convert(&t);}typedef map<uint32_t, uint32_t> WordsMap;typedef map<uint32_t, WordsMap> FieldsMap;typedef map<uint64_t, FieldsMap> DocsMap;int main(int argc, char** argv){uint32_t MAX_BUFF = 1024*1024*100; //100MBchar* BUFF = new char[MAX_BUFF];DocsMap docsMap;msgpack::unpacked msg;msgunpack("/data/wikidoc/tf_dict_for_nodes/1-1000", docsMap, BUFF, MAX_BUFF);//        msg.get().convert(&docsMap);cout << docsMap.size() << endl;        delete[] BUFF;}

Refer:

Http://wiki.msgpack.org/pages/viewpage.action? PageId = 1081387 # QuickStartforC % 2B % 2B-ImplementationStatus

Below is a self-encapsulated msgpack interface header file mymsgpack. h

# Ifndef my_msgpack_h # define my_msgpack_h # include <fstream> # include <msgpack. HPP> using namespace STD; Template <class T> void load_from_file (const char * binary_file, T & T) {ifstream binaryfstream (binary_file, IOS: In | IOs :: binary | IOs: ATE); uint32_t size = binaryfstream. tellg (); char * buff = new char [size]; binaryfstream. seekg (0, IOS: Beg); binaryfstream. read (buff, size); binaryfstream. close (); msgpack: unpacked MSG; msgpack: unpack (& MSG, buff, size); MSG. get (). convert (& T); Delete [] Buff;} template <class T> void load_from_str (const char * binary_str, int Len, T & T) {msgpack: unpacked MSG; msgpack: unpack (& MSG, binary_str, Len); MSG. get (). convert (& T) ;}template <class T> void dump_to_file (T & T, const char * dump_file) {msgpack: sbuffer sbuf; msgpack: Pack (sbuf, t); ofstream dumpfstream (dump_file, IOS: Out | IOs: Binary | IOs: trunc); dumpfstream. write (sbuf. data (), sbuf. size (); dumpfstream. close ();} template <class T> void dump_to_str (T & T, char ** dump_str, Int & Len) {// external release * dump_strmsgpack: sbuffer sbuf; msgpack:: Pack (sbuf, T); Len = sbuf. size (); * dump_str = (char *) malloc (sbuf. size (); memcpy (* dump_str, sbuf. data (), sbuf. size () ;}# endif

 

Msgpack is compiled and cannot be linked. undefined reference to '_ sync_sub_and_fetch_4'

The problem is normal on x84_64 and occurs on 32-bit machines.

[xudongsong@BigServerU-4 msgpack-0.5.7]$ cat /etc/issueCentOS release 5.4 (Final)Kernel \r on an \m[xudongsong@BigServerU-4 msgpack-0.5.7]$ file /sbin/init/sbin/init: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.6.9, dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped

./Configure: no error is reported, but an error is displayed when you view config. log. An error is returned when the library of msgpack is linked to the program.

Cause: gcc does not recognize the CPU system and needs to be manually specified

[xudongsong@BigServerU-4 msgpack-0.5.7]$ CFLAGS="-march=pentium -mtune=pentium" ./configure --prefix=/home/xudongsong/msgpack_static --enable-static=yes --enable-shared=no

Make, make install

[Xudongsong @ BigServerU-4 jobs] $ g + + response-o job_calc_weight-I/home/xudongsong/msgpack_static/include/-L/home/xudongsong/msgpack_static/lib/-lmsgpack

Pass!

 

 

The following is a demo program for msgpack and cPickle to perform performance pk (pickle is not compared because it is slower than cPickle, as described in Python cook book ):

mport sys,time,msgpack,pickle,cPickle,randomtest_list = []i = 0while i<100000:test_list = random.randrange(1,100000)i += 1print "common len(serialize) = %s"%len(cPickle.dumps(test_list,0))print "compress len(serialize) = %s"%len(cPickle.dumps(test_list,1))#------------------------------------------------------------------------results = {}time_start = time.time()for i in range(1,1000000):        results[i] = cPickle.dumps(test_list,1)time_mid_1 = time.time()print "cPickle dumps eats %s s"%str(time_mid_1-time_start)for i in range(1,1000000):cPickle.loads(results[i])time_mid_2 = time.time()print "cPickle loads eats %s s"%str(time_mid_2-time_mid_1)#------------------------------------------------------------------------results = {}time_start = time.time()for i in range(1,1000000):results[i] = msgpack.dumps(test_list)time_mid_1 = time.time()print "msgpack pack eats %s s"%str(time_mid_1-time_start)for i in range(1,1000000):msgpack.loads(results[i])time_mid_2 = time.time()print "msgpack unpack eats %s s"%str(time_mid_2-time_mid_1)