Python Compress

Large flow of log if the direct write Hadoop to Namenode load, so the merge before storage, you can each node log together into a file to write HDFs.  It is synthesized on a regular basis and written to the HDFs. Let's look at the size of the log, 200G DNS log files, I compress to 18G, if you can use Awk Perl, of course, but the processing speed is certainly not distributed as the force. Hadoop Streaming principle Mapper and reducer ...

In the past Client-server, RPC framework hierarchies such as CORBA and RMI did not seek because such technologies could extend a stand-alone IPC (inter-process communication, interprocess communication) to communication between multiple computers, This is very helpful for extensibility, but for a variety of reasons these RPC frameworks have not been adopted by the industry on a large scale. In the era of cloud computing, more and more machines are needed for distributed communications, although they can be easily communicated by using the HTTP protocol.

First of all: Hadoop is disk-level computing, when computing, data on disk, need to read and write disk; http://www.aliyun.com/zixun/aggregation/13431.html ">storm is a memory-level calculation, Data imports memory directly over the network. Read/write memory is faster n order of magnitude than read-write disk. According to the Harvard CS61 Courseware, disk access latency is about 75,000 times times the latency of memory access. 　　So storm faster. ...

Preface The construction of enterprise security building Open source SIEM platform, SIEM (security information and event management), as the name suggests is for security information and event management system, for most businesses is not cheap security system, this article combined with the author's experience describes how to use Open source software to build enterprise SIEM system, data depth analysis in the next chapter. The development of SIEM compared Gartner global SIEM rankings in 2009 and 2016, we can clearly see that ...

According to foreign media reports, Juniper Network company in December last year to acquire 176 million U.S. dollars Software definition network (sdn,software tabbed receptacle) contrail Bae, and before that contrail company is not known to many people. Juniper Network launched its own SDN plan one months later, and released the beta code this May. Now that part of the code is ready for a formal launch, the Juniper Network has announced that it will be available to users in open source licensing mode. ...

You may not realize it, but the significance of the data is no longer limited to the key elements of the computer system--the data has been scattered across the field, becoming the hub of the world. Citing the comments from a managing director at JPMorgan Chase, the data have become "the lifeblood of the business". He threw his remarks at an important technical conference recently held, with data as the main object of discussion, and the meeting also gave an in-depth analysis of the ways in which institutions move to the "data-driven" path. The Harvard Business Review magazine says "data scientists" will be "21 ...