Mapreduce

At present, a large number of new technologies are emerging every year in the big data field, which is an effective means for big data, storage, processing analysis or visualization.

In recent years, big data has changed from the popular words and concepts unique to large companies to the driving force behind the development of our digital life. The following are five trends in the processing and development of big data in the future.

In order to meet the ever-changing business changes, Jingdong's Jingmai team has adopted a popular open source big data calculation engine such as Hadoop on the basis of Jingdong Big Data Platform to create a decision-making data product for JD operations and products.

Hadoop is more suitable for solving big data problems, and relies heavily on its big data storage system, namely HDFS and big data processing system. For MapReduce, we know a few questions.

MapReduce in Hadoop is a simple software framework based on which an application can run on a large cluster of thousands of commercial machines and process terabytes of data in parallel with a reliable fault tolerance.

MapReduce is a programming model for parallel computing of large-scale data sets (greater than 1TB) to solve the computational problems of massive data.

This article briefly describes the execution steps and workflow of the mapreduce programming model in the form of graphics, which is simple and easy to understand.

Earlier we used HDFS for related operations, and we also understood the principles and mechanisms of HDFS. With a distributed file system, how do we handle files? This is the second component of Hadoop-MapReduce.

Hadoop MapReduce is a programming model for data processing that is simple but powerful enough to be designed for parallel processing of big data.

In MapReduce, shuffle is more like the inverse process of shuffling, which refers to "disrupting" the random output of the map end according to the specified rules into data with certain rules so that the reduce end can receive and process it.