Research on lightweight MapReduce model based on embedded multiprocessor

Research on lightweight MapReduce model based on embedded multiprocessor

With the rapid development of cloud computing and the rapid growth of multi-core processor core, the application of parallel development technology is more and more common. A variety of parallel development technologies such as MPI, OpenMP and so on have been very mature applications in various fields, and Google's MapReduce programming model led to a lot of MapReduce model based development framework, such as Phoenix, Metis, Hadoop, etc. However, Phoenix and Metis are implemented based on shared memory architectures and cannot be used in distributed clusters, while the implementation of Hadoop cluster deployments in embedded environments is inefficient. So there is no mature distributed computing framework based on embedded platform, although the Mpi+openmp method is a more common method, but using OpenMP to control the parallelism increases the programming difficulty and the cost of development and maintenance.

In order to study the application defects of Hadoop in embedded processor platform, this paper firstly constructs a cloud computing platform based on embedded multiprocessor, realizes parallel image processing method on it, and analyzes the efficiency, then discovers the insufficiency of Hadoop application in embedded environment. Then, the Phoenix and Metis of MapReduce development framework under Multi-Core platform are studied, and the performance evaluation based on TILERA36 platform is used to discover the advantages and disadvantages of these two frameworks.

Finally, on the basis of the above research, a lightweight distributed computing framework Mpi+phoenix for embedded environment is redesigned, which mainly considers the heterogeneity between embedded processor nodes and how to maximize the efficiency of the system with the processor core inside the node. At last, the application and test of image parallel processing for the frame are carried out, and the Phoenix standard set test is carried out to evaluate the performance of the framework, and the test results show that the proposed framework can improve the parallelism of data processing and improve the execution efficiency of the program.

Research on lightweight MapReduce model based on embedded multiprocessor