Concurrent Programming Framework-flumejava

Based on map-shffle-Reduce, some Google developers have also implemented some useful parallel computing programming frameworks. flumejava is one of them.

Http://faculty.neu.edu.cn/cc/zhangyf/cloud-bigdata/papers/big%20data%20programming/FlumeJava-pldi-2010.pdf

From the introduction, the starting point of flumejava should be that map-reduce job is not that easy to write, and it will be more troublesome if a task requires a lot of Map-Reduce processes. It is not easy to optimize a series of Map-reduce tasks.

Flumejava makes concurrent programming easier based on the design. It integrates map-shuffle-combine-reduce as a meta element and provides some basic operation functions, such as Count, group, join, this makes some common programming tasks easier.

When a user has a series of concurrent tasks to run together, flumejava will also try to optimize the solution by using the classic loop fusion optimization, there are also push down condition optimizations used by traditional databases. The core of optimization is to reduce file IO, reduce repetitive tasks, and reduce communication.

Experiments show that flumejava'sCodeThe number of jobs is more than 20% less than that of common map-reduce jobs, and the execution efficiency is close to that of Map-reduce jobs after manual optimization.

On the whole, flumejava is mainly a engineering problem. Developers consider how to perform concurrent programming more simply, rather than simply proposing a new model and new concept, this pragmatic spirit is Google's consistent style.

I am interested in the fact that flumejava developers have developed another concurrent programming framework called lumberjack. It has proved that this tool is difficult to use, as a result, flumejava was born and won the favor of many developers. Compared with the reality, some developers only focus on advanced code rather than users, which is much higher.