A large data processing algorithm for energy efficient in heterogeneous cluster

A large data processing algorithm for energy efficient in heterogeneous cluster

Ding, Qin Xiaolin, Liuliang, Wang Tao

The energy consumption of the cluster has exceeded the cost of its own hardware acquisition, while large scale data processing requires a large amount of time, so how to carry out the high energy and efficient data processing is a problem to be solved by the users and consumers. is also a huge challenge to energy and the environment. Existing research generally reduces energy consumption by shutting down some nodes or design new data storage strategies to implement energy efficient data processing. Through analysis, it is found that even with the fewest nodes there is a great waste of energy, and the new data storage strategy for the already deployed cluster will cause large-scale data migration, the use of additional energy. Needles For large I/O intensive data processing tasks under heterogeneous clusters, a new energy efficient algorithm is proposed, which divides the minbalance into two steps of node selection and load balancing. In the node selection phase, 4 different greedy strategies are adopted to fully consider the heterogeneity of the nodes, so as to select the most suitable node for task processing. Load balancing of selected nodes in load balancing phase, to reduce the amount of energy wasted by waiting for each node. This method is universal and is not affected by the data storage strategy. Experiments show that minbalance method can reduce more than 60 in the case of large data sets compared with traditional closed part nodes. % of energy consumption.

A large data processing algorithm for energy efficient in heterogeneous cluster