談Hadoop的C++擴充

原文在 http://blog.sina.com.cn/s/blog_6e273ebb0100pid0.html

 

長期一來，Hadoop因為其Java實現帶來的效能問題而飽受爭議，同時也湧現了很多方案來緩解這一問題。


Jeff Hammerbacher（Cloudera首席科學家）曾在Quora上寫過這樣一段：

-------------------------------------------------------------------------------------------------------------------------------------

Doug's newest project, Avro [1], will allow for cross-language serialization and RPC. If you think individual components of Hadoop could be implemented more efficiently in another language, you'll be welcome to try your hand once the migration to Avro for RPC
[2] is complete.


In my experience, distributed systems should focus on reliable performance under stress, horizontal scalability, and ease of debugging before optimizing for efficiency. Matt Welsh does a great job of highlighting this issue in his retrospective on SEDA [3].
Sean Quinlan of Google mentions a similar policy at Google, noting that "it's atypical of Google to put a lot of work into tuning any one particular binary." [4] Java has advantages and disadvantages along these dimensions, but I'll leave that for others to
discuss.


For HDFS in particular, libhdfs [5] implements a C API to HDFS by communicating with Java over JNI. Using libhdfs and FUSE, one can mount HDFS just like any other file system [6]. Once Avro is in place, the client could be implemented in C and placed in the
kernel to make this process even smoother and more efficient. Currently it's not the most pressing issue in Hadoop development.


For Hadoop MapReduce, you can use Hadoop Streaming to write your MapReduce logic in any language, or Hadoop Pipes [7] if you want a C++-specific API. If you can't wait for Avro, there's also the "Hadoop C++ Extension" [8] from Baidu which implements the Task
execution environment in Hadoop in C++, and appears to provide moderate performance gains.


[1] http://avro.apache.org

[2] https://issues.apache.org/jira/browse/HADOOP-6659

[3] http://matt-welsh.blogspot.com/2010/07/retrospective-on-seda.html

[4] http://queue.acm.org/detail.cfm?id=1594206

[5] http://hadoop.apache.org/common/docs/current/libhdfs.html

[6] https://wiki.cloudera.com/display/DOC/Mountable+HDFS

[7] http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/pipes/package-summary.html

[8] https://issues.apache.org/jira/browse/MAPREDUCE-1270

-------------------------------------------------------------------------------------------------------------------------------------



百度在使用Hadoop過程中同樣發現了Hadoop因為Java語言帶來的低效問題，並對Hadoop進行擴充。


而在此之前，百度也嘗試了 Hadoop PIPES 和 Hadoop Streamming，但是發現這些問題：

- 這兩種方案都無法很好地控制Child JVM（Map TaskTracker和Reduce TaskTracker）記憶體的使用，這部分都由JVM自己控制，而你能做的就只是使用-Xmx設定記憶體限制而已；

- 這兩種方案都只能影響到Mapper和Reducer回呼函數，而真正影響效能的sort和shuffle過程依然在Java實現的TaskTracker中執行完成；

- 資料流問題。兩種方案中，資料處理流都必須由TaskTracker流向Mapper或者Reducer然後再流回來。而無論是使用pipeline還是socket方式傳遞資料，都難以避免資料的移動。對於大規模資料處理，其代價是不可忽視的。

究其根本，實際上是C++模組所承擔的邏輯太少。於是百度提出了更徹底的方案，即"Hadoop C++ Extention"，該方案中C++代碼對Hadoop入侵得更多。它將原來TaskTracker中完成的資料處理工作都交給C++模組去完成，而只讓其負責協議通訊和控制等。如此一來，上面的問題就都解決了：

- TaskTracker JVM只負責少量通訊工作，其記憶體需求很小並且可以預見，從而容易控制，譬如設為-Xmx100m就足夠了；

- sort和shuffle過程都使用C++模組實現，效能得到提高；

- 資料在其整個生命週期都只在C++模組中，避免不必要的移動。

這就猶如將C++模組的戰線往前推進了。當然，也許在很多人看來，這隻是五十步與百步的區別，但是這多出來的五十步，卻正是效能瓶頸所在。