Centralized HDFS cache management, which allows users to cache specific HDFS paths that are cached in out-of-heap memory. Namenode instructed Datanode to complete the work.
Centralized cache management in HDFS have many significant advantages.
- Explicit pinning prevents frequently used data from being evicted from memory. This was particularly important when the size of the working set exceeds the size of main memory, which was common for many HDFS workloads. Block frequently used data from being evicted from memory.
- Because DataNode caches is managed by the NameNode, applications can query the set of cached block locations when making Task placement decisions. Co-locating a task with a cached block replica improves read performance.
- When the block has been cached by a DataNode, the clients can use a new, more-efficient, Zero-copy read API. Since checksum verification of cached data is doing once by the DataNode, clients can incur essentially zero overhead when Using the this new API. You can read these blocks using a more efficient, non-replicated API.
- Centralized caching can improve overall cluster memory utilization. When relying on the OS buffer cache at each DataNode, repeated reads of a block would result in all nreplicas of T He block being pulled into buffer cache. With centralized cache management, a user can explicitly pin only m of the n replicas, saving n-m memory. Reduce the use of repeated reads
Source: http://hadoop.apache.org/docs/r2.6.4/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html
Applicable situation: often need to read the file. Like a small file.
Structure:
Datanode periodically sends cache block information through heartbeats to Namenode,namdenode to send the new cache path to the Datanode to let it cache.
Namenode will periodically review the namespace and cache lists to determine what needs to be cached and does not require caching, and the cache information will be updated in Fsimage and edit log. Note: Inconsistent blocks are not cached and the target object of the shortcut is not cached. Note: Only file and directory level caches are currently supported, and block levels are not supported. The directory supports only the first level of data caching in the directory, and does not support loops.
Commands and Configuration: http://hadoop.apache.org/docs/r2.6.4/hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html
In File reference: http://blog.csdn.net/colzer/article/details/24786283
From for notes (Wiz)
11: Centralized cache Management in HDFS centralized caching management
