HDFS scribe integration []

It is finally here: you can configure the Open Source log-aggregator, scribe, to log data directly into the hadoop distributed file system.

Compile Web 2.0 companies have to deploy a bunch of costly filers to capture weblogs being generated by their application. currently, there is no option other than a costly filer because the write-rate for this stream is huge. the hadoop-scribe integration allows this write-load to be distributed among a bunch of commodity machines, thus cing the total cost of this infrastructure.

The challenge was to make HDFS be real-timeish in behaviour. scribe uses libhdfs which is the c-interface to the HDFS client. there were varous bugs in libhdfs that needed to be solved first. then came the filesystem API. one of the major issues was that the filesystem API caches filesystem handles and always returned the same filesystem handle when called from multiple threads. there was no reference counting of the handle. this caused problems with scribe, because scribe is highly multi-threaded. A new API filesystem. newinstance () was introduced to support scribe.

Making the HDFS write code path more real-time was painful. there are varous timeouts/settings in HDFS that were hardcoded and needed to be changed to allow the application to fail fast. at the bottom of this blog-post, I am attaching the settings that we have currently configured to make the HDFS-write very real-timeish. the last of the Jiras, HADOOP-2757 is in the pipeline to be committed to hadoop trunk very soon.

What about namenode being the single point of failure? This is acceptable in a warehouse type of application but cannot be tolerated by a realtime application. scribe typically aggregates click-logs from a bunch of webservers, and losing * All * Click Log Data of a website for a 10 minutes or so (minimum time for a namenode restart) cannot be tolerated. the solution is to configure two overlapping clusters on the same hardware. run two separate namenode S N1 and N2 on two different machines. run one set of datanode software on all slave machines that report to N1 and the other set of datanode software on the same set of slave machines that report to N2. The two datanode instances on a single slave machine share the same data directories. this configuration allows HDFS to be highly available for writes!

The highly-available-for-writes-HDFS configuration is also required for software upgrades on the cluster. we can shutdown one of the overlapping HDFS clusters, upgrade it to new hadoop software, and then put it back online before starting the same process for the second HDFS cluster.

What are the main changes to scribe that were needed? Scribe already had the feature that it buffers data when it is unable to write to the configured storage. the default scribe behaviour is to replay this buffer back to the storage when the storage is back online. scribe is configured to support no-buffer-replay when the primary storage is back online. scribe-HDFS is configured to write data to a cluster N1 and If n1 fails then it writes data to cluster N2. scribe treats N1 and N2 as two equivalent primary stores.

From: http://hadoopblog.blogspot.hk/2009/06/hdfs-scribe-integration.html