Fast import of massive amounts of data through a blukload approach

Summary

There are several ways to load data into hbase, either through the HBase API Import or command line import or by using a third party (such as Sqoop) to import or use Mr for bulk import (disk I/O is an easy way to use node downtime during the import process). But these methods are not slow is the import process occupies the region data resulting in inefficient, today is to use hbase in the HDFs storage principle and MapReduce features to quickly import massive amounts of data How is hbase data stored under HDFS? Each table in HBase is stored with a folder under the root directory (/hbase), the table name is a folder name, and each region in the table folder is also stored with a folder, and each column family under each region folder is stored with a folder. And each column family is stored under some hfile file, hfile is the hbase data in Hfds storage format, the overall directory structure is as follows:/hbase/<tablename>/<encoded-regionname>/ <column-family>/<filename> hbase Data Write path(Figure from Cloudera) in the put data will be the data update operation information and data information written to the Wal, after writing to the Wal, the data will be put into the memstore, when the Memstore full data will be flushed to the disk (that is, the formation of hfile files), In this process involved in the flush,split,compaction and other operations are prone to cause node instability, data import slow, resource-consuming problems, in the import process of massive data greatly consumes the system performance,AvoidThe best way to do these problems is to use blukload to load data into hbase. principleWith hbase data stored in HDFS based on the hfile format, using MapReduce to generate the hfile format file directly, Regionservers then move the hfile file to the appropriate region directory with its processes such as: (Figure from Cloudera) Import Process 1. Generating a hfile file using MapReduce Generatehfile class

public class Generatehfile extends Mapper<longwritable, Text, immutablebyteswritable, put>{protected void map (longwritable key, Text value, Context context) throws IOException, interruptedexception {String line            = Value.tostring ();            string[] items = line.split ("\ t");            String ROWKEY = items[1] + items[2] + items[3];            Immutablebyteswritable Rowkey = new Immutablebyteswritable (Rowkey.getbytes ());   Put put = new put (rowkey.getbytes ());            ROWKEY put.addcolumn ("INFO". GetBytes (), "URL". GetBytes (), items[0].getbytes ());  Put.addcolumn ("INFO". GetBytes (), "SP". GetBytes (), items[1].getbytes ());  Starting point Put.addcolumn ("INFO". GetBytes (), "EP". GetBytes (), items[2].getbytes ());   Destination Put.addcolumn ("INFO". GetBytes (), "ST". GetBytes (), items[3].getbytes ());  Departure time Put.addcolumn ("INFO". GetBytes (), "Price". GetBytes (), Bytes.tobytes (integer.valueof (items[4))); Price PUt.addcolumn ("Info". GetBytes (), "Traffic". GetBytes (), items[5].getbytes ());//Traffic Mode Put.addcolumn ("info". GetBytes (), "HOTEL". GetBytes (), items[6].getbytes ());         Hotel Context.write (Rowkey, put); }}

　

Generatehfilemain class

public class Generatehfilemain {public static void main (string[] args) throws IOException, ClassNotFoundException, Interr uptedexception {final string input_path= "Hdfs://master:9000/info/input"; final string output_path= "Hdfs://master : 9000/hfile/output "; Configuration conf = hbaseconfiguration.create (); htable table = new htable (conf, "travel"); Job job=job.getinstance (conf); Job.getconfiguration (). Set ("Mapred.jar", "/home/hadoop/travelproject/out/artifacts  /travel/travel.jar "); Pre-packaging the program and distributing the jar to the cluster Job.setjarbyclass (generatehfilemain.class); Job.setmapperclass (Generatehfile.class); Job.setmapoutputkeyclass (Immutablebyteswritable.class); Job.setmapoutputvalueclass (Put.class); Job.setoutputformatclass (Hfileoutputformat2.class); Hfileoutputformat2.configureincrementalload (Job,table,table.getregionlocator ()); Fileinputformat.addinputpath (job,new Path (Input_path)); Fileoutputformat.setoutputpath (job,new Path (Output_path)); System.exit (Job.waitforcompletion (true)? 0:1);}}


NoteThe output key type of 1.Mapper must be the immutablebyteswritable format that contains Rowkey, the value type must be keyvalue or put type, and when the imported data has multiple columns, use put when there is only one column keyvalue 2. The value of Job.setmapoutputvalueclass determines the value of Job.setreduceclass, where reduce mainly plays the role of sorting the data, when Job.setmapoutputvalueclass values Put.class and Keyval The ue.class corresponds to the putsortreducer and Keyvaluesortreducer of the Job.setreduceclass 3 respectively. Pre-partitioning a table when creating a table the parallel computer system of MapReduce can speed up the generation of the hfile file, and if the table is pre-partitioned (region), set the reduce number equal to the number of partitions (region) 4. Multiple context.write are required in the case of multi-column families 2. Loading hfile files via blukload mode

public class Loadincrementalhfiletohbase {public    static void Main (string[] args) throws Exception {        Configuration conf = hbaseconfiguration.create ();        Connection Connection = connectionfactory.createconnection (configuration);        Loadincrementalhfiles Loder = new Loadincrementalhfiles (configuration);        Loder.dobulkload (New Path ("Hdfs://master:9000/hfile/output"), New htable (conf, "Travel");}    }

since Blukload is bypassing the Write to Wal,write to Memstore and flush to disk process, it is not possible to do some copy data through WALAdvantages: 1. The import process does not occupy region resources 2. Can quickly import massive amounts of data 3. Save Memory Reference article: Apache hbase Write Path how-to:use hbase Bulk Loading, and why

Fast import of massive amounts of data through a blukload approach