Flume-ng inserts data into the HBase-0.96.0

This article introduces flume data insert hdfs and common directory (), this article continues to introduce flume-ng to insert data into the hbase-0.96.0.

First, modify the flume-node.conf file in the conf directory under the flume folder in node (for the original configuration, refer to the above) and make the following changes to it:

1. Agent. sinks = k1
3. Agent. sinks. k1.type = hbase
4. Agent. sinks. k1.table = hello
5. Agent. sinks. k1.columnFamily = cf
6. Agent. sinks. k1.column = col1
7. Agent. sinks. k1.serializer = org. apache. flume. sink. hbase. SimpleHbaseEventSerializer
8. Agent. sinks. k1.channel = memoryChannel

agent.sinks = k1agent.sinks.k1.type = hbaseagent.sinks.k1.table = helloagent.sinks.k1.columnFamily = cfagent.sinks.k1.column = col1agent.sinks.k1.serializer = org.apache.flume.sink.hbase.SimpleHbaseEventSerializeragent.sinks.k1.channel = memoryChannel

 

However, unlike the above, it is not that easy to get a successful result this time because of the dependency version. Here you need to replace protobuf under the lib folder of flume with 2.5.0 version in the Hadoop-2.2.0, you also need to replace guava under the lib folder of flume with guava in the hadoop-2.2.0, delete the original corresponding jar file. Start to take effect.

SimpleHbaseEventSerializer in flume-ng only provides the simplest function of inserting hbase data. If there are other requirements, you have to write the HbaseEventSerializer class by yourself, define your own class in apache-flume-1.4.0-src/flume-ng-sinks/flume-ng-hbase-sink/src/main/java to implement the HbaseEventSerializer interface in flume. A simple example is as follows:

1. Publicclass MyHBaseSerializer implements HbaseEventSerializer {
3. Privatestaticfinal String [] COLUMNS = "column1, column2". split (",");
4. Privatestaticfinal String [] PARAMS = "col1, col2". split (",");
6. Privatebyte [] columnFamily = "cf". getBytes ();
7. Privatebyte [] content;
10. @ Override
11. Publicvoid configure (Context context ){
13. }
15. @ Override
16. Publicvoid configure (ComponentConfiguration conf ){
18. }
20. @ Override
21. Publicvoid initialize (Event event, byte [] columnFamily ){
22. This. content = event. getBody ();
23. This. columnFamily = columnFamily;
24. }
26. @ Override
27. Public List <Row> getActions (){
28. String string = Bytes. toString (content );
29. String value1 = string. substring (0, string. length ()/2 );
30. String value2 = string. substring (string. length ()/2, string. length ());
32. Map <String, String> map = Maps. newHashMap ();
33. Map. put (PARAMS [0], value1 );
34. Map. put (PARAMS [1], value2 );
37. List <Row> actions = new partition List <Row> ();
38. String rowKey = String. valueOf (System. currentTimeMillis ());
40. Put put = new Put (Bytes. toBytes (rowKey ));
41. For (int I = 0; I <COLUMNS. length; I ++ ){
42. String value = map. get (PARAMS [I]);
43. If (value = null)
44. Value = "";
45. Put. add (columnFamily, Bytes. toBytes (COLUMNS [I]), Bytes. toBytes (value ));
46. }
47. Actions. add (put );
48. Return actions;
49. }
51. @ Override
52. Public List <Increment> getIncrements (){
53. List <Increment> incs = new Entity List <Increment> ();
54. Return incs;
55. }
57. @ Override
58. Publicvoid close (){
60. }
62. }

public class MyHBaseSerializer implements HbaseEventSerializer {    private static final String[] COLUMNS = "column1,column2".split(",");    private static final String[] PARAMS = "col1,col2".split(",");    private byte[] columnFamily = "cf".getBytes();    private byte[] content;    @Override    public void configure(Context context) {    }    @Override    public void configure(ComponentConfiguration conf) {    }    @Override    public void initialize(Event event, byte[] columnFamily) {        this.content = event.getBody();        this.columnFamily = columnFamily;    }    @Override    public List<Row> getActions() {        String string = Bytes.toString(content);        String value1 = string.substring(0,string.length()/2);        String value2 = string.substring(string.length()/2, string.length());        Map<String,String> map = Maps.newHashMap();        map.put(PARAMS[0], value1);        map.put(PARAMS[1], value2);        List<Row> actions = new LinkedList<Row>();        String rowKey = String.valueOf(System.currentTimeMillis());        Put put = new Put(Bytes.toBytes(rowKey));        for (int i = 0; i < COLUMNS.length; i++) {            String value = map.get(PARAMS[i]);            if (value == null)                value = "";            put.add(columnFamily, Bytes.toBytes(COLUMNS[i]), Bytes.toBytes(value));        }        actions.add(put);        return actions;    }    @Override    public List<Increment> getIncrements() {        List<Increment> incs = new LinkedList<Increment>();        return incs;    }    @Override    public void close() {    }}

The function of this class is to split the content in the file by line and insert the content into the column1 and column2 columns respectively. The rowKey is the current time. After that, recompile and package the flume-ng code. Replace the corresponding jar files in the lib folder in the flume-ng directory. Then, change the value of agent. sinks. k1.serializer in the preceding figure to test. MyHBaseSerializer. Test indicates the package name.

HBase details: click here
HBase: click here

Hadoop + HBase cloud storage creation summary PDF

Regionserver startup failed due to inconsistent time between HBase nodes

Hadoop + ZooKeeper + HBase cluster configuration

Hadoop cluster Installation & HBase lab environment setup

HBase cluster configuration based on Hadoop cluster'

Hadoop installation and deployment notes-HBase full distribution mode installation

Detailed tutorial on creating HBase environment for standalone Edition