Kettle Series: Inserting data into kudu using the Kudu API

This article describes in detail the use of the Kudu API to write data to Kudu in kettle, which can be learned from this article:
1. How to write a simple kettle used defined Java class.
2. How to read the fields for each record in kettle. It is important to note that Getinteger () returns a long object; The method to get the Timestamp field is getdate ().
3. How to invoke the Kudu API.

The kettle example is very simple, and the data Grid component defines some sample data (which contains a number of types), and Java class writes the sample data to kudu.

Kudu Table Schema:

CREATE TABLEkudu_testdb.perf_test_t1 (ID string ENCODING plain_encoding COMPRESSION SNAPPY, Int_valueint, Bigint_valuebigint, Timestamp_valuetimestamp, Bool_valueint,    PRIMARY KEY(Histdate,id)) PARTITION byHASH (Histdate,id) partitions2STORED asKudutblproperties ('Kudu.table_name' = 'testdb.perf_test_t1',  'kudu.master_addresses' = '10.205.6.1:7051,10.205.6.2:7051,10.205.7.3:7051');

Focus on Java class Code:

ImportJava.sql.Timestamp;ImportJava.util.UUID;Import Staticjava.lang.Math.toIntExact;ImportOrg.apache.kudu.client.Insert;Importorg.apache.kudu.client.KuduClient;Importorg.apache.kudu.client.KuduException;Importorg.apache.kudu.client.KuduSession;Importorg.apache.kudu.client.KuduTable;ImportOrg.apache.kudu.client.PartialRow;Importorg.apache.kudu.client.SessionConfiguration;Private Final StaticString kudu_table= "Testdb.perf_test_t1";Private Final StaticString kudu_servers= "10.205.6.1:7051,10.205.6.2:7051,10.205.7.3:7051";Private Final Static intOperation_batch = 50; Kuduclient Client=NULL; Kudusession Session=NULL; kudutable Table=NULL; Integer RecordCount=NULL; Sessionconfiguration.flushmode mode;Privateobject[] Previousrow;Privateobject[] CurrentRow; Public BooleanProcessrow (stepmetainterface SMI, stepdatainterface SDI)throwskettleexception {if(first) { first=false; } CurrentRow=GetRow (); if(CurrentRow = =NULL) {setoutputdone (); return false; }     Try{session.setflushmode (mode);        Session.setmutationbufferspace (Operation_batch); intUncommit = 0;  while(CurrentRow! =NULL) {Insert Insert=Table.newinsert (); Partialrow Kudurow=Insert.getrow (); intinttmp;            Long longtmp;            String stringtmp;            Java.util.Date datetmp;                                        Boolean booleantmp; //Kettle String-Kudu string//kudurow.addstring ("id", Uuid.randomuuid (). toString ());Stringtmp = Get (fields.in, "id")). getString (CurrentRow); if(stringtmp!=NULL) {kudurow.addstring ("id", stringtmp); }                        //kettle int, kudu int//import static java.lang.Math.toIntExact;Longtmp=get (fields.in, "Int_value"). Getinteger (CurrentRow); if(longtmp!=NULL) {inttmp=tointexact (Get (fields.in, "Int_value"). Getinteger (CurrentRow)); Kudurow.addint ("Int_value", inttmp); }                         //kettle bigint, kudu bigintLongtmp=get (fields.in, "Bigint_value"). Getinteger (CurrentRow); if(longtmp!=NULL) {Kudurow.addlong ("Bigint_value", longtmp); }             //Kettle Date/timestamp, kudu timestampDatetmp= Get (fields.in, "Timestamp_value"). GetDate (CurrentRow); if(datetmp!=NULL) {longtmp=datetmp.gettime () +8*3600*1000;//time to go to East Zone 8 .Kudurow.addlong ("Timestamp_value", longtmp*1000); }                  //Kettle boolean-Kudu intBooleantmp= Get (fields.in, "Boolean_value"). Getboolean (CurrentRow); if(booleantmp!=NULL) {inttmp=0; if(booleantmp) {inttmp=1;} Kudurow.addint ("Boolean_value", inttmp); }                        //for manual submission, the buffer needs to be flush when it is not full, which is submitted when half of the buffer is used.Uncommit = uncommit + 1; if(Uncommit > OPERATION_BATCH/2) {Session.flush (); Uncommit= 0;            } session.apply (insert); Previousrow=CurrentRow; CurrentRow=GetRow (); }        //for manual submission, make sure to complete the final submission        if(Uncommit > 0) {Session.flush (); }       } Catch(Exception e) {e.printstacktrace (); Throwe; }  //Send the row to the next step. //PutRow (Data.outputrowmeta, currentrow);  return false;} Public BooleanInit (stepmetainterface stepmetainterface, Stepdatainterface stepdatainterface) {Try{Client=NewKuduclient.kuduclientbuilder (kudu_servers). build (); Session=client.newsession (); Table=client.opentable (kudu_table); Mode=SessionConfiguration.FlushMode.MANUAL_FLUSH; } Catch(Exception e) {e.printstacktrace (); Throwe; }  returnParent.initimpl (Stepmetainterface, stepdatainterface);} Public voidDispose (stepmetainterface SMI, stepdatainterface SDI) {Try {            if(!session.isclosed ())            {Session.close (); }       } Catch(Exception e) {e.printstacktrace (); Throwe; } Parent.disposeimpl (SMI, SDI);} 

Kettle Series: inserting data into kudu using the Kudu API