Uploading data to hive via Flume

Target:  store in the Hive database by accepting HTTP request information for Port 1084,

Osgiweb2.db the name of the database created for hive

Periodic_report5 for the Created data table,

The flume configuration is as follows:

a1.sources=R1  a1.channels=C1  a1.sinks=    k1 =0.0.  0.01084a1.sources.r1.handler=Jkong. Test.httpsourcedpihandler  

#a1. Sources.r1.interceptors=i1 I2


#a1. Sources.r1.interceptors.i2.type=timestamp

A1.channels.c1.type=Memory A1.channels.c1.capacity=10000a1.channels.c1.transactionCapacity= +A1.channels.c1.keep-alive= -A1.sinks.k1.type=HDFs A1.sinks.k1.channel=C1 A1.sinks.k1.hdfs.path=hdfs://Gw-sp.novalocal:1086/user/hive/warehouse/osgiweb2.db/periodic_report5A1.sinks.k1.hdfs.filetype=DataStream A1.sinks.k1.hdfs.writeFormat=Text A1.sinks.k1.hdfs.rollInterval=0a1.sinks.k1.hdfs.rollSize=10240A1.sinks.k1.hdfs.rollCount=0A1.sinks.k1.hdfs.idleTimeout= -A1.sources.r1.channels=C1a1.sinks.k1.channel=c1

2. Data Sheet creation:

CREATE TABLE Periodic_report5 (ID BIGINT, deviceId string,report_time string,information STRING) row format Serde"org.openx.data.jsonserde.JsonSerDe"With Serdeproperties ("ID"="$.id","deviceId"="$.deviceid","Report_time"="$.report_time","Information"="$.information");

2.1 The fields in the data table are also split into the creation statement of the data field (not yet tested, temporarily unused)

CREATE TABLE Periodic_report4 (ID BIGINT, deviceId string,report_time string,information Struct<actualtime:bigint, Dpiversioninfo:string,subdeviceinfo:string,wantrafficdata:string,poninfo:string,eventtype:string,potsinfo: string,deviceinfo:string,devicestatus:string>) row Format Serde"org.openx.data.jsonserde.JsonSerDe"With Serdeproperties ("Input.invalid.ignore"="true","ID"="$.id","deviceId"="$.deviceid","Report_time"="$.report_time","Requestparams.actualtime"="$.requestparams.actualtime","Requestparams.dpiversioninfo"="$.requestparams.dpiversioninfo","Requestparams.subdeviceinfo"="$.requestparams.subdeviceinfo","Requestparams.wantrafficdata"="$.requestparams.wantrafficdata","Requestparams.poninfo"="$.requestparams.poninfo","Requestparams.eventtype"="$.requestparams.eventtype","Requestparams.potsinfo"="$.requestparams.potsinfo","Requestparams.deviceinfo"="$.requestparams.deviceinfo","Requestparams.devicestatus"="$.requestparams.devicestatus");

3. Start the Flume statement: Flume root directory

Bin/flume-ng agent--conf./conf/-F./conf/flume.conf--name A1-dflume.root.logger=debug,console

4. Launch Hive statement: Hive Bin directory

Hive    or:. /hive-hiveconf hive.root.logger=debug,console  #带log信息启动

Uploading data to hive via Flume