- Receive multiple flume agents with one spark receiver
StringHost = args[0];intPort = Integer.parseint (args[1]);StringHost1 = args[2];intPort1 = Integer.parseint (args[3]); Inetsocketaddress Address1 =NewInetsocketaddress (Host,port); Inetsocketaddress Address2 =NewInetsocketaddress (HOST1,PORT1); Inetsocketaddress[] Inetsocketaddressarray = {ADDRESS1,ADDRESS2}; Javastreamingcontext JSSC =NewJavastreamingcontext (NewSparkconf (). Setappname ("Javaflumeeventhandle_1"), Durations.seconds (2)); javareceiverinputdstream<sparkflumeevent> Flumestream = Flumeutils.createpollingstream (JSSC, Inetsocketaddressarray, storagelevel.memory_and_disk_ser_2 ());
- 2 Spark Receiver receives multiple flume agents
String host = args[0];intPort = Integer.parseint (args[1]); String host1 = args[2];intPort1 = Integer.parseint (args[3]); Inetsocketaddress Address1 =NewInetsocketaddress (Host,port); Inetsocketaddress Address2 =NewInetsocketaddress (HOST1,PORT1); Inetsocketaddress[] Inetsocketaddressarray = {ADDRESS1,ADDRESS2}; Javastreamingcontext JSSC =NewJavastreamingcontext (NewSparkconf (). Setappname ("Javaflumeeventhandle_1"), Durations.seconds (2));//Javareceiverinputdstream<sparkflumeevent> Flumestream = Flumeutils.createpollingstream (JSSC, InetSocke Taddressarray, storagelevel.memory_and_disk_ser_2 ());javareceiverinputdstream<sparkflumeevent> Flumestream = Flumeutils.createpollingstream (Jssc,host,port); javareceiverinputdstream<sparkflumeevent> flumeStream1 = Flumeutils.createpollingstream (JSSC, host1, Port1); Javadstream<sparkflumeevent>Union= Flumestream.Union(FLUMESTREAM1);
Performance reference:
Http://blog.sina.com.cn/s/blog_13122bdbc0102vay7.html
One spark receiver or multiple spark receiver receives multiple flume agents