Spark-sql Custom Function UDF and UDAF

sparkconf sparkconf =Newsparkconf (). Setmaster ("Local"). Setappname ("Mysqltest"); Javasparkcontext Javasparkcontext=NewJavasparkcontext (sparkconf); SqlContext SqlContext=NewSqlContext (Javasparkcontext); List<String> list =NewArraylist<string>(); List.add ("2018-9-9,1"); List.add ("2018-5-9,1124"); List.add ("2018-9-9,1125"); List.add ("2018-5-9,1126"); List.add ("2016-10-9,1127"); Javardd<String> rdd_list = javasparkcontext.parallelize (list, 5); Javardd<Row> rdd_row_list = Rdd_list.map (NewFunction<string, row>() {@Override PublicRow Call (String s)throwsException {returnRowfactory.create (S.split (",") [0], Long.parselong (S.split (",") [1]);//convert to a Row object            }        }); List<StructField> structfieldlist =NewArraylist<structfield>(); Structfieldlist.add (Datatypes.createstructfield ("Date", Datatypes.stringtype,true)); Structfieldlist.add (Datatypes.createstructfield ("S", Datatypes.longtype,true)); Structtype Dytype=Datatypes.createstructtype (structfieldlist); DataFrame Df_dytype=sqlcontext.createdataframe (rdd_row_list, Dytype); Df_dytype.registertemptable ("Tmp_req");        Df_dytype.show (); //1, register a simple user Custom functionSQLCONTEXT.UDF (). Register ("zzq123",NewUdf1<string, integer>() {@Override PublicInteger Call (String str)throwsException {returnstr.length ();        }}, Datatypes.integertype); DataFrame Df_group= Sqlcontext.sql ("Select date,s,zzq123 (date) as zzq123 from Tmp_req");//UDF If no name is specified, the random namedf_group.show (); //1, register a complex user-defined aggregate functionSQLCONTEXT.UDF (). Register ("Zzq_agg",NewStringlen ());//Zzq_agg, simulating a similar count aggregation functionDataFrame Df_group_agg = Sqlcontext.sql ("Select Date,zzq_agg (s) as Zzq_agg from Tmp_req Group by date");//UDAF used for aggregation casesDf_group_agg.show ();

Aggregation functions:

 Public classStringcountextendsuserdefinedaggregatefunction {@Override PublicStructtype Inputschema () {//Inputschema refers to the type of data enteredlist<structfield> fields =NewArraylist<structfield>(); Fields.Add (Datatypes.createstructfield ("Str", Datatypes.stringtype,true)); returndatatypes.createstructtype (fields); } @Override PublicStructtype Bufferschema () {//Bufferschema refers to the type of data that is processed during aggregation in the middlelist<structfield> fields =NewArraylist<structfield>(); Fields.Add (Datatypes.createstructfield ("Count", Datatypes.integertype,true)); returndatatypes.createstructtype (fields); } @Override PublicDataType DataType () {//datatype refers to the type of function return value        returnDatatypes.integertype; } @Override Public BooleanDeterministic () {//consistency Check, if true, the result of the calculation is unchanged if the input is not changed        return true; }    /*** Update buffer value with input data, similar to Combinebykey * *@paramBuffer *@paraminput*/@Override Public voidUpdate (mutableaggregationbuffer buffer, Row input) {buffer.update (0, Integer.valueof (Buffer.getas (0). ToString ()) + 1); }    /*** Merge two buffer, merge buffer2 into Buffer1. will be used when merging the results of two partitions, similar to Reducebykey * Note that the method has no return value, and when implemented, merges buffer2 into Buffer1 , you need to implement this merge detail * *@paramBuffer1 *@paramBuffer2*/@Override Public voidmerge (Mutableaggregationbuffer buffer1, Row buffer2) {buffer1.update (0, Integer.valueof (Buffer1.getas (0). ToString ()) + integer.valueof (Buffer2.getas (0). toString ())); }    /*** Set the initial value of the aggregation intermediate buffer, but this semantics need to be guaranteed: Two initial buffer calls should also be the initial buffer after the merge method implemented below (if your initial value is 1, then you merge is to perform an additive action, two initial buffer And then equals 2, * does not equal the initial buffer. This initial value is problematic, so the initial value is also called "zero value" *@paramBuffer*/@Override Public voidInitialize (mutableaggregationbuffer buffer) {buffer.update (0, 0); }    /*** Calculates and returns the final aggregated result * *@paramBuffer *@return     */@Override PublicObject Evaluate (Row buffer) {returnBuffer.getint (0); }

Spark-sql Custom Function UDF and UDAF