How to write a custom hive UDF function

Hive allows users to write their own defined function UDFs to be used in queries. There are 3 types of UDFs in hive:

UDF: Manipulating a single row of data to produce a single row of data;

UDAF: Operates multiple rows of data, producing a single row of data.

UDTF: Operations a row of data, producing multiple rows of data for a table as output.

The user-built UDF uses the following procedures:

The first step: inheriting UDFs or UDAF or UDTF to implement a specific method.

UDF instances See http://svn.apache.org/repos/asf/hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/ Udfexampleadd.java

Package Org.apache.hadoop.hive.contrib.udf.example;import Org.apache.hadoop.hive.ql.exec.description;import org.apache.hadoop.hive.ql.exec.udf;/** * Udfexampleadd. * *///UDF is a single data row that produces a data row//the user must inherit the UDF and must implement at least one Evalute method that is not in the UDF// But hive checks to see if the user's UDF has a Evalute method @description (name = "Example_add", value = "_func_ (expr)-example UDAF that returns the sum public class Udfexampleadd extends UDF {//Implement concrete logical public  integer Evaluate (integer ... a) {    int total = 0;    for (Integer element:a) {      if (element! = null) {Total        + = element;      }    }    return total;  }  Public Double evaluate (double ... a) {double total    = 0;    for (Double element:a) {      if (element! = null) {Total        + = element;      }    }    return total;  }}

UDAF examples See

http://svn.apache.org/repos/asf/hive/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/udaf/example/ Udafexampleavg.java

Package Org.apache.hadoop.hive.contrib.udaf.example;import Org.apache.hadoop.hive.ql.exec.description;import Org.apache.hadoop.hive.ql.exec.udaf;import org.apache.hadoop.hive.ql.exec.udafevaluator;/** * This was a simple UDAF that calculates average. * * It should is very easy-follow and can be used as a example for writing * New Udafs. * * Note that Hive internally uses a different mechanism (called GENERICUDAF) to * implement built-in aggregation Functio NS, which is harder to program but * more efficient. * *///UDAF is the input of multiple rows of data, resulting in a data row//user-defined UDAF must be inherited UDAF, and the internal contains several implementation of the Exec static class @description (name = "Example_avg", value = "_func _ (Col)-Example UDAF to compute average ") publicly final class Udafexampleavg extends Udaf {/** * The internal state of   An aggregation for average.   * * Note that this was only needed if the internal state cannot was represented * by a primitive. * The internal state can also contains fields with types like * arraylist<string> and Hashmap<   string,double> if needed.    */public static class Udafavgstate {private Long mCount;  Private double mSum; }/** * The actual class for doing the aggregation.   Hive would automatically look * for all internal classes of the UDAF that implements Udafevaluator.    */public static class Udafexampleavgevaluator implements Udafevaluator {udafavgstate state;      Public Udafexampleavgevaluator () {super ();      state = new Udafavgstate ();    Init (); }/** * Reset the state of the aggregation.      * Reset the status of the aggregation process */public void init () {state.msum = 0;    State.mcount = 0;     }/** * Iterate through one row of original data.     * The number and type of arguments need to the same as we call this UDAF * from Hive command line. * * This function should always return true. * The number and type of iterations on the original value's row of data must be the same as the parameters of the UDF called in the Hive command line. * This function should always return true */public Boolean iterate (Double o) {if (o! = null) {state.msum += O;      state.mcount++;    } return true; }/** * Terminate a partial aggregation and return the state.     If the state was a * primitive, just return primitive Java classes like Integer or String. *///hive This method is called when a partial aggregate result is required//returns an object that encapsulates the current state of the aggregation calculation public udafavgstate terminatepartial () {//the IS SQL Standa      Rd-average of zero items should be null. return State.mcount = = 0?    Null:state;     }/** * Merge with a partial aggregation. * * This function should always had a single argument which had the same * type as the return value of terminate     Partial ().        *///merging two partial aggregation values calls this method public boolean merge (Udafavgstate o) {if (o! = null) {state.msum + = O.msum;      State.mcount + = O.mcount;    } return true; }/** * Terminates the aggregation and return the final result.  * Terminates the aggregation process, returning the final result *///hive the method is called when the final aggregation result is required public Double terminate () {//This is SQL standard-average of zero Items should be null. return State.mcount = = 0?    Null:Double.valueOf (State.msum/state.mcount); }} private Udafexampleavg () {//Prevent instantiation}}

The second step: register the well-written UDF function in hive, with the following two ways.

Method One

(1) Package A well-written class as a jar.

(2) Enter into the Hive shell environment, register the jar file with the Add Jar

(3) An alias for the class to use for the query.

Refer to the command below:

Add jar Udfexample.jar//Register jarcreate temporary function My_add as ' org.apache.hadoop.hive.contrib.udf.example. Udfexampleadd ';  

However, this method registers a UDF that only takes effect in the current hive session. If you want to take effect permanently, you can register the UDF function in the hive source code, see method two

Method Two

(1) Registering the UDF function in Org.apache.hadoop.hive.ql.exec.FunctionRegistry

REGISTERUDF ("My_add", Udfexampleadd.class, False) Registerudaf ("My_avg", Udafexampleavg.class);

(2) Packing and compiling hive source package

(3) Deploy the hive package and the UDF package and place the UDF package in the classpath of hive.

How to write a custom hive UDF function