Go Map reduce code framework template for HIVE UDF/UDAF/UDTF

from:http://hugh-wangp.iteye.com/blog/1472371

The template to use when you write your own code

UDF steps:

1. org.apache.hadoop.hive.ql.exec.UDF2 must be inherited. The Evaluate function must be implemented, and the Evaluate function supports overloading

Java code

2. <span style="Font-size:x-small;"  > Packagecom.alibaba.hive.udf;
6. Import Org.apache.hadoop.hive.ql.exec.UDF
10. Public class Helloword extends udf{
12. Public String Evaluate () {
14. return "Hello world!";
16. }
20. Public string Evaluate (String str) {
22. return "Hello World:" + str;
24. }
26. }</span>

UDAF steps:

1. Must Inherit Org.apache.hadoop.hive.ql.exec.UDAF (function class inheritance) Org.apache.hadoop.hive.ql.exec.UDAFEvaluator (inner class evaluator Real Now Udafevaluator interface) 2.Evaluator need to implement INIT, iterate, terminatepartial, merge, terminate these functions init (): Similar to constructors, for UDAF initialization of ITE Rate (): Receives incoming parameters and makes internal rotation. Its return type is Boolean terminatepartial (): No parameter, which returns the iterate function after the rotation of the functions, iterate and terminatepartial similar to Hadoop combiner ( Iterate--mapper;terminatepartial--reducer) Merge (): Receive terminatepartial return result, data merge operation with a return type of Boolean terminate () : Returns the result of the final aggregation function

Java code

2. <span style="Font-size:x-small;"  > Packagecom.alibaba.hive;
6. Import Org.apache.hadoop.hive.ql.exec.UDAF;
8. Import Org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
12. Public class Myavg extends udaf{
16. public static class avgscore{
18. private long pSum;
20. private double pcount;
22. }
26. public static class Avgevaluator extends udafevaluator{
28. Avgscore score;
32. Public Avgevaluator () {
34. Score = new Avgscore ();
36. Init ();
38. }
42. /* 
44. The *init function is similar to a constructor for UDAF initialization
46. */
48. public void init () {
50. Score.psum = 0;
52. Score.pcount = 0;
54. }
58. /* 
60. The *iterate receives the incoming parameters and makes internal rotations. Its return type is Boolean
62. * Similar to mapper in Combiner
64. */
66. Public Boolean iterate (Double in) {
68. if (in = null) {
70. Score.psum + = in;
72. Score.pcount + +;
74. }
76. return true;
78. }
82. /* 
84. *terminatepartial no parameters, which is the iterate function after the end of the rotation, return to the rotation data
86. * Similar to reducer in combiner
88. */
90. Public Avgscore terminatepartial () {
92. return score.pcount = = 0?  Null:score;
94. }
98. /* 
100. The *merge receives the return result of the terminatepartial and makes a data merge operation with a return type of Boolean
102. */
104. Public Boolean merge (Avgscore in) {
106. if (in = null) {
108. Score.psum + = In.psum;
110. Score.pcount + = In.pcount;
112. }
114. return true;
116. }
120. /* 
122. *terminate returns the result of the final aggregation function
124. */
126. Public Double Terminate () {
128. return score.pcount = = 0?  null:Double.valueof (Score.psum/score.pcount);
130. }
132. }
134. }</span>

UDTF steps: 1. Must inherit Org.apache.hadoop.hive.ql.udf.generic.GenericUDTF
2. Implement initialize, process, close three methods
3.UDTF will first
A. Call the Initialize method, which returns information about the returned rows of the UDTF (number of returns, type)
B. Once the initialization is complete, the process method is called, the passed parameters are processed, and the result can be returned by the ForWord () method.
C. Last Close () method call to clean up the method that needs to be cleaned Java code

2. <span style="Font-size:x-small;" ><span style="Font-size:xx-small;" >public class Genericudtfexplode extends GENERICUDTF {
6. private Listobjectinspector listoi = null;
10. @Override
12. public Void Close () throws hiveexception {
14. }
18. @Override
20. Public Structobjectinspector Initialize (objectinspector[] args) throws udfargumentexception {
22. if (args.length! = 1) {
24. throw New Udfargumentexception ("explode () takes only one argument");
26. }
30. if (args[0].getcategory ()! = ObjectInspector.Category.LIST) {
32. throw New Udfargumentexception ("explode () takes an array as a parameter");
34. }
36. Listoi = (listobjectinspector) args[0];
40. arraylist<string> FieldNames = new arraylist<string> ();
42. arraylist<objectinspector> Fieldois = new arraylist<objectinspector> ();
44. Fieldnames.add ("col");
46. Fieldois.add (Listoi.getlistelementobjectinspector ());
48. return Objectinspectorfactory.getstandardstructobjectinspector (FieldNames,
50. Fieldois);
52. }
56. private final object[] forwardobj = new object[1];
60. @Override
62. public void Process (object[] o) throws hiveexception {
64. list<?> list = listoi.getlist (o[0]);
66. if (list = = null) {
68. return;
70. }
72. For (Object r:list) {
74. forwardobj[0] = R;
76. Forward (forwardobj);
78. }
80. }
84. @Override
86. Public String toString () {
88. return "explode";
90. }
92. }</span></span>

Go Map reduce code framework template for HIVE UDF/UDAF/UDTF