Pig Common operations

In this section we look at the common operations of pig.

All commands and scripts are tested under Pig 0.12.0 & Hadoop 2.2.0.

Prepare two data files:

1) The Student.txt structure is (class number, school number, result), separated by commas between the fields.

c01,n0101,82

c01,n0102,59

c01,n0103,65

c02,n0201,81

c02,n0202,82

c02,n0203,79

c03,n0301,56

c03,n0302,92

c03,n0306,72

2) The Teacher.txt structure is (class number, teacher), separated by commas between the fields.

C01,zhang

C02,sun

C03,wang

C04,dong

Load and store (Load,store)

Execute the following command

Records = Load ' hdfs://localhost:9000/input/student.txt ' using Pigstorage (', ') as (Classno:chararray, Studno:chararray , Score:int);

Dump Records;

Store records into ' Hdfs://localhost:9000/input/student_out ' using Pigstorage (': ');

Then look at the part-m-00000 file under the Hdfs://localhost:9000/input/student_out directory, which reads as follows:

c01:n0101:82

c01:n0102:59

C01:n0103:65

c02:n0201:81

c02:n0202:82

c02:n0203:79

c03:n0301:56

c03:n0302:92

c03:n0306:72

Where load is the load operation, the store is the store operation. They can specify their separators, such as commas and semicolons in the previous example.

Filtering (Filter)

Execute the following command:

RECORDS_C01 = Filter Records byclassno== ' C01 ';

Dump RECORDS_C01;

The results are as follows:

(c01,n0101,82)

(c01,n0102,59)

(c01,n0103,65)

Note: You should use two equal signs to determine the equality of the two.

Foreach Generate

foreach loops through each record in the relationship, and then generates a new relationship by the specified pattern.

Execute the following command:

SCORE_C01 = foreach records_c01generate ' Teacher ', $1,score;

Dump SCORE_C01;

The results are as follows:

(teacher,n0101,82)

(teacher,n0102,59)

(teacher,n0103,65)

The new relationships that are generated include three fields, the first field is a constant, the second field is a study number (we refer to it by index number), and the third field is a fraction (which we refer to by the field name).

Group (Group)

Execute the following command:

Grouped_records = Group Recordsby classno parallel 2;

Dump Grouped_records;

The results are as follows:

(c02,{(c02,n0203,79), (c02,n0202,82), (c02,n0201,81)})

(c01,{(c01,n0103,65), (c01,n0102,59), (c01,n0101,82)})

(c03,{(c03,n0306,72), (c03,n0302,92), (c03,n0301,56)})

One of the Paraller 2 means that 2 reduce operations are enabled.

How to count the number of passing and outstanding students in each class? Execute the following two commands:

result = foreach Grouped_records {

Fail =filter records by score < 60;

Excellent =filter records by score >=90;

Generategroup, COUNT (fail) as fail, count (excellent) as excellent;

};

Dump result;

The results are as follows:

(c01,1,0)

(c02,0,0)

(c03,1,1)

Off Topic:

Flatten operation, you can flatten the data format. We look at the role of flatten by tuple and bag respectively:

1) Effect of flatten on the tuple

Execute the following command:

A= foreach records generate, ($1,$2);

Dumpa;

The output results are as follows:

(C01, (n0101,82))

(C01, (n0102,59))

(C01, (n0103,65))

(C02, (n0201,81))

(C02, (n0202,82))

(C02, (n0203,79))

(C03, (n0301,56))

(C03, (n0302,92))

(C03, (n0306,72))

Then, execute:

b = foreach a generate $0,flatten ($);

Dump B;

The results are as follows:

(c01,n0101,82)

(c01,n0102,59)

(c01,n0103,65)

(c02,n0201,81)

(c02,n0202,82)

(c02,n0203,79)

(c03,n0301,56)

(c03,n0302,92)

(c03,n0306,72)

As a result, when flatten acts on a tuple, the fields in the flatten corresponding field (tuple) are flattened into the fields in the relationship. (I don't know how to explain it better)

2) The effect of flatten on bag

Execute the following command

c = foreach records generate $0,{($), ($1,$2)};

Dump C;

The results are as follows:

(c01,{(N0101), (n0101,82)})

(c01,{(N0102), (n0102,59)})

(c01,{(N0103), (n0103,65)})

(c02,{(N0201), (n0201,81)})

(c02,{(N0202), (n0202,82)})

(c02,{(N0203), (n0203,79)})

(c03,{(N0301), (n0301,56)})

(c03,{(N0302), (n0302,92)})

(c03,{(N0306), (n0306,72)})

Next Execute:

D = foreach C Generate $0,flatten ($);

Dump D;

The results are as follows:

(c01,n0101)

(c01,n0101,82)

(c01,n0102)

(c01,n0102,59)

(c01,n0103)

(c01,n0103,65)

(c02,n0201)

(c02,n0201,81)

(c02,n0202)

(c02,n0202,82)

(c02,n0203)

(c02,n0203,79)

(c03,n0301)

(c03,n0301,56)

(c03,n0302)

(c03,n0302,92)

(c03,n0306)

(c03,n0306,72)

As you can see, when flatten acts on bag, it eliminates nested relationships and produces results similar to the Cartesian product. (bad expression, the reader can understand it carefully).

Stream operation

You can embed a Python program in pig for use.

Create a Python file pass.py with the following content:

#! /usr/bin/envpython

Import Sys

For line Insys.stdin:

(c,n,s) = Line.split ()

if int (s) >= 60:

Print "%s\t%s\t%s"% (c,n,s)

Execute the following command:

Define pass ' pass.py ' ship ('/home/user/pass.py ');

Records_pass = Stream records through pass as (Classno:chararray, Studno:chararray, Score:int);

Dump Records_pass;

The results are as follows:

(c01,n0101,82)

(c01,n0103,65)

(c02,n0201,81)

(c02,n0202,82)

(c02,n0203,79)

(c03,n0302,92)

(c03,n0306,72)

As can be seen, the statistical results are for all passing records (>=60).

Where ship is used to submit Python programs to the Hadoop cluster.

Notice that the ' pass.py 'in the first command, not enclosed in single quotes, is enclosed by the characters on the left side of keyboard 1. (Do not know what to call this character, only know is a notation)

Join

Execute the following two commands first:

r_student = Load ' hdfs://localhost:9000/input/student.txt ' using Pigstorage (', ') as (Classno:chararray, StudNo: Chararray, Score:int);

R_teacher2 = Load ' hdfs://localhost:9000/input/teacher.txt ' using Pigstorage (', ') as (Classno:chararray, teacher: Chararray);

Back to the beginning of this article, we have two data files for students (class, school number, score), Teacher (class, name).

Execute the following command:

r_joined = Join R_student by Classno,r_teacher by Classno;

Dump r_joined;

(C01,n0103,65,c01,zhang)

(C01,n0102,59,c01,zhang)

(C01,n0101,82,c01,zhang)

(C02,n0203,79,c02,sun)

(C02,n0202,82,c02,sun)

(C02,n0201,81,c02,sun)

(C03,n0306,72,c03,wang)

(C03,n0302,92,c03,wang)

(C03,n0301,56,c03,wang)

Similar to the inner join inner join in SQL. Of course you can also use external connections, such as:

r_joined = Join R_student by Classno left outer,r_teacher by Classno;

Dump r_joined;

Note: The left outer/right outer to be written after the first relationship name. The following syntax is incorrect:

r_joined = Join R_student by Classno, r_teacher by Classno Leftouter; Error

Cogroup

The operation result of join IS planar (a set of tuples), and the result of Cogroup is nested structure.

Run the following command:

R1 = Cogroup r_student by Classno,r_teacher by Classno;

Dump R1;

The results are as follows:

(c01,{(c01,n0103,65), (c01,n0102,59), (c01,n0101,82)},{(C01,zhang)})

(c02,{(c02,n0203,79), (c02,n0202,82), (c02,n0201,81)},{(C02,sun)})

(c03,{(c03,n0306,72), (c03,n0302,92), (c03,n0301,56)},{(C03,wang)})

(C04,{},{(C04,dong)})

As can be seen from the results:

1) Cogroup and join operations are similar.

2) The generated relationship has 3 fields. The first field is the Join field, the second field is a package, the value is all tuples in relationship 1 that satisfy the matching relationship, and the third field is a package with values of all the tuples in relationship 2 that satisfy the matching relationship.

3) an outer join similar to join. For example, the fourth record in the result, the second field value is empty, because there are no records in relationship 1 that meet the criteria. In fact, the first statement is equivalent to the following statement:

r1= cogroup r_student by Classno Outer,r_teacher by Classno outer;

If you want to not appear in the results when there are no matching records in the relationship 1 or 2, you can use the inner in the relationship instead of the keyword to exclude them.

Execute the following statement:

R1 = Cogroup r_student by classno inner,r_teacher byclassno outer;

Dump R1;

The result is:

(c01,{(c01,n0103,65), (c01,n0102,59), (c01,n0101,82)},{(C01,zhang)})

(c02,{(c02,n0203,79), (c02,n0202,82), (c02,n0201,81)},{(C02,sun)})

(c03,{(c03,n0306,72), (c03,n0302,92), (c03,n0301,56)},{(C03,wang)})

As we talked about earlier in the flatten, execute the following command:

r2 = foreach R1 Generate Flatten ($), flatten ($);

Dump R2;

The results are as follows:

(C01,n0103,65,c01,zhang)

(C01,n0102,59,c01,zhang)

(C01,n0101,82,c01,zhang)

(C02,n0203,79,c02,sun)

(C02,n0202,82,c02,sun)

(C02,n0201,81,c02,sun)

(C03,n0306,72,c03,wang)

(C03,n0302,92,c03,wang)

(C03,n0301,56,c03,wang)

Cross

Execute the following command:

R = Cross R_student,r_teacher;

Dump R;

The results are as follows:

(C03,n0306,72,c04,dong)

(C03,n0306,72,c03,wang)

(C03,n0306,72,c02,sun)

(C03,n0306,72,c01,zhang)

(C03,n0302,92,c04,dong)

(C03,n0302,92,c03,wang)

(C03,n0302,92,c02,sun)

(C03,n0302,92,c01,zhang)

(C03,n0301,56,c04,dong)

(C03,n0301,56,c03,wang)

(C03,n0301,56,c02,sun)

(C03,n0301,56,c01,zhang)

(C02,n0203,79,c04,dong)

(C02,n0203,79,c03,wang)

(C02,n0203,79,c02,sun)

(C02,n0203,79,c01,zhang)

(C02,n0202,82,c04,dong)

(C02,n0202,82,c03,wang)

(C02,n0202,82,c02,sun)

(C02,n0202,82,c01,zhang)

(C02,n0201,81,c04,dong)

(C02,n0201,81,c03,wang)

(C02,n0201,81,c02,sun)

(C02,n0201,81,c01,zhang)

(C01,n0103,65,c04,dong)

(C01,n0103,65,c03,wang)

(C01,n0103,65,c02,sun)

(C01,n0103,65,c01,zhang)

(C01,n0102,59,c04,dong)

(C01,n0102,59,c03,wang)

(C01,n0102,59,c02,sun)

(C01,n0102,59,c01,zhang)

(C01,n0101,82,c04,dong)

(C01,n0101,82,c03,wang)

(C01,n0101,82,c02,sun)

(C01,n0101,82,c01,zhang)

As you can see, cross is similar to the Cartesian product. In general, it is not recommended to use cross directly, but the data set should be filtered beforehand to improve efficiency.

Sort (order)

Execute the following command:

R = Order R_student by score Desc, classno ASC;

Dump R;

The results are as follows:

(c03,n0302,92)

(c01,n0101,82)

(c02,n0202,82)

(c02,n0201,81)

(c02,n0203,79)

(c03,n0306,72)

(c01,n0103,65)

(c01,n0102,59)

(c03,n0301,56)

Union (Union)

Execute the following statement:

r_union = Union r_student, R_teacher;

Dump r_union;

The results are as follows:

(c01,n0101,82)

(c01,n0102,59)

(c01,n0103,65)

(c02,n0201,81)

(c02,n0202,82)

(c02,n0203,79)

(c03,n0301,56)

(c03,n0302,92)

(c03,n0306,72)

(C01,zhang)

(C02,sun)

(C03,wang)

(C04,dong)

Can be seen:

1) union is to fetch two sets of recordsets.

2) The schema of the relational R_union is unknown (unknown), because the schema of the two relationships in the Union is not the same. If the schema of the two relationships is consistent, then the union relationship will be consistent with the schema of the Union's relationship.

Pig Common operations