Install pig and use pig to process this log

Configure pig environment variables to meet the mapreduce Mode

Vi. bashrc

Path = $ path: $ home/bin:/usr/lib/JVM/Java-6-sun-1.6.0.24/bin:/home/yangze/soft/pig-0.9.2/bin: /home/yangze/soft/hadoop-0.20.2/bin
Export path
Export java_home =/usr/lib/JVM/Java-6-sun-1.6.0.24
Export pig_classpath =/home/yangze/soft/pig-0.9.2/Conf
Export hadoop_conf_dir =/home/yangze/soft/hadoop-0.20.2/Conf

Start pig local mode is pig-x local

 

 

Import files to HDFS

 

Copyfromlocal/home/yangze/newdisk/study/class08/access_log.txt accessg

 

-- Load the log file into the table log

Grunt> log = load '/user/yangze/access/access_log'

> Using pigstorage ('')

> As (IP, A1, A3, A4, A5, A6, A7, A8 );

-- Filter A to retain only IP Fields

Grunt> B = foreach A generate IP;

-- Group by IP

Grunt> C = Group B by IP;

-- Calculate C by IP Address

Grunt> d = foreach C generategroup, count ($1 );

-- Display result:

Grunt> dump D;

Output (s ):
Successfully stored 476 records (14039 bytes) in: "HDFS: // master: 9000/tmp/temp-912504264/tmp675085582"

Counters:
Total records written: 476
Total Bytes written: 14039
Spillable Memory Manager spill count: 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job dag:
Job_201304161138_0004

2013-04-16 13:55:28, 963 [main] info org.apache.pig.backend.hadoop.exe cutionengine. mapreducelayer. mapreducelauncher-success!
2013-04-16 13:55:28, 970 [main] info Org. apache. hadoop. mapreduce. lib. input. fileinputformat-total input paths to process: 1
13:55:28, 971 [main] info org.apache.pig.backend.hadoop.exe cutionengine. util. mapredutil-total input paths to process: 1

(127.0.0.1, 2)
(1.59.65.67, 2)
(112.4.2.19, 9)
(112.4.2.51, 80)
(60.2.99.33, 42)
(69.28.58.5, 1)
(69.28.58.6, 9)
(69.28.58.8, 5)
(1.193.3.227, 3)
(1.202.221.3, 6)
(117.136.9.4, 6)
(121.31.62.3, 26)
(182.204.8.4, 59)
(183.9.112.2, 25)
(221.12.37.6, 25)
(223.4.16.88, 2)
(Maid, 122)
(61.189.63.2, 24)
(69.28.58.12, 3)
(111.161.72.7, 1)
(117.136.12.6, 61)
(117.136.19.9, 4)
(117.136.2.98, 1)
(117.136.20.3, 1)
(117.136.20.9, 1)
(117.136.3.46, 5)
(117.136.4.18, 5)
(117.136.4.19, 1)
(117.136.5.39, 9)
(117.136.5.70, 1)
(117.136.5.73, 17)
(117.136.7.67, 5)
(117.136.8.11, 32)
(117.136.8.48, 1)
(117.136.8.52, 1)
(117.136.8.97, 2)
(117.136.9.52, 2)
(117.136.9.68, 7)
(117.24.22.57, 2)
(121.28.95.48, 1597)

....
Grunt>