Installing Tez 0.9.0

Installing Tez 0.9.0

It took me 3 days to install this thing .......

Most of the information on the Internet is either the low version, or the writing is very messy, not detailed ...

Took a lot of detours, and finally, with the help of the great God, completed the installation of tez0.9.0. Hope to be of help to everyone.

first, to the official website to download the tez0.9.0-bin.tar.gz version of the compressed package

The reason for choosing the bin package here is that you do not need to compile manually to save time

second, upload to the master machine installation location, decompression

The extracted folder name is: Apache-tez-0.9.0-bin, under Apache-tez-0.9.0-bin/share has a tez.tar.gz compression package

Put it on the HDFs.

Remember to start Hadoop first

Hadoop fs-mkdir-p/user/tez

Hadoop fs-put Tez.tar.gz/user/tez

third. under/etc/hadoop in the Hadoop installation directory, create a new Tez-site.xml file

VI tez-site.xml
<configuration>                                                                                                                                                                                
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/user/tez/tez.tar.gz</value>  <!--here point to tez.tar.gz on HDFs---
< /property>
<property>
<name>tez.container.max.java.heap.fraction</name>  <! --This is because my machine is running out of memory and adding parameters--
<value>0.2</value>
</property>
</configuration >

fourth, the same directory, set Mapred-site.xml, yarn modified to Yarn-tez

<configuration>
<property>
     <name>mapreduce.framework.name</name>
     <!-- <value>yarn</value>                                                                                                                                                                      
     <value>yarn-tez</value>
     <final>true</final >
</property>
<property>
        <name>mapreduce.jobtracker.http.address</name>
        <value>master.hadoop:50030</value>
</property>
<property>
        <name >mapreduce.jobhistory.address</name>
        <value>master.hadoop:10020</value>
</ property>
<property>
</configuration>

fifth. In the same directory, modify the hadoop-env.sh, add at the end

Tez_conf_dir=hadoop Mounting Position/etc/hadoop/tez-site.xml

Tez is the name I renamed Apache-tez-0.9.0-bin, not renamed, under Configuration note

Tez_conf_dir=/usr/local/src/hadoop/etc/hadoop/tez-site.xml
Tez_jars=/usr/local/src/tez
export HADOOP_ classpath=${hadoop_classpath}:${tez_conf_dir}:${tez_jars}/*:${tez_jars}/lib/*

Sixth, the Tez-site.xml,mapred-site.xml,hadoop-env.sh,tez is apache-tez-0.9.0-bin to the other machines in the cluster in the same directory

For example:

Scp-r Tez-site.xml root@192.168.202.129:/usr/local/src/hadoop/etc/hadoop/

Seventh, restart Hadoop

eighth. Preparing test documents

In the master non-installation directory, create a new File01,file02 file

echo "Hello World Hello Tez" > file01
echo "Hello World Goodbye tez" > File02

and put them into HDFs.

Hadoop fs-mkdir-p/user/ceshi/tez_input
hadoop fs-put file01 file02/user/ceshi/tez_input

Ninth, test to see if the installation is successful

CD Tez

Hadoop jar Tez-examples-0.9.0.jar Orderedwordcount/user/ceshi/tez_input/user/ceshi/tez_output

After performing the error, view the files under/user/ceshi/tez_output

Enter http://192.168.202.128:8088/cluster on the Web page to view status and run calculation type (IP is master host IP)

This installation is complete at this time.

tenth. If you want to change the compute engine for hive to Tez, just start hive with the start of Hadoop and set it up:

After entering Hive

Set Hive.execution.engine=tez;

Then execute the SQL statement to test.

If the recovery uses MapReduce, enter set HIVE.EXECUTION.ENGINE=MR;

additional error during installation:

1. Just download the apache-tez-0.9.0-bin.tar.gz package and drop it directly onto HDFs. is the wrong approach.

Run an error saying that the Dag class could not be found

You should unzip it first and then drop the tez-tar.gz in the Apache-tez-0.9.0-bin/share/directory onto HDFs.

2. Insufficient machine memory problem, need to modify the Tez-site.xml file, if not added, the following error will appear:

<property>
<name>tez.container.max.java.heap.fraction</name>  
<value>0.2</ Value>
</property>

What is the specific value, depending on your own memory

3. Accidentally when configuring the hadoop-env.sh, the previous hadoop_classpath is overwritten, the other applications cannot be started (hive,hbase, etc.)

Start Hive Error:

Originally, when the configuration hadoop-env.sh, the previous Hadoop_classpath covered. Add on the Can

Light Teng is, the official online writing steps, really not detailed, spent a lot of time troubleshooting