Hadoop streaming Multi-user permissions issue

Original address: http://blog.sina.com.cn/s/blog_605f5b4f0101897z.html

the permissions of HDFs are very simple to judge, which is to compare the user name of the issuing instruction with the user name of the file.private void Check (Inode inode, fsaction access) throws Accesscontrolexception {if (inode = = null) {RET     Urn } fspermission mode = Inode.getfspermission ();
if (User.equals (Inode.getusername ())) {//user class if (Mode.getuseraction (). implies (Access)) {return;} } else if (Groups.contains (Inode.getgroupname ())) {//group class if (Mode.getgroupaction (). implies (Access)) {R Eturn;     }} else {//other class if (Mode.getotheraction (). implies (Access)) {return;} } throw new Accesscontrolexception ("Permission denied:user=" + user + ", access=" + Access + ", inode=" + ino   DE); } }
The reason for the permission denied encountered when a multiuser submits a task is related to the process of the task submission. 1the client that submits the task first will package the task-related files in Hadoop.tmp.dir, which is a local directory that needs to be verified by local system permissions. Because it is a temporary directory directly set to 777 on the line.2. The client willTask FilePackage WriteThe Mapreduce.jobtracker.staging.root.dir + "/" + user + "/.staging" directory for HDFs requires HDFS permission validation. You can usually choose between two methods of resolving. 1) KeepMapreduce.jobtracker.staging.root.dir is the default, this directory is chmod 7772)The User directory is established under the/USER directory in HDFs, and Chown is the user, which is equivalent to creating a user under HDFs. then set Mapreduce.jobtracker.staging.root.dir to/USER, which is officially recommended, and you can see that the description of this property is written like this: the root of the staging area for users ' job files in practice, this should is the directory where users ' home director IES is located (usually/user). 3) Some people say that you can specify users and groups by adding the Hadoop.job.ugi property when you start a task. I verify that this is not possible. See the source code is also directly get UGI = Usergroupinformation.getcurrentuser (); But look at the HDFs permissions policy this should be set.
There are some, such as setting the Dfs.permissions parameter off the check, change the source code directly do not check ...
Finally, be aware thatthe out put of the task requires the current user to have permissions, otherwise it is unsuccessful. Set to be the user directory just finefrom:http://blog.163.com/darkness@yeah/blog/static/131774484201272155937382/
Also, note about the/tmp directory: ERROR org.apache.pig.impl.io.filelocalizer-org.apache.hadoop.security.accesscontrolexception: Org.apache.hadoop.security.AccessControlException:Permission Denied:user=shashaa, Access=write, inode= "tmp":p wang7:supergroup:rwxr-xr-x In this case, of course, use chmod to change the TMP directory's permissions to 777.
About Mapreduce.jobtracker.staging.root.dir

Parameter	Value	Description
Mapred.system.dir	/var/mapr/cluster/mapred/jobtracker/system	The shared directory where MapReduce stores control files.
Mapred.job.tracker.persist.jobstatus.dir	/var/mapr/cluster/mapred/jobtracker/jobsinfo	The directory where the job status information is persisted in a file system to being available after it drops of the memory Queue and between Jobtracker restarts.
Mapreduce.jobtracker.staging.root.dir	/var/mapr/cluster/mapred/jobtracker/staging	the root of the staging area for users ' job files in practice, this should is the directory where users ' home director IES is located (usually/user)

users who launch Hadoop HDFS systems are super users and can do anything.
To enable leaf users to perform hadoop jobs, do the following:
Create a file Hadoop fs-mkdir/user/leaf with super user Hadoop
Use the Hadoop fs-chown and-chgrp command to change the file's user and user group permissions to the leaf
Add Item dfs.permissions.enabled value to True (hadoop0.21 version) in configuration file Hdfs-site-xml
To ensure that the leaf of the user a few nodes can ssh without password login, the Java environment variable is not a problem (we are configured to ignore this, thanks to the help of Big Brother flew to take care of)
This allows the leaf user to submit task data to the/user/leaf directory in HDFs and perform hadoop tasks, Note If the staging is not modified, the task still cannot be submitted, although the/user/leaf directory of HDFs can be manipulated