Problem Description:
Hadoop in the virtual machine under the Linux system
Local files are uploaded to the specified directory on the Hadoop platform by writing code locally through eclipse
The code is as follows:
@Test Public voidUpload ()throwsIOException {Configuration conf=NewConfiguration (); Conf.set ("Fs.defaultfs", "Hdfs://lujie01:9000/"); FileSystem FS=filesystem.get (conf); Path Path=NewPath ("Hdfs://lujie01:9000/aa/lulu.txt"); Fsdataoutputstream OutputStream=fs.create (path); FileInputStream InputStream=NewFileInputStream ("D:/lulu.txt"); Ioutils.copy (InputStream, OutputStream); }
There are more simple ways to upload files:
Copyfromlocalfile method for direct use of filesystem
Fs.copyfromlocalfile (New Path ("D:/lulu.txt"), New Path ("Hdfs://lujie01:9000/aa/lulu.txt"));
The user identity of this machine cannot be authenticated by Hadoop, and cannot be uploaded until the user is set or the permissions of the/aa folder under Hadoop are changed.
The error is as follows:
Workaround One:
Set the permissions of the/aa folder to any user readable, writable, actionable
Execute the test program again, run normally, and the client can view the file Lulu.txt in AA. Indicates the upload was successful, note that the owner here is Lujie, the local user name of the computer
Workaround Two:
Set the arguments in the run configuration to change the user name to the user name of the Linux system Hadoop
Workaround Three:
Specify the user as Hadoop directly in the code
FileSystem fs = Filesystem.get (New URI ("Hdfs://lujie01:9000/"), conf, "Hadoop");
@Test Public voidUpload ()throwsIOException, Interruptedexception, urisyntaxexception {Configuration conf=NewConfiguration (); Conf.set ("Fs.defaultfs", "Hdfs://lujie01:9000/"); FileSystem FS = Filesystem.get (New URI ("Hdfs://lujie01:9000/"), conf, "Hadoop"); Path Path=NewPath ("Hdfs://lujie01:9000/aa/lulu1.txt"); Fsdataoutputstream OutputStream=fs.create (path); FileInputStream InputStream=NewFileInputStream ("D:/lulu.txt"); Ioutils.copy (InputStream, OutputStream); }
After the upload is successful, the client view
Hadoop HDFs Upload file permissions issue