Hadoop HDFS wrong FS: HDFS:/expected file :///

Source: Internet
Author: User

HDFS is a distributed file system. For programmers, HDFS is a common file system. hadoop encapsulates the underlying layer. Programmers operate files on HDFS according to the corresponding APIs, there is no much difference from operations on local disk files. However, this problem may still occur during initial contact.

 

For example, when obtaining a filesystem instance

Java. Lang. nullpointerexception
At org. Apache. hadoop. conf. configuration. Get (configuration. Java: 382)
At org. Apache. hadoop. conf. configuration. getboolean (configuration. Java: 570)
At org. Apache. hadoop. fs. filesystem. Get (filesystem. Java: 192)
At hadoop. Test. urlcat. copyfiletoanotherfile (urlcat. Java: 38) // This is a method I wrote and an error is returned.
At hadoop. Test. urlcat. Main (urlcat. Java: 83)

 

Code:

Package hadoop. test;

Import java. Io. bufferedinputstream;
Import java. Io. fileinputstream;
Import java. Io. ioexception;
Import java. Io. inputstream;
Import java. Io. outputstream;
Import java.net. malformedurlexception;
Import java.net. Uri;
Import java.net. url;

Import org. Apache. hadoop. conf. configuration;
Import org. Apache. hadoop. conf. configured;
Import org. Apache. hadoop. fs. filesystem;
Import org. Apache. hadoop. fs. fsurlstreamhandlerfactory;
Import org. Apache. hadoop. fs. path;
Import org. Apache. hadoop. HDFS. distributedfilesystem;
Import org. Apache. hadoop. Io. ioutils;
Import org. Apache. hadoop. util. progressable;

Public class urlcat extends configured {

Static {
Configuration. adddefaultresource ("hdfs-default.xml ");
Configuration. adddefaultresource ("hdfs-site.xml ");
Configuration. adddefaultresource ("mapred-default.xml ");
Configuration. adddefaultresource ("mapred-site.xml ");
}
×/Errors will be reported if this static block does not exist


Public void copyfiletoanotherfile (string [] ARGs)
{
Inputstream in = NULL;
Outputstream out = NULL;
Try {
String sourcefile = ARGs [0];
String targetfile = ARGs [1];
In = new bufferedinputstream (New fileinputstream (sourcefile ));

Configuration conf = new configuration ();
System. Out. println (CONF );
System. Out. println (URI. Create (targetfile) = NULL );
System. Out. println (CONF = NULL );
System. Out. println (filesystem. Get (URI. Create (targetfile), conf) = NULL );

Filesystem FS = distributedfilesystem. Get (URI. Create (targetfile), conf );
System. Out. println (FS );
Out = FS. Create (New Path (targetfile), new progressable (){
Public void progress () {system. Out. Print (".");}
});
Ioutils. copybytes (In, out, 4096, true );
} Catch (exception e ){
// Todo auto-generated Catch Block
E. printstacktrace ();
} Finally
{
Ioutils. closestream (in );
Ioutils. closestream (out );
}
}

Static {
URL. seturlstreamhandlerfactory (New fsurlstreamhandlerfactory ());
}

Public static void displayfile (string [] ARGs)
{
Inputstream in = NULL;
Try {
In = new URL (ARGs [0]). openstream ();
Ioutils. copybytes (in, system. Out, 4096, false );
} Catch (malformedurlexception e ){
// Todo auto-generated Catch Block
E. printstacktrace ();
} Catch (ioexception e ){
// Todo auto-generated Catch Block
E. printstacktrace ();
} Finally
{
Ioutils. closestream (in );
}
}
/**
* @ Param ARGs
*/
Public static void main (string [] ARGs ){
// Todo auto-generated method stub
New urlcat (). copyfiletoanotherfile (ARGs );
// Urlcat. displayfile (ARGs );
//
}

}

 

Cause: configuration seems to load only two basic files, so you need to manually import other configuration files

Configuration class: defaultresources. Add ("hadoop-default.xml ");
Finalresources. Add ("hadoop-site.xml ");

 

 

Next we will describe the entire code to the execution process, hoping to help people who are new to hadoop programming:

 

1. You must configure the Java environment java_home and class_path.

Export java_home =/usr/lib/JVM/Java-6-sun
Export Path = $ path: $ java_home/bin
Export classpath =.:/usr/lib/JVM/Java-6-sun/lib

 

2. write code locally. Of course, you can use the eclipse tool.

 

3. Set hadoop_classpath.

Hadoop_classpath points to the root directory of the class file, for example,/home/hadoop/eclipseworkspace/testproject/bin in the root directory of hadoop. Test.

 

4. Run the commandHadoop. Test. urlcat/home/hadoop/documents/test.txt HDFS: // 192.186.54.1: 8020/user/hadoop/test.txt

Another error occurred: Java. Lang. illegalargumentexception: Wrong FS: HDFS: // 192.186.54.1: 8020/user/hadoop/test.txt, expected: HDFS: // hadoop1
At org. Apache. hadoop. fs. filesystem. checkpath (filesystem. Java: 310)
At org. Apache. hadoop. HDFS. distributedfilesystem. checkpath (distributedfilesystem. Java: 99)
At org. Apache. hadoop. HDFS. distributedfilesystem. getpathname (distributedfilesystem. Java: 155)
At org. Apache. hadoop. HDFS. distributedfilesystem. Create (distributedfilesystem. Java: 195)
At org. Apache. hadoop. fs. filesystem. Create (filesystem. Java: 484)
At org. Apache. hadoop. fs. filesystem. Create (filesystem. Java: 384)
At hadoop. Test. urlcat. copyfiletoanotherfile (urlcat. Java: 46)
At hadoop. Test. urlcat. Main (urlcat. Java: 86)
Cause: the HDFS command cannot be used to describe the IP address. The hostname is required. Run the following command:

 

 Hadoop. Test. urlcat/home/hadoop/documents/test.txt HDFS: // hadoop1: 8020/user/hadoop/test.txt

Everything is OK.

 

My configuration file is IP rather than hostname, because there is no DNS server to help resolve, but the command still needs to use hostname.

 

To sum up, pay attention to the following two points. Configuration and HDFS: // hostname: Port/user/pathtofile/File

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.