Simple use of hdfshttpfs and webhdfs

HttpFS and WebHDFS have two components for operating hdfs through the http protocol: httpfs and webhdfs. At first I thought these two were the same thing, but they were not. Webhdfs comes with namenode and datanode, and httpfs is a completely independent component. To upload files through webhdfs, you must use a datanode instead of using namen directly.

HttpFS and WebHDFS have two components for operating hdfs through the http protocol: httpfs and webhdfs. At first I thought these two were the same thing, but they were not. Webhdfs comes with namenode and datanode, and httpfs is a completely independent component. To upload files through webhdfs, you must use a datanode instead of using namen directly.

HttpFS and WebHDFS

There are two components for operating hdfs through the http protocol: httpfs and webhdfs. At first I thought these two were the same thing, but they were not. Webhdfs comes with namenode and datanode, and httpfs is a completely independent component.

To upload files through webhdfs, you must use a datanode instead of using namenode. The client may access multiple machines. And httpfs, all operations are performed through httpfs.

The use of webhdfs and httpfs is basically the same, with only a small difference.

HttpFS Common Operations

View home Directory
Curl & quot; http: // osg11-vm06: 8014/webhdfs/v1? Op = GETHOMEDIRECTORY & user. name = kpi"

Create directory
Curl-I-X PUT "http: // osg11-vm06: 8014/webhdfs/v1/tmp/chenyk? Op = MKDIRS & user. name = kpi"

Curl-I-x put "http: // osg11-vm06: 8014/webhdfs/v1/tmp/chenyk/a/B/c? Op = MKDIRS & user. name = kpi"

Delete directory. The recursive Parameter deletes non-empty directories.
Curl-I-X DELETE "http: // osg11-vm06: 8014/webhdfs/v1/tmp/chenyk/? Op = DELETE & recursive = true & user. name = kpi"

Creating a file (httpfs does not make sense to execute this step, and the upload is also done through httpfs)
Curl-I-X PUT "http: // osg11-vm06: 8014/webhdfs/v1/tmp/chenyk/test? Op = CREATE & user. name = kpi"

Create and upload files
Add the header here, otherwise an error is prompted, it may be a bug https://issues.cloudera.org/browse/HUE-679

Curl-I-X PUT-T/tmp/test.txt "http: // osg11-vm06: 8014/webhdfs/v1/tmp/chenyk/test? Op = CREATE & data = true & user. name = kpi "-H" Content-Type: application/octet-stream"

Append a file
Curl-I-X POST-T/tmp/test.txt "http: // osg11-vm06: 8014/webhdfs/v1/tmp/chenyk/test? Op = APPEND & data = true & user. name = kpi "-H" Content-Type: application/octet-stream"

Open the file and read it
Curl-I-L "http: // osg11-vm06: 8014/webhdfs/v1/tmp/chenyk/test? Op = OPEN & user. name = kpi"

Reference

Http://hadoop.apache.org/docs/r1.0.4/webhdfs.html

Original article address: simple use of hdfs httpfs and webhdfs, thanks to the original author for sharing.