Build a Web service download HDFs files

Requirements Description

In order to facilitate quick access to the files in HDFs, simple to build a Web service to provide download is very convenient and fast, and in the Web server side do not leave temporary files, only do stream relay, the efficiency is quite high!
The framework used is the Springmvc+hdfs API

Key code

@Controller@RequestMapping("/file")public class FileDownloadController {          private static final String BASE_DIR = "/user/app/dump/";    @RequestMapping(value = "/download/{filename}", method = RequestMethod.GET)    @ResponseBody    public void fileDownload(@PathVariable("filename") String fileName, HttpServletRequest request, HttpServletResponse response) {        try {            response.setContentType("application/octet-stream; charset=utf-8");            response.addHeader("Content-Disposition", "attachment; filename=" + URLEncoder.encode(fileName + ".csv", "UTF-8"));            String path = BASE_DIR + fileName;            HdfsUtils.copyFileAsStream(path, response.getOutputStream());        } catch (Exception e) {            e.printStackTrace();        }    }}

• Loading the files to be downloaded is in the/user/app/dump/directory
• Download Path Http://ip:port/file/download/xxxfile

Hdfsutils.copyfileasstream implementation

public class HdfsUtils {    private static FileSystem hdfs = null;    static {        URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());        Configuration conf=new Configuration();        try {            hdfs = FileSystem.get(URI.create("hdfs://xxxxxxx"), conf, "app");        } catch (Exception e) {            e.printStackTrace();        }     }    public static void copyFileAsStream(String fpath, OutputStream out) throws IOException, InterruptedException {        org.apache.hadoop.fs.Path path = new org.apache.hadoop.fs.Path(fpath);        FSDataInputStream fsInput = hdfs.open(path);        IOUtils.copyBytes(fsInput, out, 4096, false);        fsInput.close();        out.flush();    }}

Isn't it very simple? The file stream of HDFs is down on the Web service and is copied directly to the browser's OutputStream

Further improve performance, compression

Modify the Web-side code, ZIP compression, the default compression ratio is 1:5, greatly reducing the flow of traffic on the network

@Controller @requestmapping ("/file") public class Filedownloadcontroller {private static final String Base_dir = "/user        /app/dump/"; @RequestMapping (value = "/download/zip/{filename}", method = Requestmethod.get) @ResponseBody public void Hdfsdownlo            Ad2 (@PathVariable ("filename") String filename, httpservletrequest request, httpservletresponse response) {try { Response.setcontenttype ("Application/octet-stream;            Charset=utf-8 "); Response.setheader ("Content-disposition", "attachment;            Filename= "+ urlencoder.encode (fileName +". zip "," UTF-8 "));            Zipoutputstream zipout = null;                try {zipout = new Zipoutputstream (New Bufferedoutputstream (Response.getoutputstream ()));            Zipout.putnextentry (New ZipEntry (FileName + ". csv"));            } catch (Exception e) {e.printstacktrace ();            } String Path = Base_dir + fileName; Hdfsutils.copyfileasstream (path, zipout);           Zipout.close ();        } catch (Exception e) {e.printstacktrace (); }    }}

Some of the main jar versions used

<properties> <spring.version>4.2.5.RELEASE</spring.version> 

Build a Web service to download HDFs files