hadoop file formats and compression

Label: style HTTP color Io OS ar Java1 compression Generally, data processed by computers has some redundancy and there is correlation between data, especially between adjacent data. Therefore, data can be stored in special encoding methods different from the original encoding, make the storage space occupied by data relatively small, this process is generally called compression. The concept corresponding t

file suffix Unzip Command Compress Command . zip (Requires zip) Unzip File.zip Zip File.zip DirName . RAR (Requires RAR) RAR x File.rar RAR a File.rar . Tar (packaged, not compressed) Tar xvf File.tar Tar cvf File.tar DirName . tar.gz,. tgz Tar zxvf File.tar.gz Tar zcvf File.tar.gz DirName . tar.bz2,. tar.bz Tar jxvf File.t

Exception { TODO auto-generated Method Stub String inputfile = "Bigfile.txt"; String OutputFolder = "hdfs://192.168.129.35:9000/user/hadoop-user/Codec/"; String outputfile= "bigfile.gz"; Read the configuration of the Hadoop file system Configuration conf = new Configuration (); Conf.set ("Hadoop.job.ugi", "

calculate the checksum again when the data is transmitted through an unreliable channel, in this way, you can see whether the data is damaged. If the two calculation checksum does not match, you think the data is damaged. However, this technology cannot repair the data and can only detect errors. Common error detection code is CRC-32 (cyclic redundancy check), any size of data input is calculated to get a 32-bit integer checksum. 6. Compress and input parts

Algorithm File name extension Multiple files Severability DEFLATE No DEFLATE . Deflate No No Gzip Gzip DEFLATE . gz No No Zip Zip DEFLATE . zip Is Yes, within the scope of the file Bzip2 Bzip2 Bzip2 . bz2 No Is LZO

Algorithm File name extension Multiple files Severability DEFLATE No DEFLATE . Deflate No No Gzip Gzip DEFLATE . gz No No Zip Zip DEFLATE . zip Is Yes, within the scope of the file Bzip2 Bzip2 Bzip2 . bz2 No Is LZO

Summary of five commonly used picture formats and whether they have data compressionDisclaimer: Reference Please specify source http://blog.csdn.net/lg1259156776/Description: This article focuses on the five most common and most commonly used image formats: bmp,png,jpeg,jpeg200, and GIF. The first step before image processing related applications is to be able to read these image files, although many develo

compression of common data compression algorithmsThere are two main advantages of file compression, one is to reduce the space for storing files, and the other is to speed up data transmission. In the context of Hadoop big data, these two points are especially important, so

fileDbx:databearn image; Microsoft Visual FoxPro Table FileDct:microsoft Visual FoxPro Database ContainerDcu:delphi Compiling a unit fileDcx:microsoft Visual FoxPro Database container; pcx-based fax image; macroDir:macromediadirector fileDLL: Dynamic Link LibraryDoc:framemaker or framebuilder documents; Word star document, WordPerfect documents, Microsoft:word documents, displaywrite documentsDot:microsoft Word Document TemplateDpl:borland Delph 3 Compressi