Count the number of HDFs files, size, and number of files in a range of sizes

Count the number of HDFs files, less than the number of 20M files Hadoop fs-du-h/# statistics File size Hadoop fs-count/# Statistics file number, returned data is directory number, number of files, file total size, input path Hadoop Fs-ls-r/path/data | grep ^-> ~/data.txt #统计所有文件的信息, Filter folders, only statistics files. Because after using ls-l, you can see that the file is "-" beginning with the "D" folder.

And write a little python.

Import pandas as PD
path= '/desktop/data.txt '
df=pd.read_table (path,delim_whitespace=true,names=[ 1,2,3,4,5,6,7,8] # Statistics total 8 column
print (len (DF))
df1=df[df[5]<20971520] # Fifth column is size, take less than 20M (converted to B) file data
Print (Len (DF1)) # Number of statistics