PrefaceHDFS provides administrators with a quota control feature for the directory that can control
name Quotas(The total number of files & folders in the specified directory), or
Space Quotas(the upper limit for disk space). This paper explores the quota control characteristics of HDFs, and records the detailed process of various quota control scenarios. The lab environment is based on Apache Hadoop 2.5.0-cdh5.2.0. Welcome reprint, please specify Source: http://blog.csdn.net/u010967382/article/details/44452485
name quota function trialset the name quota, which is the maximum number of files and directories under the current directory:[Email protected]:~$
HDFs Dfsadmin-setquota 3 /workspace/quotas/15/03/18 14:53:52 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable
Upload file: [email protected]:~$HDFs dfs-put slf4j-log4j12-1.6.4.jar/workspace/quotas/15/03/18 14:54:16 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable
Upload file: [email protected]:~$HDFs dfs-put dict.txt/workspace/quotas/15/03/18 14:55:08 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable
Continue uploading files Error: [Email protected]:~$HDFs dfs-put examples.desktop/workspace/quotas/15/03/18 14:55:28 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicableput :The NameSpace quota (directories and files) of Directory/workspace/quotas is exceeded:quota=3 file count=4
View this directory status: [email protected]:~$ hdfs dfs-ls/workspace/quotas15/03/18 17:11:31 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where Applicablefou nd 2 items-rw-r--r--3 casliyang supergroup 2015-03-18 14:55/workspace/quotas/dict.txt-rw-r--r--3 Casliya ng SuperGroup 9748 2015-03-18 14:38/workspace/quotas/slf4j-log4j12-1.6.4.jarThere are only two files in the directory.
View quota Status: [Email protected]:~$ hdfs DFS-count-q/WORKSPACE/QUOTAS15/03/18 16:00:53 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable
3 0 None inf 1 2 9762/workspace/q Uotas
Note the meaning of several numbers in turn:
- name Quota Total (none represents not set)
- name Quota remainder (INF representative not set)
- Total space quotas (none represents not set)
- space quota remainder (INF representative not set)
-
- Number of files
- Content Footprint
- target address
Name Quota Remaining AmountThe calculation formula:
Name Quota Remaining Amount = Total Name Quota-(Number of directories + files)
View results based on above quota: Total name quota = 3 Number of directories = 1 files = 2
Therefore, the name quota remaining Amount =3-(1+2) =0 so the upload file will exceed the limit of the name quota at this time.
BTW, actually put the root directory also counted! Let's build a directory with no quotas set, and test whether the root directory is also included in the count statistic: [Email protected]:~$HDFs dfs-mkdir/workspace/quotas115/03/18 17:19:21 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where Applicable[em Ail protected]:~$HDFs dfs-count-q/workspace/quotas115/03/18 17:20:24 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable None inf none inf
10 0/WORKSPACE/QUOTAS1
The above experiment shows that the name quota and space quota are not set, so all display None/inf, sure enough the current root directory is counted, the directory number is 1. Cancel by command belowname quotas for the/workspace/quotas directory:[Email protected]:~$ hdfs dfs-count-q/workspace/quotas15/03/18 17:28:25 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable30None inf 1 2 9762/workspace/quotas[email protected]:~$
HDFs Dfsadmin-clrquota/WORKSPACE/QUOTAS15/03/18 17:28:40 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where Applicable[em Ail protected]:~$ hdfs dfs-count-q/workspace/quotas15/03/18 17:28:50 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicableNoneinfNone inf 1 2 9762/workspace/quotas
Space Quota function TrialClear Directory/workspace/quotas:[Email protected]:~$HDFs dfs-rm-r/workspace/quotas/*15/03/18 17:33:08 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable15/ 03/18 17:33:11 INFO fs. Trashpolicydefault:namenode Trash Configuration:deletion interval = 0 minutes, emptier interval = 0 minutes. DELETED/WORKSPACE/QUOTAS/DICT.TXT15/03/18 17:33:11 INFO fs. Trashpolicydefault:namenode Trash Configuration:deletion interval = 0 minutes, emptier interval = 0 minutes. Deleted/workspace/quotas/slf4j-log4j12-1.6.4.jar[email protected]:~$HDFs dfs-ls/workspace/quotas/15/03/18 17:33:36 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where Applicable[em Ail protected]:~$HDFs dfs-count-q/workspace/quotas15/03/18 17:33:49 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable None infNone inf1 00/workspace/quotas
Set a space limit for the directory: [email protected]:~$
HDFs Dfsadmin-setspacequota 8000/WORKSPACE/QUOTAS15/03/18 17:36:17 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where Applicable[em Ail protected]:~$ hdfs dfs-count-q/workspace/quotas15/03/18 17:36:27 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable None inf8000 80001 00/workspace/quotas for/workspace/quotas Directory Set a space quota of 8000 bytes, you can see that the total amount of space quota 8000, the space quota remaining 8000, the use of space quota 0.
Try uploading a file larger than 8000 bytes in size belowSlf4j-log4j12-1.6.4.jar, the file size is 9748 bytes:[Email protected]:~$ ll slf4j-log4j12-1.6.4.jar-rw-r--r--1 Casliyang Casliyang9748Mar 6 14:50 slf4j-log4j12-1.6.4.jar[email protected]:~$ HDFs dfs-putSlf4j-log4j12-1.6.4.jar/WORKSPACE/QUOTAS15/03/18 17:40:33 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable15/03/18 17:40:36 WARN HDFs. Dfsclient:datastreamer Exceptionorg.apache.hadoop.hdfs.protocol.DSQuotaExceededException:The diskspace quota Of/workspace/quotas is exceeded:
quota = 8000 B = 7.81 KB but diskspace consumed = 402653184 B = 384 MBAt Org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyDiskspaceQuota ( directorywithquotafeature.java:144)At Org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota (Directorywithquotafeature.java : 154)At Org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyQuota (fsdirectory.java:1815)At Org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount (fsdirectory.java:1650)At Org.apache.hadoop.hdfs.server.namenode.FSDirectory.updateCount (fsdirectory.java:1625)At Org.apache.hadoop.hdfs.server.namenode.FSDirectory.addBlock (fsdirectory.java:373)
The error is expected, but the last sentence of disk space consumption of 384MB, which is a bit unexpected, after consulting information:The space quota we need to set is not the size of the file itself, but the maximum value of the disk that the block occupies, for example, the file size in the experiment above is9748bytes, i.e. around 9.5k, the block size configured in Hdfs-site.xml is 128MB, the number of replicas is 3, the file can be stored in 1 blocks, so the maximum amount of disk to occupy (that is, the size of block) is 128*3=384mb!
We modify the quota for the directory to set it to 384mb:[email protected]:~$ HDFs Dfsadmin-setspacequota
384m/workspace/quotas
Upload file again: [email protected]:~$ hdfs dfs-put slf4j-log4j12-1.6.4.jar/workspace/quotas15/03/19 09:02:38 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where Applicable[em Ail protected]:~$ hdfs dfs-count-q/workspace/quotas15/03/19 09:03:06 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable None inf402653184 4026239401 19748/workspace/quotas
Visible, upload successful,space quota is
402653184 , the remaining quota is
402623940, the content takes up 9748 of the space
. The computational relationship of these three numbers is
402653184- 9748 = 402623940! even if the remaining space shows402623940, but you can't continue uploading anything, because the 3 128MB blocks that are used to store 9748 files are already full of disk space quotas!
****** spit groove A sentence, such a display mode is really unfriendly, do not understand the rules of the user must not read! ****
Do not forget, and then try to upload the file, as an experiment is a very small file, only14bytes: [Email protected]:~$ hdfs dfs-putDict.txt/workspace/quotas15/03/19 09:04:30 WARN util. nativecodeloader:unable to load Native-hadoop library for your platform ... using Builtin-java classes where applicable15/ 03/19 09:04:33 WARN HDFs. Dfsclient:datastreamer Exceptionorg.apache.hadoop.hdfs.protocol.DSQuotaExceededException:The diskspace Quota of/ Workspace/quotas is Exceeded:quota = 402653184 B =384 MBBut diskspace consumed = 402682428 B =384.03 MBUnable to upload, indeed the quota is full.
Summary of Points
- HDFs can set a name quota for a specified directory Name quotas and space quotas Space Quotas.
- Name quotas controls the number of directories and files under the specified root directory (see above for a specific calculation rule), space quotas control all file footprint under the specified root directory (see above for detailed calculation rules) .
- according to the official website, the maximum number of name quotas and space quotas is Long.max_value.
"Gandalf" Apache Hadoop 2.5.0-cdh5.2.0 HDFS Quotas Quota control
