HDFS quota operations

In HDFS, administrators can set certain names and space quotas for each directory. The name quota and space quota can be set separately, but from the management and implementation aspects, these two quotas are close to parallel. Namequota is a hard limit on the quantity of all files and directory names in this directory. When the quota is exceeded

In HDFS, administrators can set certain names and space quotas for each directory. The name quota and space quota can be set separately, but from the management and implementation aspects, these two quotas are close to parallel. Name quota is a hard limit on the number of files and directory names in the directory. When the quota is exceeded

In HDFS, administrators can set certain names and space quotas for each directory. The name quota and space quota can be set separately, but from the management and implementation aspects, these two quotas are close to parallel.
Name quota)Is a hard limit on the number of files and directory names in this directory. When the quota is exceeded, the file or directory creation fails. After renaming, the naming quota still works. If the RENAME operation violates the quota limit, the rename will fail. The Created directory does not have any quota settings. The upper limit of the name quota is Long. Max_Value. If the quota is 1, the directory is forcibly empty because the directory itself occupies one quota. The quota settings are persisted in fsimage. After the fsimage is started, if fsimage finds that it violates the quota limit, this generates a warning. In addition, an empty log is created when the quota is set or deleted.
Only the administrator can set the name quota and space quota. The following command sets/clears the name quota. N must be a positive integer.
dfsadmin -setQuota N directory...directory
dfsadmin -clrQuota directory...directory
For example, the following limits the/data directory to three name quotas:

Hdfs dfs-mkdir/datahdfs dfs-setQuota 3/datahdfs dfsadmin-setQuota 3/datahdfs dfs-touchz/data/f {1, 2} # create the third file hdfs dfs-touchz/data/ f3touchz: the NameSpace quota (directories and files) of directory/data is exceeded: quota = 3 file count = 4 # clear name quota hdfs dfsadmin-clrQuota/data

Space quota)Is the space size limit of the directory. If this quota is exceeded, block write operations will fail. Replicas are also part of the quota (GB of data and a total of three replicas consume 3 GB of space ). When the quota is 0, the file can be created, but the block cannot be written to the file. The command for creating a space quota is as follows. N can be 5 GB, 50 GB, 2 TB, or 2 TB. If N is a negative number, you cannot create a file or write any data.
dfsadmin -setSpaceQuota ...
dfsadmin -clrSpaceQuota ...

# Recreate/datahdfs dfs-rm-r/datahdfs dfs-mkdir/datahdfs dfsadmin-setSpaceQuota 30 m/data # generate a 9M file dd if =/dev/zero of =/ tmp/f bs = 1 M count = 200 # append to the dfs-appendToFile/tmp/f/data/f # Second append, the size of the dfs-appendToFile/tmp/f/data/f ...... appendToFile: The DiskSpace quota of/data is exceeded: quota = 1073741824 B = 1 GB but diskspace consumed = 1207959552 B = 1.13 GB

In additionfs -count -q directory...directoryCommand to display the directory name quota, remaining name quota, space quota, and available space quota. If no quota is set in the directory, none and inf are displayed.

? Refer:

Http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html

Original article address: HDFS quota operation practice. Thank you for sharing it with me.