HDFs quota settings and test _hadoop

Name Quota (Quota)

A name quota is a limit on the number of files and directory names in the corresponding directory. When this quota is exceeded, the file or directory is created and the name quota is still valid after renaming.

Because it's simpler, so we test directly: Step one: Create a test directory

[Root@testbig1 ~]# HDFs dfs-mkdir/data/test_quota1

Step two: Set the name quota for the created directory

[Root@testbig1 ~]# HDFs Dfsadmin-setquota 2/data/test_quota1

Step Three: View quota information for the directory

[Root@testbig1 ~]# HDFs dfs-count-q/data/test_quota1

2 1 None inf 1 0 0/data/test_quota1

Note: Here is an explanation of the return results above:

The 1th value indicates the limit of the number of files

A 2nd value indicates the number of available files

The 3rd value indicates the space limit size (bytes)

The 4th value indicates the amount of free space (in bytes)

The 5th value represents the number of directories

The 6th value represents the number of files

The 7th value represents the total size

The 8th value represents the file/directory name

Step four: Put file for name quota test

[Root@testbig1 ~]# HDFs dfs-put fair-scheduler.xml/data/test_quota1/

[Root@testbig1 ~]# HDFs dfs-count-q/data/test_quota1

2 0 None inf 1 1 1469/data/test_quota1

[root@testbig1~]# HDFs dfs-put griffin.keytab/data/test_quota1/

Put:the NameSpace Quota (directories andfiles) of directory/data/test_quota1 is exceeded:quota=2 file count=3

May have a friend to find the problem, the above set/data/test_quota1 name quota is 2, why after testing found that only a number of files can be stored.

This is because HDFs in the design of the directory itself, that is, to occupy a quota number, so set to 2 o'clock, in fact, only one file or directory.

Step five: Clear quota limits

[Root@testbig1 ~]# HDFs dfsadmin-clrquota/data/test_quota1

Step six: Put a file again

[Root@testbig1 ~]# HDFs dfs-put griffin.keytab/data/test_quota1/

You can see that the put is successful and view the quota information again:

[Root@testbig1 ~]# HDFs dfs-count-q/data/test_quota1

None inf none inf 1 2 2575/data/test_quota1

If the previous two values are none and INF, the name quota is not set.

Spatial quotas (space Quota)

A space quota is a space size limit for a directory. If this quota is exceeded, the block write operation will fail. Copies are also counted as part of the quota. When the space quota is 0, the file can be created, but the content cannot be written to the file.

Let's just test it and get a better understanding. Step One: Create a test directory

[Root@testbig1 ~]# HDFs Dfs-mkdir/data/test_quota2

Step two: Generate a file that tests 100MB of size

[ROOT@TESTBIG1 ~]# dd If=/dev/zero of=./file bs=1m count=100

100+0 Records in

100+0 Records out

104857600 bytes (MB) copied, 0.12234 s,857 MB/s

[Root@testbig1 ~]# du-sh file

100M file

Step three: Set the space quota size to 200MB

[Root@testbig1 ~]# HDFs Dfsadmin-setspacequota 209715200/data/test_quota2

Step four: Upload the file test

[Root@testbig1 ~]# HDFs dfs-put file/data/test_quota2/

Put:the diskspace Quota of/data/test_quota2 is Exceeded:quota = 209715200 B = mb but diskspaceconsumed = 402653184 b = 384 MB

It is estimated that friends are curious, this is what ghosts, clearly set to 200MB size of the quota, why 100MB of documents are not put down. I will explain here:

In fact, this is related to the size of the HDFS block. The HDFs blocksize (dfs.block.size, dfs.blocksize) Size of our environment is set to 128MB and the replica factor is 3.

Namenode the size of the block multiples when the file is written, and then checks the space quota for the corresponding directory.

When the "directory has written capacity + current Blocksize*3" is compared to the "directory space quota", an error is made if the former is greater than the latter. For our test above, 0+384MB > 200MB, which is larger than the space quota setting, will fail.

For more intuitive understanding, we set the space quota to 385MB, and then test:

[Root@testbig1 ~]# HDFs Dfsadmin-setspacequota 403701760/data/test_quota2

[Root@testbig1 ~]# HDFs dfs-put file/data/test_quota2/

Can be seen to have succeeded.

Of course, we can also view the space quota information:

[Root@testbig1 ~]# HDFs dfs-count-q/data/test_quota2/

None inf 403701760 89128960 1 1 104857600/data/test_quota2

The 3rd value indicates: space limit

A value of 4th indicates: free space

Upload again, and there will be an error:

[Root@testbig1 ~]# HDFs dfs-put FILE/DATA/TEST_QUOTA2/FSF

Put:the diskspace Quota of/data/test_quota2 is Exceeded:quota = 403701760 B = 385 MB but diskspaceconsumed = 717225984 b = 684 MB

Step five: Clear space quotas

[Root@testbig1 ~]# HDFs Dfsadmin-clrspacequota/data/test_quota2

[Root@testbig1 ~]# HDFs dfs-put FILE/DATA/TEST_QUOTA2/FSF

As you can see, after clearing the space quota, successfully upload the above file again.