Use Hadoop ACL to control access permissions.

Use Hadoop ACL to control access permissions.
Use Hadoop ACL to control access permissions 1. HDFS Access Control

Hdfs-site.xml settings startup acl

<property> 
<name>dfs.permissions.enabled</name> 
<value>true</value> 
</property> 

<property> 
<name>dfs.namenode.acls.enabled</name> 
<value>true</value> 
</property> 

Core-site.xml sets user group default permissions.

<property>
<name>fs.permissions.umask-mode</name>
<value>002</value>
</property>

The requirements and solutions are as follows:

• 1. Apart from the data warehouse owner, normal users cannot create databases or tables in the default database.
  The default permissions of/user/hive/warehouse are changed to 755. If the owner is hadoop (or the data warehouse owner), no one can create a database or create a table in the default database.

• 2. After the data warehouse owner creates a database, it can be assigned to the project team, where the project team can create tables.
  Change/user/hive/warehouse/database. db owner to project team.

• 3. After the data warehouse owner creates a database, he does not assign the table creation permission to the project team. Instead, he creates a table for it and only allows the project team to insert partitions.
  The data warehouse owner keeps the permissions of/user/hive/warehouse/database. db. The project team cannot create tables. After the data warehouse owner creates a table for the project team, the table directory is assigned to the project team.

• 4. Some tables can only be read and written by the project team.
  /User/hive/warehouse/database. db/table name directory changed to 770.

• 5. Some tables can only be read and written by special users in the project team.
  Change the owner of the/user/hive/warehouse/database. db/table name directory to this user and change the permission to 700.

• 6. For tables in the project team, special users in other groups are required to insert data.
  Use the following command to map the dntest. the database table testp1 has the write permission for hdfs dfs-setfacl-R-m user: mapengxu: rwx/user/hive/warehouse/cdntest. db/testp1

• 7. The table of the project team requires special users in other groups to have the permission to read data.
  Hdfs dfs-setfacl-R-m user: mapengxu: r-x/user/hive/warehouse/cdntest. db/testp1

• 8. For tables in the project team, all users in other groups must have the permission to read data.
  Hdfs dfs-setfacl-R-m group: data_sum: r-x/user/hive/warehouse/cdntest. db/testp1

• 9. Create a default database. All users of this database have the permission to create tables, but only save for 30 days.
  The permission of/user/hive/warehouse/database. db is changed to 777, and the scheduled task is set to scan this directory and hive database. If a table has been created for more than 30 days, delete the table and its directory.

• 10. This measure is combined with basic SQL access control.
  

Task Scheduling

Manage queues by user group, unified permissions in the portal and jenkins, allocate resources by group, to facilitate statistics by project team every day, the number of cluster resources occupied by each week. mapred-site.xml configuration is as follows:
<property> 
<name>mapred.acls.enabled</name> 
<value>true</value> 
</property> 
<property> 
<name>mapred.fairscheduler.poolnameproperty</name> 
<value>group.name</value> 
</property> 
Fair-scheduler.xml configuration is as follows:

<?xml version="1.0"?> 
<allocations> 

<pool name="cdn"> 
<maxResources>1000 vcores</maxResources> 
<maxRunningJobs>10</maxRunningJobs> 
<weight>1.0</weight> 
<schedulingPolicy>fair</schedulingPolicy> 
</pool> 
<pool name="data_sum"> 
<maxResources> 1000 vcores</maxResources> 
<maxRunningJobs>10</maxRunningJobs> 
<weight>1.0</weight> 
<schedulingPolicy>fair</schedulingPolicy> 
</pool> 

<userMaxAppsDefault>2</userMaxAppsDefault> 

<queuePlacementPolicy> 
<rule name="primaryGroup" create="false" /> 
<rule name="secondaryGroupExistingQueue" create="false" /> 
<rule name="user" create="false"/> 
<rule name="reject"/> 
'

'