Hadoop HDFS Access Update latency issue

I recently encountered a problem with Solr's file permissions, causing the index to crash. Colleagues also encountered a similar phenomenon of FS. Many experiences have found that Hadoop currently has synchronization issues with permissions management for the directory.

Under normal circumstances, when a user starts, the directory permissions become the user. As for the group that the user is in, you can ignore it directly and see no use. But there are times when this happens. Clearly running the program is Superuser, which is the default use of HDFs users (if there is a default), but can access the SOLR, permissions of the directory, and has been running stably.

But one day, maybe you suddenly want to change some configuration. Some applications have been restarted. Perhaps suddenly it has changed its authority. You won't be able to access it unless you start with the right user and fix the problem with the permissions. Two times, one is Cloudear solr (solrcloud) problem. Another time is an extension application of HDFs, which is encountered when the file system. It's all starting to work, and when it comes to directory cleanup or configuration changes, restarting some apps suddenly doesn't work. Like a mine that has delayed seizures. Time-delayed explosion of thunder.

Think about the mechanism of Hadoop, the mechanism of the HDFS system, and the Linux file System management mechanism. The last thought, perhaps, is the reason.

The HDFs file system, primarily nameserver, does not take effect immediately when managing permissions, unless your phone uses chmod to change permissions. It has a trigger condition for updating data. Until then, all files can be accessed across permissions. And it is allowed to do so. This is due to avoiding too fast synchronization and improving the management efficiency of the file system. Also reduces the number of problems caused by synchronization. Strictly speaking, its permission synchronization is triggered by "Man for Manual". Unless you specifically say that I need to change only, otherwise it will not be updated.

In this way, all the started processes, all namenode files that are already active, can be accessed across permissions. However, in the occasional case, it will trigger the permission synchronization, resulting in later applications can not be accessed suddenly, displayed as directory without permission.

All directory permissions need to be updated manually at this time, it is recommended to restart all service nodes at the same time, all related applications.

This article is from the Big Data AI research blog, so be sure to keep this source http://wisyou.blog.51cto.com/9556478/1626406

Hadoop HDFS Access Update latency issue