There is a case in which a partial request timed out when troubleshooting Redis as an RDB. The preliminary judgment is that weTHP (Transparent Huge Pages) is turned on on the Redis server。 1) the page size of Linux itself is fixed 4KB, the 2.6.38 kernel adds THP, transparently supports the use of huge page (2MB), and is turned on by default. The advantage of starting THP is:-Reduce page fault. One page fault can load a larger chunk of memory. -A smaller page table. The same memory size requires fewer pages. -the translation of the virtual address to the physical address is faster because the page table is smaller. The disadvantage is:-Reduce the allocation memory efficiency. Chunks of contiguous blocks of memory are required, and kernel threads are more aggressive in compaction, resolving memory fragmentation, and aggravating lock contention. -Reduce IO throughput. Because swapable huge page, the swap needs to be cut into the original 4K page. Oracle's test data display reduces IO throughput by 30%.
2) for Redis, the advantage of the open THP is that the time of the fork sub-process is greatly reduced. The main cost of the fork process is copying process data structures such as page tables, FD lists, and so on. Because the page table is significantly smaller (2MB/4KB = 512 times times), the fork time is significantly reduced. The disadvantage is that:-after fork, the parent-child process shares the address space in a copy-on-write manner. If the parent process has a large number of writes and does not have locality, a large number of pages are written and need to be copied. At the same time, due to the open THP, each page 2MB, will greatly increase the memory copy.
3) For this feature, I have done a test to test the fork and response time of Redis, respectively, in the case of open and close THP. Test conditions:Redis DataSet Size: 16G rdb File Size: 3.4G./redis-benchmark-p 4-t set-r 5000000-n 1000000000
(1) Fork Time comparisonWhen THP is turned on, fork is drastically reduced.
(2) The number of timeouts using Redis-benchmark test, a single kv only a few bytes, there is no way to simulate the real line delay, where the task delay more than 300US request is time-out, count the number of these requests. When THP is turned on, the number of timeouts increases significantly, but each time-out is shorter. When the THP is off, only 4 times out, because the request with the fork in the same event loop is affected by the fork, Chu ' L.off THP affects only a few sporadic requests, and when turned on, although the timeout period is short, but the impact surface expanded.
4) View THP status$ cat/sys/kernel/mm/transparent_hugepage/enabled [always] madvise neverAlways means open, madvise based on the configuration of the program, never off.
Close THPsudo echo never >/sys/kernel/mm/transparent_hugepage/enabled Restart process required
5) ConclusionWith the experience of the industry, we recommend that we close the THP on our online redis, MySQL, MongoDB and other machines.
6) References Transparent Huge Pages various pits:Https://blogs.oracle.com/linux/entry/performance_issues_with_transparent_hugeHttp://docs.splunk.com/Documentation/Splunk/6.2.3/ReleaseNotes/SplunkandTHPhttp://structureddata.org/2012/06/18/linux-6-transparent-huge-pages-and-hadoop-workloads/Http://antirez.com/news/84
Request Timeout case when Redis makes an RDB
