Recording Redis "Connection timed out" processing

Recent program error: Failed to connect to Redis:connection timed out

Find information about THP, decide to test and modify

1.page and Huge Pages

Page

In general, memory management of the minimum block-level units called page, a page is 4096bytes,1m memory will have 256 PAGE,1GB words will have 256,000 page.

The CPU maintains the page table record through the built-in memory management unit.

Huge Pages:

Huge pages is a 2M to 1GB memory page, mainly used to manage thousands of trillion of memory, such as 1GB page for 1TB memory is relatively appropriate.

Hugepage is widely enabled starting at Kernal 2.6, and some versions of the 2.4 kernel can also be used. In an operating system Linux environment, memory is allocated as page pages, with a default size of 4K. If you need to compare large memory space, you need to make frequent page allocations and manage addressing actions.

Hugepage is an alternative to traditional 4K page. As the name implies, using Hugepage allows us to have a larger memory paging size. Whether it's a hugepage or a traditional normal page, this process involves the OS memory addressing process

When a process accesses memory, it is not directly in memory location access, it is necessary to transfer the transformation through the page table. In the case of Hugepage, Pagetable has an additional attribute, which is to determine whether the page record is hugepage or regular.

2.THP

THP (Transparent Huge pages) is an abstraction layer that enables the administration of Huge pages automation.

Linux itself has a fixed page size of 4KB, a new THP in the 2.6.38 kernel, transparently supports the use of huge page (2MB), and is turned on by default.

THP Benefits:

(1) Reduce page fault. One page fault can load a larger chunk of memory.

(2) A smaller page table. The same memory size requires fewer pages.

(3) Because the page table is smaller, the translation of the virtual address to the physical address is also faster.

THP Cons:

(1) Reduce the allocation memory efficiency. Chunks of contiguous blocks of memory are required, and kernel threads are more aggressive in compaction, resolving memory fragmentation, and aggravating lock contention.

(2) Reduce IO throughput. Because swapable huge page, the swap needs to be cut into the original 4K page. Oracle's test data display reduces IO throughput by 30%.

3. Test performance

To view the system version:

Uname-r Linux Ckl-master1 2.6.32-504.el6.x86_64

View the current huge pages settings:

grep huge/proc/meminfo anonhugepages:5971968 KBHUGEPAGES_TOTAL:0HUGEPAGES_FREE:0HUGEPAGES_RSVD: 0hugepages_surp:0hugepagesize:2048 KB

See if huge pages is turned on:

cat/sys/kernel/mm/transparent_hugepage/enabled [always] madvise never

Test THP to turn on Redis performance:

# /usr/local/redis/bin/redis-benchmark -p 4 -t set -r 5000000 -n  10000000====== set ======  10000000 requests completed in 41.43  seconds  50 parallel clients  3 bytes payload  keep  alive: 198.62% <= 1 milliseconds98.62% <= 2 milliseconds98.63%  <= 4 milliseconds98.63% <= 5 milliseconds98.63% <= 7  milliseconds98.63% <= 12 milliseconds98.64% <= 13 milliseconds98.64%  <= 14 milliseconds98.65% <= 15 milliseconds98.66% <= 16  milliseconds98.68% <= 17 milliseconds98.69% <= 18 milliseconds98.74%  <= 19 milliseconds98.83% <= 20 milliseconds98.96% <= 21  Milliseconds99.13% <= 22 milliseconds99.32% <= 23 milliseconds99.49% <= 24  milliseconds99.60% <= 25 milliseconds99.67% <= 26 milliseconds99.74%  <= 27 milliseconds99.79% <= 28 milliseconds99.81% <= 29  milliseconds99.84% <= 30 milliseconds99.85% <= 31 milliseconds99.86%  <= 32 milliseconds99.86% <= 33 milliseconds99.87% <= 34  milliseconds99.87% <= 35 milliseconds99.88% <= 36 milliseconds99.89%  <= 37 milliseconds99.89% <= 38 milliseconds99.89% <= 39  milliseconds99.89% <= 40 milliseconds99.90% <= 42 milliseconds99.90%  <= 43 milliseconds99.91% <= 44 milliseconds99.91% <= 49  Milliseconds99.91% <= 53 milliseconds99.91% <= 54 milliseconds99.92% <= 55 milliseconds99.92%  <= 56 milliseconds99.93% <= 57 milliseconds99.93% <= 62  milliseconds99.93% <= 63 milliseconds99.93% <= 65 milliseconds99.93%  <= 66 milliseconds99.93% <= 67 milliseconds99.94% <= 68  milliseconds99.94% <= 74 milliseconds99.94% <= 76 milliseconds99.94%  <= 77 milliseconds99.94% <= 78 milliseconds99.95% <= 79  milliseconds99.95% <= 81 milliseconds99.95% <= 82 milliseconds99.95%  <= 83 milliseconds99.95% <= 85 milliseconds99.96% <= 86  milliseconds99.96% <= 91 milliseconds99.96% <= 92 milliseconds99.97%  <= 93 milliseconds99.97% <= 95 milliseconds99.97% <= 96 milliseconds99.98% <=  97 milliseconds99.99% <= 98 milliseconds99.99% <= 99  milliseconds99.99% <= 100 milliseconds99.99% <= 101 milliseconds99.99%  <= 126 milliseconds100.00% <= 157 milliseconds100.00% <= 158  milliseconds100.00% <= 168 milliseconds100.00% <= 169  Milliseconds100.00% <= 169 milliseconds241382.62 requests per second

To turn off THP:

echo never >/sys/kernel/mm/transparent_hugepage/enabled cat/sys/kernel/mm/transparent_hugepage/enabled always ma Dvise [Never]

Test shutdown Thpredis Performance:

/usr/local/redis/bin/redis-benchmark-p 4-t set-r 5000000-n 10000000====== set ====== 10000000 requests completed in 2 6.72 seconds Parallel clients 3 bytes payload Keep alive:199.70% <= 1 milliseconds99.71% <= 2 milliseconds100 .00% <= 3 milliseconds100.00% <= 3 milliseconds374321.53 requests per second

Redis response time is significantly shortened after THP is turned off, it is recommended to turn off THP and restart the Redis process

This article from "Deep Breathing again attack" blog, declined reprint!

Recording Redis "Connection timed out" processing