Redis recently stepped on the pit summary

1, master-slave problem

Core system: The company previously developed its own deployed REDIS3 3 from 3 Sentinels, program-end fragmentation, and deployed Sentinels to the Lord.

Just the main one hung, causing the entire system to be available.

Optimized deployment: Add a virtual machine as a Sentinel dedicated machine, a total of 9 Sentinel (3 main 3 from 9 Sentinel), after testing, can switch normally.

2, with the business switching problems

A few days ago just a physical machine hung, Sentinel normal switch, but the program end error, found that connection Redis pool error, restart Web application

After the program is restored.

Optimizer: auto-connect

Online search, the next day let the development of students to try

Http://www.mamicode.com/info-detail-1896700.html

3. Maintenance Physics Machine

Because the master has switched to other machines, the virtual machine on this physical machine is all ready, it feels fine, and all system cards

, look at the Web server log, found to have been looking for this hanging standby, but also affect the business, it seems that I was too simple. Or should it be in the non

Business time to do downtime maintenance, whether the master or standby.

[ERROR] 2017-11-28 21:36:30.119 [Thread-8] [Error_logger]-Lost connection to Sentinel at 192.168.2.99:36381. Sleeping 5000ms and retrying. [ERROR] 2017-11-28 21:36:30.134 [Thread-5] [Error_logger]-Lost connection to Sentinel at 192.168.2.98:36380. Sleeping 5000ms and retrying.

[ERROR] 2017-11-28 21:36:30.241 [Thread-2] [Error_logger]-Lost connection to Sentinel at 192.168.2.97:36379. Sleeping 5000ms and retrying.

4, because of the double 11 activities, and then November did a lot of activities Redis cached data is expired, memory is not enough to alarm.

But found that the system still has 2g of memory

Two solutions (overcommit_memory)
1.  echo "Vm.overcommit_memory=1" >/etc/sysctl.conf  or vi/etcsysctl.conf, then reboot restarts Machine
2.  echo 1 >/proc/sys/vm/overcommit_memory  does not need to start the machine to take effect

Overcommit_memory parameter Description: Set the memory allocation policy (optional, set according to the actual situation of the server)/proc/sys/vm/overcommit_memory optional values: 0, 1, 2.
0 indicates that the kernel will check if there is enough available memory to use the process, and if there is enough memory available, the memory request is allowed, otherwise the memory request fails and the error is returned to the application process.
1 indicates that the kernel allows all physical memory to be allocated regardless of the current state of memory. 2, which indicates that the kernel allows you to allocate more memory than the sum of all physical memory and swap space. Note: Redis in the dump data, will fork a child process, the theory of children in the process of memory and parent is the same, such as the memory occupied by parent is 8G, This time also to allocate 8G of memory to the child, if the memory can not afford, will often cause the Redis server down machine or IO load too high, inefficient.

So here the more optimized memory allocation policy should be set to 1 (indicating that the kernel allows all physical memory to be allocated regardless of the current state of memory).

It also involves Overcommit and oom. What is Overcommit and Oom in Unix, when a user process uses the malloc () function to request memory, if the return value is NULL, the process will do the work if it knows that there is currently no available memory space.

Many processes print error messages and exit. Linux uses a different approach, and it responds "yes" to most requests for memory so that it can run more and more large programs. Memory is not used immediately after the memory is applied.
This technique is called overcommit. OOM Killer (oom=out-of-memory) occurs when there is not enough memory.

It chooses to kill some processes (user-state processes, not kernel threads) to free up memory. Overcommit's strategy under Linux Overcommit has three different strategies (documentation/vm/overcommit-accounting): 0. Heuristic strategy.
Reasonable overcommit will be accepted and unreasonable overcommit will be rejected.
1. Any overcommit will be accepted.
2. When the system allocates more memory than swap+n%* physical RAM (N% is determined by Vm.overcommit_ratio), a commit is rejected.
The Overcommit policy is set by Vm.overcommit_memory.
The percentage of Overcommit is set by Vm.overcommit_ratio.
# echo 2 >/proc/sys/vm/overcommit_memory # echo >/proc/sys/vm/overcommit_ratio when oom-killer occurs, Linux chooses which processes to kill
The function that selects the process is the Oom_badness function (in mm/oom_kill.c), which computes the number of points (0~1000) for each process.
The higher the number of points, the more likely this process will be killed. The number of points per process is related to Oom_score_adj, and Oom_score_adj can be set (-1000 lowest, 1000 highest).