Erl_0015 "Erlang in the Smoke" Reading notes 002 "Plan for Overloading"

Plan for overloading

So far, the most common mistakes I've encountered in my actual work are basically running out of node memory. And they are usually related to the long message Queue 37.    There are many ways to solve such problems, but only after a thorough and comprehensive understanding of the system can we make the right choice. Basically, all the projects I work on can simplify the analogy into a very large bathroom sink. User requests and data flow from the faucet. The Erlang system is a sink and pipe that can be seen as a sewer system where water flows (databases, external APIs or services, etc.).

When the Erlang node dies due to a queue overflow, it is critical to find the cause. Is there too much water flowing into the trough? Is the sewer clogged? Or is the pipe designed too thin? Finding an inflated queue is not a difficult task, and this information can be obtained from crash dump. It is much more difficult to know why the queue is inflated. Depending on the role of the process, or doing some run-time checks, you can find some possible causes: message flooding, process blocking cannot quickly process messages, and so on.

Deciding how to fix it is the hardest part. We can change a larger sink (the collapsed part of the program, on the edge) because of water abuse that causes the sink to get stagnant. Next, find that the sink outlet is too small to optimize it. Then find that the pipe is too thin to optimize it. The load has been pushed downstream of the system until the sewer is unable to withstand it. At this point, we may try to increase the sink or add the bathroom to alleviate this global input problem 38.

However, things always go to the point where they can't be solved at the bathroom level. Too many logs are issued, a database to maintain consistency is a bottleneck, or simply because the company does not have the knowledge or manpower needed to solve the problem.

Only then did we find the real bottleneck in the system, and the optimizations we made were good (and the cost was not low), but it probably didn't work. We need to be smarter, take a step back and solve the problem upstream. You can handle the information that goes into the system, making it lighter (can be compressed, with better algorithms and data representation, caching, and so on). Even so, there is a situation where the system load is too high, and we have to make a difficult choice between restricting system input, discarding input, or accepting that the system will continue to degrade the quality of service before it crashes. These mechanisms are subordinate to two large strategies: Back Pressure (back-pressure) and load reduction (load-shedding).

Erl_0015 "Erlang in the Smoke" Reading notes 002 "Plan for Overloading"