Tens concurrency Implementation secrets: The kernel is not the solution, but the problem!

Http://www.csdn.net/article/2013-05-16/2815317-The-Secret-to-10M-Concurrent-Connections

Summary: The c10k problem has made us realize that when a concurrent connection reaches 10K, a different solution is chosen, and the laptop performance may exceed the 16-core server. For c10k problems, we either bypass, or overcome; however, as concurrency increases, do you ever think about how to overcome c10m in this post-10K era?

Now that we have solved the c10k concurrent connection problem, how should I increase the level of support for tens concurrent connections? You might say it's impossible. No, the system is already supporting tens other concurrent connections in a way that you may not be familiar with or even aggressive.

To know how it's done, we first need to understand Errata Security CEO Robert Graham, and his "nonsense"--c10m defending the Internet at scale in the ShmooCon 2013 conference.

Robert Cleverly explained the problem in a way that I had never heard before. He first introduced a bit about the history of Unix, Unix was designed not to be a general server operating system, but the telephone network control system. Because it is the telephone network that actually transmits the data, there is a clear boundary between the control layer and the data layer. The problem is that we should not use UNIX servers as part of the data tier at all. Just as designing a server kernel that runs only one application, it is certainly different from designing a multi-user Server Core.

That 's what he said--the key to understanding the kernel is not the solution, the kernel is the problem.

This means that:

• Do not allow the kernel to perform all heavy tasks. Transfer tasks such as packet processing, memory management, processor scheduling, etc. from the kernel to the application to complete efficiently. Let Linux handle only the control layer, and the data layer is completely handed to the application for processing.

Ultimately, this system is designed to handle tens other concurrent connections, which process packets within 200 clock cycles and process application logic within 140,000 clock cycles. This is the key to minimizing code and cache loss due to the 300 clock cycles that are spent on primary memory access.

The data-tier-oriented system can process 10 million packets per second, and systems for the control layer can handle only 1 million packets per second.

This seems extreme, remember an old saying: extensibility is specialization. In order to do something, you can't outsource the performance problem to the operating system, you have to do it yourself.
Now, let's learn how Robert creates a system that can handle tens concurrent connections.

c10k Problem--the last ten years

Ten years ago, when engineers handled c10k scalability issues, try to avoid the server processing more than 10,000 concurrent connections. This problem has been resolved by improving the operating system kernel and replacing the thread server (Apache) with event-driven servers such as Nginx and node. With 10 years of migration from Apache to a scalable server, the adoption rate of scalable servers has grown faster in recent years.

Apache's Problem

• The problem with Apache is that server performance can become worse as the number of connections increases.

• Key points: Performance and scalability are not the same thing. When people talk about scale, they tend to talk about performance, but scale and performance are different, like Apache.

• A short-term connection that lasts a few seconds, such as a fast transaction, if 1000 transactions per second are processed, only about 1000 concurrent connections to the server.

• A transaction is extended to 10 seconds, and 10,000 concurrent connections must be opened to maintain 1000 transactions per second. In this case: although you disregard Dos attacks, Apache will also fall in performance, while a large number of download operations will also cause Apache to crash.

• What would you do if the number of connections processed per second increased from 5,000 to 10,000? For example, you upgrade the hardware and increase the processor speed to twice times the original. What happened? You get twice times the performance, but you don't get twice times the scale of processing. The connection processed per second may only reach 6000. You continue to improve the speed, the situation has not improved. Even 16 times times the performance, still cannot handle 10,000 concurrent connections. So, performance and scalability are different.

• The problem is that Apache creates a CGI process and then shuts it down, and this step is not extended.

• Why is it? The O (n^2) algorithm used by the kernel makes it impossible for the server to process 10,000 concurrent connections.

• Two basic issues in the kernel:

• Number of connections = number of threads/processes. When a packet comes in, the kernel iterates through all its processes to determine which process is handling the packet.

• Number of connections = SELECT/Poll count (single thread). The same scalability issue, each packet has to go to all the sockets on the list.

• WORKAROUND: Improve the kernel to find it within a constant time.

• Causes the thread switching time to be independent of the number of threads.
• Use a new extensible epoll ()/iocompletionport constant time to do a socket query.

• Because thread scheduling is not extended, the server uses the Epoll approach to sockets on a large scale, which leads to the need to use asynchronous programming patterns, which are the Nginx and node type service ; So when migrating from Apache to Nginx and node type servers, the performance does not drop even if you increase the number of connections on a low-profile server, so when you connect to 10K, a laptop can even exceed the 16-core server.

c10m problem--the next 10 years

In the near future, the server will handle millions of of concurrent connections. IPV6 protocol, the number of potential connections per server is millions, so the scale of processing needs to be upgraded.

• Applications such as ids/ips need to support this scale because they are connected to a server backbone. Other examples: DNS root server, Tor node, internet nmap, video streaming, bank, Carrier Nat,voip PBX, load balancer, Web cache, firewall, email reception, spam filtering.

• Often people put the problem of Internet scale in the application rather than the server, because they are selling hardware + software. You buy the device and apply it to your data center. These devices may contain an Intel motherboard or network processor and a dedicated chip for encrypting and detecting packets.

• As of February 2013, the 40gpbs, 32-cores, 256gigs Ram X86 server on the Newegg website was quoted at $5000. The server can handle more than 10,000 concurrent connections if they can't, because you chose the wrong software, not the underlying hardware. This hardware can be easily extended to 10 million concurrent connections.

What does the 10M concurrent Connectivity Challenge mean:

1. 10 million number of concurrent connections
2. 1 million connections/sec-each connection lasts about 10 seconds at this rate
3. 10gb/second Connection--fast connection to the Internet.
4. 10 million packets/sec-it is estimated that the current server processes 50K of packets per second, more later. In the past, the server can handle 100K interrupts per second, and each packet generates interrupts.
5. Latency of 10 microseconds--scalable servers may be able to handle this scale, but latency may soar.
6. Jitter of 10 microseconds--limit maximum delay
7. Concurrent 10-core technology-software should support more servers with cores. Typically, the software can easily scale to four cores. The server can be scaled to more cores, so the software needs to be rewritten to support more cores of the server.

What we're learning is UNIX, not network programming.

• Many programmers pass the W. "UNIX Network Programming", Richard Stevens, learns network programming techniques. The problem is that this book is about UNIX, not just network programming. It tells you that to get UNIX to do all the heavy lifting, you just need to write a small server on top of UNIX. But the kernel size is not enough, and the solution is to move the business out of the kernel as much as possible and handle all the heavy business yourself.

• An example of this impact is the Apache model of each connected thread. This means that the thread scheduler determines which read () function to call next, based on the data that will be coming, that is, the thread dispatch system is used as a packet dispatch system. (I really like this, never thought of it).

• Nginx declares that it does not treat thread scheduling as a packet scheduler, but rather it does its own packet scheduling. Using Select to find the socket, we know that the data is coming, we can immediately read and process the data, the data will not be blocked.

• Experience: Let UNIX handle the network stack, but then you will handle the business.

How to write large scale software?

How to change your software and make it scale? Many of the experiences that only improve hardware performance to support project expansion are wrong, and we need to know the actual performance.

To reach a higher level, the following issues need to be addressed:

1. Scalability of the packet
2. Multi-Core Scalability
3. The scalability of memory

Enable packet Extensibility-write your own personalization drive to bypass the stack

• The problem with packets is that they need to be processed by the Unix kernel. The network stack is complex and slow, and the packet is best to reach the application directly, rather than after the operating system is processed.

• The way to do this is to write your own driver. All drivers send packets directly to the application, not through the stack. You can find this driver: Pf_ring,netmap,intel DPDK (Data Layer Development kit). Intel is not open source, but has a lot of relevant technical support.

• How fast is it? Intel's benchmark is to process 80 million packets per second (200 clock cycles per packet) on a fairly lightweight server. This is also done through user mode. Pass the packet up, use user mode, and then return after processing is complete. The number of packets per second processed by Linux is not more than million, the UDP packets are raised to user mode, and go out again. The client driver and Linux performance ratio is 80:1.

• For the target of 10 million packets per second, if 200 clock cycles are used to fetch packets, 1400 clock cycles will be left to implement similar dns/ids functions.

• You get the raw packets through pf_ring, so you have to do your TCP stack. What people do is the user-mode stack. Intel has a ready-made extensible TCP stack

Multi-Core Scalability

Multicore scalability differs from multithreaded scalability. We are all familiar with the idea that the speed of the processor is not getting faster, and we are simply relying on increasing the number to achieve the goal.
Most of the code does not implement parallelism above 4 cores. When we add more cores, it's not just the performance level that goes down, the processing speed may also get slower, which is the problem with the software. We want the speed of the software to be nearly linearly positive relative to the increase in the kernel.
Multithreaded programming differs from multi-core programming

• Multithreading

• More than one thread per CPU core
• Use locks to coordinate threads (through system calls)
• Each thread has a different task

• Multi-core

• Only one thread per CPU core
• When two threads/cores access the same data, they cannot stop waiting for each other
• Different threads for the same task

• The problem to solve is how to distribute an application across multiple cores

• The locks in Unix are implemented in the kernel. 4 The kernel uses locks when most software starts waiting for other lines to threads unlocked. As a result, the gain from increasing the kernel is much lower than the performance loss in waiting.

• We need an architecture that is more like a highway than a traffic light control intersection, without waiting for everyone to travel at their own pace and save as much as possible.

• Solution:

• The data structure is saved in each core and then aggregated to read.

• Atomic nature. CPU support can be called by the C language instruction, to ensure atomicity, to avoid collisions. It costs a lot, so don't use it everywhere.

• Lock-free data structure. Threads do not have to wait to be accessed and are complex to work under different architectures, please do not do it yourself.

• Threading model, that is, pipelining and worker threading models. This is not just a matter of synchronization, but how your thread is architected.

• Processor Association. Tell the operating system to take precedence over the first two cores, and then set the thread to run on which kernel, and you can get there by interrupting. So, the CPU is up to you to control instead of Linux.

The scalability of memory

• If you have 20G of RAM, assuming that each connection consumes 2K of memory, if you also have a 20M level three cache, there will be no data in the cache. Data transfer to main memory takes 300 clock cycles, and the CPU doesn't do anything at this time.

• For each packet to have 1400 clock cycles (Dns/ids function) and 200 clock cycles (get packets) overhead, each packet we only have 4 cache misses, this is a problem.

• Federated location Data

• Do not place data in full memory with the pointer. Every time you track a pointer, it will be a cache missing: [hash pointer], [Task Control Block], [Socket], [APP], which is four cache misses.

• Keep all the data in one memory block: [TCB |socket| APP]. Pre-allocates memory for all blocks, reducing cache misses from 4 to 1.

• Page out

• 32GB of data takes up 64MB of paging tables and is not suitable for storage in cache. So there are two cache misses-the paging table and the data it points to. This is the detail that can not be ignored to develop extensible software.

• Solution: Compress data, use a cache schema with a lot of memory access, rather than a binary search tree

• The NUMA architecture doubles the main memory access time. The memory may not be on the local socket, but on another socket.

• Memory pool

• Pre-allocating all memory immediately upon startup

• Allocations are made on the basis of objects, threads, and sockets.

• Hyper-Threading

• Each network processor can run up to 4 threads, and Intel can only run 2.

• In the appropriate case, we also need to cover up the delay, such as a thread in memory access waiting for another full speed.

• Large Memory Pages

• Reduce the size of the page table. Reserve memory from the start and let your application manage memory.

Summarize

• Card

• Problem: Low productivity through the kernel
• Solution: Use your own drivers and manage them so that the adapter is away from the operating system.

• Cpu

• Problem: Using traditional kernel methods to coordinate your application is not feasible.
• Solution: Linux manages the top two CPUs, and your application manages the rest of the CPU. Interrupts only occur on the CPUs that you allow.

• Memory

• Problem: Memory needs special attention for efficiency.

• Solution: Allocate most memory to the large memory pages you manage at system startup

The control layer is given to Linux and the application manages the data. There is no interaction between the application and the kernel, no thread scheduling, no system calls, no interrupts, nothing.
However, you have some code that runs on Linux and you can debug normally, which is not some weird hardware system that requires a specific engineer. You need custom hardware to improve performance on the data tier, but it must be done in your familiar programming and development environment.

Original connection: The Secret to Million Concurrent connections-the Kernel are the problem, not the solution (text/Zhou Xiaolu, reviewer/Zhonghao)

The biggest event in China's cloud computing industry-the " China Cloud Computing Conference " will be held at the Beijing National Convention Center in June 2013 5-7th . At the same time,2013, China's cloud computing ecosystem has been embryonic .

Tens concurrency Implementation secrets: The kernel is not the solution, but the problem!