Linux Kernel Optimization

Linux Kernel Optimization

Disclaimer: This document is obtained from the Internet collation section by adding your own experiment section:

TCP

Common Terms:

Retries (try again ),

TCP server <---> Client Communication Status

SYN ---------------->

<-------------- SYN, ACK

ACK ---------------> establish a connection

Data1 ---------------->

<--------------- Data1, ACK

Data2 ---------------->

<--------------- No response

Data2 ----------------> retransmission [sequence Parameter tcp_sack, tcp_fack]

[Retransmission count parameters: tcp_retries1, tcp_retries2, tcp_orphan_retries]

Data Transmission

FIN ------------------>

<----------------- FIN and ACK (sometimes FIN and ACK are divided twice)

ACK -----------------> disconnect. close automatically

FIN ------------------>

<--------------- CLOSE_WAIT

SYN indicates establishing a connection, RTT (Round-Trip Time): Round-Trip latency

FIN indicates that the connection is closed. RTO (Retransmission TimeOut) indicates the Retransmission TimeOut time.

ACK indicates a response,

PSH indicates DATA transmission,

RST indicates Connection Reset

TCP packet capture Common Errors

Tcp out-of-order (tcp is faulty) # most of them are caused by network congestion

Tcp segment of a reassembled PDU # TCP segment ID

Tcp previous segment lost (tcp previous part loss)

Tcp acked lost segment (tcp response loss)

Tcp window update (tcp window update)

Tcp dup ack (tcp repeated response)

Tcp keep alive (tcp persistence activity)

Tcp retransmission (tcp retransmission)

 

Kernel Parameter explanation

Net. ipv4.tcp _ timestamps = 1

Note:

This parameter controls the RFC 1323 Timestamp and Window Scaling Options. By default

The window is scaled, but the flag bit can be used for control. 0-bit Control Window Scaling, 1-bit control timestamp.

The value is 0 (RFC 1323 is disabled)

The value is 1 (only Window Scaling is enabled)

The value is 2 (only the timestamp is enabled)

The value is 3 (both options are enabled)

Net. ipv4.tcp _ timestamps = 0

Note:

Timestamp can avoid serial number winding. A 1 Gbit/s link must have a previously used serial number. The timestamp allows the kernel to accept such "abnormal" packets. Disable it here.

0 (Time Stamp disabled)

1 (enable timestamp)

Only when the timestamp is enabled on both the client and server can ping the tcp three-way handshake be established. Therefore, as a service provider, it is impossible to ensure that all users disable timestamp. Therefore, we must disable timestamp to provide normal services to users.

Net. ipv4.tcp _ window_scaling = 1

Net. ipv4.tcp _ sack = 1

Using Selective ACK, it can be used to find specific lost data packets-so it helps to quickly restore the status. This file indicates whether Selective Acknowledgment is enabled ), this can improve the performance by selectively responding to messages received in disordered Order (this allows the sender to send only lost packets ). (This option should be enabled for Wan communication, but this will increase CPU usage .)

Net. ipv4.tcp _ fack = 1

Open FACK (Forward ACK)Congestion preventionAndFast retransmissionFunction. (Note: When tcp_sack is set to 0, this value is invalid even if it is set to 1)

Net. ipv4.tcp _ retrans_collapse = 1

Net. ipv4.tcp _ syn_retries = 5

For a new connection, the kernel determines how many SYN connection requests are sent before giving up. It should not be greater than 255. The default value is 5, corresponding to the left-right time of 180 seconds. (For a network with high load and good physical communication, this value is too high and can be changed to 2. This value is only for external connections,The incoming connection is determined by tcp_retries1.)

Net. ipv4.tcp _ synack_retries = 5

Tcp_synack_retries displays or sets the number of times the Linux core tries to resend the initial SYN when responding to the SYN request. The ACK packet does not give up. This is the second step of threeway handshake. That is to say how many times the system will try to establish a TCP connection starting from the remote end. The value of tcp_synack_retries must be a positive integer and cannot exceed 255. It takes about 30 to 40 seconds to wait for each re-send packet to decide whether to try the next re-send or give up. The default value of tcp_synack_retries is 5, that is, the timeout time for each line is determined after about 180 seconds (3 minutes.

Net. ipv4.tcp _ max_orphans = 131072

The maximum number of TCP sockets that the system can process for any process. If this number is exceeded, connections that do not belong to any process will be immediately reset and a warning will be displayed. The reason for setting this limit is purely to defend against simple DoS attacks. Do not rely on this limit or manually reduce it. You should also increase this value (if the memory is increased ). Each orphan socket can eat up to 64 KB of memory that cannot be exchanged.

Net. ipv4.tcp _ max_tw_buckets = 5000

This indicates that the system maintains the maximum number of TIME_WAIT sockets at the same time. If this number is exceeded, the TIME_WAIT socket is immediately cleared and warning information is printed. The default value is 180000. Set to a smaller value. This parameter can control the maximum number of TIME_WAIT sockets to prevent the server from being dragged to death by a large number of TIME_WAIT sockets.

Net. ipv4.tcp _ keepalive_time = 7200

Net. ipv4.tcp _ keepalive_probes = 9

Net. ipv4.tcp _ keepalive_intvl = 75
Use an instance to describe the preceding three parameters:

If a TCP connection takes two hours after idle, the kernel initiates probe ). if the probe is 9 times (the tcp_keepalive_intvl value is not successful every 75 seconds), the kernel will give up completely and the connection is deemed invalid.

Net. ipv4.tcp _ retries1 = 3

Give up responding to a TCPBefore Connection RequestThe number of retries required. RFC specifies that the minimum value is 3, which is also the default value. According to RTO, the value is between 3 seconds and 8 minutes. (Note: This value also determines the syn connection)

(Second explanation: it indicates the maximum number of retries of the route table not detected when the TCP transmission fails. When the value is exceeded, we need to check the route table)

Net. ipv4.tcp _ retries2 = 15

Activation in discard (communication established)Before the TCP connection, how many retries are required. The default value is 15, which is determined by the RTO value, equivalent to 13-30 minutes (RFC1122 requires that the value must be greater than 100 seconds ). (This value can be adjusted to 5 according to the current network settings)

(Second explanation: it indicates the maximum number of retries, but this value is generally greater than the above value. Different from the one above, when the number of retries exceeds this value, we must stop trying again)

Net. ipv4.tcp _ orphan_retries

This is mainly for the isolated socket (that is, it has been deleted from the context of the process, but some cleanup work has not been completed). For this socket, the maximum number of retries is its

Net. ipv4.tcp _ fin_timeout = 30

This parameter determines whether the socket is kept inFIN-WAIT-2Status time

Net. ipv4.tcp _ tw_recycle = 1

Enables fast TIME-WAITsockets recovery in TCP connections. The default value is 0, indicating that TIME-WAITsockets is disabled.

Net. ipv4.tcp _ stdurg = 0

Net. ipv4.tcp _ rfc1337 = 0

Net. ipv4.tcp _ max_syn_backlog = 8192

The length of the SYN queue. The default value is 1024. The length of the queue is 8192, which can accommodate more network connections waiting for connection.

(Second explanation: Maximum port backlog kernel limit. This parameter limits the maximum port backlog value that can be set by the server application (corresponding to the syn_backlog and backlog queue length of the port ). The motivation is to limit/prevent applications from consuming the kernel memory by configuring a large backlog value on servers with limited memory. If the backlog value set by the application is greater than this value, the operating system automatically limits this value .)

Net. ipv4.tcp _ abort_on_overflow = 0

After tcp establishes a three-way handshake, the connection is placed in the ESTABLISHED status and delivered to the backlog queue of the application, and the backlog queue is full. If the connection is full, the connection is usually restored to the SYN_ACK state, this creates the illusion of accidental loss of the last ACK packet of a three-way handshake-this way, you can resend ACK after the client waits for timeout-to attempt to enter the ESTABLISHED status again-as a repair/Retry Mechanism. If tcp_abort_on_overflow is enabled, when the backlog queue is full, an RST packet is sent directly to the client to terminate the Connection. In this case, the client will receive the 104 Connection reset by peer error.

Warning enabling this option may result in a 104: Connection reset by peer or white screen error during peak periods (depending on the browser ). Before enabling this option, try to optimize and improve the performance of the server application so that it can take over and process connections faster.

Net. ipv4.tcp _ syncookies = 1

When the server receives the initial SYN request during the three-way handshake of tcp connection, it checks whether the syn_backlog queue of the application is full. If it is full, the SYN packet is usually discarded. If not, check whether the application's backlog queue is full. If the application is full and the system determines that the application will not consume the connection quickly based on the history, the SYN packet will be discarded. If tcp_syncookies are enabled, the SYN packet is not discarded when the syn_backlog queue is full. Instead, a three-way handshake is performed using syncookie technology.

Warning when syncookie is used for handshake, because the technology misappropriates the tcp_options field space, it will forcibly disable the tcp advanced traffic control technology and degrade to the original tcp mode. This mode will cause the Peer to wait for MSL time to find the packet loss event and retry, And the TIME_WAIT State will remain 2MSL time when the connection is closed.This technology should only be used to protect syn_flood attacks. If the server load in a normal server environment is heavy and the syn_backlog and backlog queues are full, the load capacity of the server application should be optimized to increase the value of the application backlog. However, fortunately, this parameter is an automatic value and will only be triggered when the syn_backlog queue is full (this behavior is disabled when the queue is available for recovery ).

? The server application sets the port backlog value. The kernel will theoretically allow this port to receive a maximum of 2 backlog concurrent connection "requests at the same time (excluding connections that have been taken over by applications) -- stored in the syn_backlog and backlog queues respectively -- the length of each queue is the backlog value. The syn_backlog queue stores connections in the SYN_ACK state, while the backlog stores connections in the ESTABLISHED State that have not been taken over by the application.

? The syn_backlog queue is actually a hash table, and the size of the hash table is the power of 2. Therefore, the length of the actual syn_backlog queue must be slightly greater than the backlog value set by the Application -- Take the power value corresponding to 2.

? When the backlog value is small and the concurrent connection request is too high during the peak period, the latency of the three-way handshake network established by tcp will become the bottleneck-when the concurrent connection is too high, the syn_backlog queue will be filled'Can't connect'Error. At this point, increasing the throughput of the server application does not work, because the connection has not yet been established, the server application cannot take over and process these connections-but needs to increase the backlog value (length of the syn_backlog Queue) to alleviate this problem.

? Enabling syncookie can also solve'Can't connect'Problem, but it will cause TIME_WAIT status fallback to maintain 2MSL time. During peak periods, the client cannot connect to the server due to reusable connections (tcp connection multiplexing is based on <src_ip, src_port, dst_ip, dst_port> the values of the four tuples must be different. for accessing the same target server, the <src_ip, dst_ip, dst_port> values remain unchanged. Therefore, the number of available connections is limited to only the number allowed by src_port, the same src_port connection in the TIME_WAIT status cannot be reused. The Linux system is even stricter. Only the <src_ip, src_port, dst_ip> Triple...) is used ...). Therefore, syncookie is not recommended.

Net. ipv4.tcp _ orphan_retries = 0

How many times did the local end retry before trying to close the TCP connection. The default value is 7, which is equivalent to 50 seconds ~ 16 minutes (depending on RTO ). If your machine is a heavy load WEB server, you should consider reducing this value, because such a socket will consume a lot of important resources. See tcp_max_orphans

Net. ipv4.tcp _ sack = 1

SACK (SelectiveAcknowledgment, selective validation) technology, so that TCP only resends the packet lost during the interaction process, do not send all subsequent packets, it also provides a mechanism for the recipient to tell the sender which data is lost, which data is resold, and which data has been received in advance. This greatly improves the efficiency of data interaction between the client and the server.

Net. ipv4.tcp _ reordering = 3

Net. ipv4.tcp _ ecn = 2

Net. ipv4.tcp _ dsack = 1

TCP is allowed to send "two identical" SACK messages.

Net. ipv4.tcp _ mem = 178368 237824 356736

There are also three values, meaning:
Net. ipv4.tcp _ mem [0]: below this value, TCP has no memory pressure.
Net. ipv4.tcp _ mem [1]: Enter the memory pressure stage.
Net. ipv4.tcp _ mem [2]: higher than this value, TCP rejects socket allocation.

Net. ipv4.tcp _ wmem = 4096 16384 4194304

TCP write buffer, which can be referenced in the optimization value: 8192436600 873200

Net. ipv4.tcp _ rmem = 4096 87380 4194304

TCP read buffer, which can be referenced in the optimization value: 32768 436600 873200

Net. ipv4.tcp _ app_win = 31

Net. ipv4.tcp _ adv_win_scale = 2

Net. ipv4.tcp _ tw_reuse = 1

Indicates that reuse is enabled. Allow TIME-WAITsockets to be re-used for a New TCP connection. The default value is 0, indicating that the TCP connection is disabled;

Net. ipv4.tcp _ frto = 2

Turn on the F-RTO, an enhanced recovery algorithm for TCP Retransmission timeout (RTOs. In a wireless environment, group loss is typically caused by random radio interference rather than intermediate router group congestion. Refer to RFC 4318 for more details.

This file has one of the following values:

? 0.

? 1. Enable the basic version of The F-RTO algorithm.

? 2. Enable the SACK-enhanced F-TRO algorithm if the stream uses SACK. But when using SACK is the basic version can also be used, because there is such a scenario, F-RTO and open sack tcp flow group count cooperation is not good.

Net. ipv4.tcp _ frto_response = 0

When the F-RTO detects that TCP timeout is pseudo (for example, avoiding timeout by setting a longer timeout value), TCP has several options to decide what to do next. Possible values:

? 1. Based on the speed halved; the smooth and Conservative response leads to a congestion window (cwnd) and slow start threshold (ssthresh) after an RTT to be halved.

? 2. Very conservative response; this is not recommended because it is effective immediately and it does not interact well with other parts of TCP; it reduces the number of congestion windows (cwnd) and slow start threshold values (ssthresh) immediately ).

? 3. Intrusive response; discard unnecessary congestion control measures that are currently known (or a retransmission that will cause TCP to be more cautious and conservative); and recover cwnd and ssthresh to the value before timeout.

Net. ipv4.tcp _ slow_start_after_idle = 1

Indicates that the congestion window is still valid after a period of idle time without re-initialization.

Net. ipv4.tcp _ low_latency = 0

Allow TCP/IP protocol stack to adapt to low latency in high throughput; this option should be disabled.

Net. ipv4.tcp _ no_metrics_save = 0

After a tcp connection is closed, the existing parameters of the connection, such as the slow start threshold snd_sthresh, the congestion window snd_cwnd, And the srtt, are saved to dst_entry, as long as the dst_entry is not invalid, you can use the saved parameters to initialize the connection when you create a new connection. if the value of tcp_no_metrics_save is set to 1, these parameters (experience values) are not maintained. After each connection is established, repeat them. I think there is no benefit. therefore, the system sets it to 0 by default.

Net. ipv4.tcp _ moderate_rcvbuf = 1

Enabled the TCP memory automatic adjustment function (1 is enabled, 0 is disabled)

Net. ipv4.tcp _ tso_win_divisor = 3

The ratio of congestion Windows that can be consumed by a single TSO segment. The default value is 3.

Net. ipv4.tcp _ congestion_control = cubic

Net. ipv4.tcp _ available_congestion_control = cubic reno

Net. ipv4.tcp _ allowed_congestion_control = cubic reno

Packet loss causes a significant reduction in TCP transmission speed mainly because of the packet loss retransmission mechanism, which is controlled by the TCP congestion control algorithm.Congestion (congestion)

Several TCP congestion control algorithms are provided in the Linux kernel. If you have added them to the kernel, you can see the following through the Kernel Parameter net. ipv4.tcp _ available_congestion_control:

If a module is not loaded into the kernel, it can be loaded using modprobe. These algorithms are applicable to different environments.

?Reno is the most basic congestion control algorithm and the prototype of TCP protocol.

?Bic is applicable to scenarios with high rtt but extremely rare packet loss, such as lines between North America and Europe. This is the default Algorithm for Linux Kernel Between 2.6.8 and 2.6.18.

?Cubic is a modified version of bic. It is applicable to a wider environment than bic. It is the default algorithm of Linux kernel after 2.6.19.

?Hybla is suitable for networks with high latency and packet loss rate, such as satellite links.

Load tcp_hybl module modprobe tcp_hybla

The TCP congestion control algorithm has a great impact on the TCP transmission rate.

Net. ipv4.tcp _ abc = 0

Net. ipv4.tcp _ mtu_probing = 0

Net. ipv4.tcp _ fastopen

The GoogleTFO feature is supported by kernel 3.6 or later. For details about the implementation method, refer to the Google TFO feature in this document.

Net. ipv4.tcp _ base_mss = 512

The initial value of search_low used in the group layer path MTU detection. If MTU test is allowed, the initial value is the initial MSS value used for the connection.

Net. Route 4.route. min_adv_mss = 256

This file indicates the minimum MSS (MaximumSegment Size) Size, depending on the first hop router MTU.

Net. ipv4.tcp _ workaround_signed_windows = 0

Net. ipv4.tcp _ dma_copybreak = 4096

Lower limit. In bytes. The socket size will be detached to a dma replication engine. If there is a configuration in the system and kernel, use the config_net_dma option.

Net. ipv4.tcp _ max_ssthresh = 0

When the current congestion window value is smaller than the slow start threshold (snd_ssthresh), the slow start stage is called the slow start stage.

When we receive a new ACK, we will call the tcp_slow_start () function and Add 1 to the congestion window. (the value of the congestion window in Linux indicates the number of packets, rather than the actual number of packets sent.

The number of bytes. The actual number of bytes that can be sent is equal to the number of data packets that can be sent * MSS .)

Data packet loss occurs during the slow start phase.

After the tcp_max_ssthresh parameter is introduced, the increase frequency of the congestion window in the slow start stage can be controlled.

This parameter is disabled by default. If the value of this parameter is set to 1000, when the congestion window value is greater than 1000,

Then, no ACK is received, and no congestion window unit is added. Instead, a window unit is added only when two ACK packets are received. 2ACK is not the final value !!

It is determined based on the current congestion window value, tcp_max_ssthresh value.

Net. ipv4.tcp _ thin_linear_timeouts = 0

The RTO timeout handler. If it is a thin stream, do not set a new RTO to twice the original one.

Net. ipv4.tcp _ thin_dupack = 0

Same as tcp_thin_linear_timeouts

Net. core. netdev_max_backlog = 300

Enter the maximum device queue of the package. The default value is 300. This value is too low for heavy load servers and can be adjusted to 1000.

Ip link set eth0mtus 1500

Sets the mtu size of the NIC.

IP

Net. ipv4.ip _ local_port_range = 1024 65000

Indicates the port range used for external connection. The default value is small: 32768 to 61000, Which is changed to 1024 to 65000.

Net. ipv4.ip _ conntrack_max = 655360

In the kernel memory, netfilter can simultaneously process the "task" (connection trace entry) another

# Avoid amplification attacks

Net. ipv4.icmp _ echo_ignore_broadcasts = 1

# Enable malicious icmp error message protection

Net. ipv4.icmp _ ignore_bogus_error_responses = 1

# Enable SYN Flood Protection

Net. ipv4.tcp _ syncookies = 1

# Enable and record spoofing, source routing, and redirection packets

Net. ipv4.conf. all. log_martians = 1

Net. ipv4.conf. default. log_martians = 1

# Handling passive Route packets

Net. ipv4.conf. all. accept_source_route = 0

Net. ipv4.conf. default. accept_source_route = 0

# Enable reverse path Filtering

Net. ipv4.conf. all. rp_filter = 1

Net. ipv4.conf. default. rp_filter = 1

# Ensure that no one can modify the route table

Net. ipv4.conf. all. accept_redirects = 0

Net. ipv4.conf. default. accept_redirects = 0

Net. ipv4.conf. all. secure_redirects = 0

Net. ipv4.conf. default. secure_redirects = 0

# Do not act as a router

Net. ipv4.ip _ forward = 0

Net. ipv4.conf. all. send_redirects = 0

Net. ipv4.conf. default. send_redirects = 0

# Enable execshild

Kernel.exe c-shield = 1

Kernel. randomize_va_space = 1

Network related part (/sys)

Sys/class/net/eth0/statistics. rx_packets:

Received data packet

Sys/class/net/eth0/statistics. tx_packets:

Number of transmitted packets

Sys/class/net/eth0/statistics. rx_bytes:

Number of bytes received

Sys/class/net/eth0/statistics. tx_bytes:

Number of transmitted bytes

Sys/class/net/eth0/statistics. rx_dropped:

Discarded packet during packet reception

Sys/class/net/eth0/statistics. tx_dropped:

Discarded packet during packet sending