Server flow control method using TC in Linux environment

Company currently has a server, network environment is too high, that server and source server connection download, ran to 400m-500m, in order to control, so studied the TC, to do flow control. Give him a little bit of control, don't let this one take up all the networks. TC is very strong, many so-called hardware routers, are based on this.

The server flow control method using TC under Linux

1) Assume that the eth0 bit is the server's extranet network interface, first in the eth0 of the Qdisca,qdisca control through the local to the external network speed, so can be used to control the server data transmission speed.

#tc qdisc add dev eth1 root handle 1:htb default 1

Add set interface topmost handle (mark used) tag defaults to 1 class

Explained as follows: whether the queue, or class and filter have IDs, and so on, generally have parent (parent, upper level), note that the ID has interface local, different network interfaces can have the same ID. For here because Qdisc at the top, so the parent is none, with ' Root ' to identify, ID with 1: to flag

' Default 91′ means that when an IP stream does not meet any of the set filter rules, it is automatically grouped into Class 1. Please refer to the manual for more detailed instruction rule instructions.

2) and then create two classes under Qdisc to specify the speed of eth0 control through the local to the extranet

#tc class Add dev eth0 parent 1:0 classid1:30 HTB rate 2mbit ceil 4mbit prio 2

Note: The above is our control output server speed, 2M, maximum can to 4M

Rate: is a class guaranteed to get the bandwidth value. If you have more than one class, make sure that the sum of all subclasses is less than or equal to the parent class.

Prio: Used to indicate the competitiveness of borrowing bandwidth, the smaller the prio, the higher the priority, the stronger the competitiveness.

Ceil:ceil is the maximum available bandwidth value for a class.

3) Then set different classes for different applications under each root class, as shown below. (If there is only one class, this is not necessary)

#tc class Add dev eth0 parent 1:30 classid 1:31 htbrate 0.5mbit ceil 2mbit prio 3

At the same time, in order not to allow a session to be permanently occupied, the node (that is, the application nodes in this article) is added to the fair Queue Sfq. (multiple IP)

#tc Qdisc Add dev eth0 parent 1:31 handle 31:SFQ perturb 10

4) then add the filter.

#tc filter Add dev eth0 parent 1:protocol Ipprio Handle-FW Flowid 1:31

5) with iptable marking, you can also use u32 and the like

#iptables-T Mangle-i forward-i! Eth1-p tcp–sport 80-s xxx.xxx.xxx.xxx–j Mark–set-mark 31

Second, TC to the most high speed control Rate ceiling rate limit

Parameter ceil Specifies the maximum bandwidth that a class can use to limit how much bandwidth a class can borrow. The default ceil is the same as the rate, which is useful for ISPs because they generally limit the total number of users that are being serviced even if other users do not request a service. (ISPS very much want users to pay more money to get better service), note that the root class is not allowed to be borrowed, so did not specify Ceil.

(Note: The Ceil value should be at least as high as the class it is in, meaning that the ceil should be at least as high as any of its subclasses.) ）

Burst Burst

Network hardware can only send a package at one time this depends only on the rate of one hardware. Link sharing software can take advantage of this ability to dynamically generate multiple connections running at different speeds. So the rate and ceil are not an instant metric but a mean of sending packets in a single time. The actual situation is how to make a class with very little traffic available at the maximum rate to other classes at a time class. The burst and Cburst parameters control how much data can be sent effortlessly to other classes as required by the hardware's maximum speed.

If the Cburst is less than a theoretical packet, the burst does not exceed the ceil rate, and the same method TBF the highest rate.

You may ask why you need bursts. Because it can be easily raised to the speed of a very crowded link on the road. For example, the WWW traffic is abrupt. You visit the homepage. Burst to gain and read. In the idle time burst will again "charge" once.

(Note: Burst and cburst must be at least as large as the values of their subclasses.) ）

Third, TC command format:

Join

TC Qdisc [Add change replace link] Dev Dev [parent qdisc-id root] [handle Qdisc-id] Qdisc [qdisc specific parameter S]

TC class [Add change Replace] Dev Dev parent qdisc-id [classid Class-id] qdisc [qdisc specific parameters]

TC Filter [Add change replace] Dev Dev [parent Qdisc-id Root] Protocol protocol Prio Priority FilterType [FilterType Specific parameters] Flowid Flow-id

Show

TC [-s-d] qdisc show [Dev Dev]

TC [-s-d] class show Dev dev tc filter show Dev Dev

View the status of TC

Tc-s-D qdisc Show Dev eth0

Tc-s-D class show Dev eth0

Delete TC Rule

TC Qdisc del Dev eth0 root

Instance

First, download

Download limit single IP

TC Qdisc Add dev eth0 root handle 1:htb r2q 1

TC class Add dev eth0 parent 1:classid 1:1 HTB rate 30mbit ceil 60mbit

TC Filter Add dev eth0 parent 1:protocol IP prio u32 match IP DST 192.168.1.2 flowid 1:1

You can limit the download speed of 192.168.1.2 to 30Mbit Max 60Mbit

r2q, which means no default root, makes the entire network bandwidth Unlimited

Download whole segment IP

TC Qdisc Add dev eth0 root handle 1:htb r2q 1

TC class Add dev eth0 parent 1:classid 1:1 HTB rate 50mbit ceil 1000mbit

TC Filter Add dev eth0 parent 1:protocol IP prio u32 match IP DST 192.168.111.0/24 flowid 1:1

can limit 192.168.111.0 to 255 of the bandwidth of 3000k, the actual download speed of about 200k.

In this case, the 200k bandwidth is shared by all machines in this network segment.

You can also add a SFQ (random fair queue)

TC Qdisc Add dev eth0 root handle 1:htb r2q 1

TC class Add dev eth0 parent 1:classid 1:1 HTB rate 3000kbit Burst 10k

TC Qdisc Add dev eth0 parent 1:1 handle 10:SFQ perturb 10

TC Filter Add dev eth0 parent 1:protocol IP prio u32 match IP DST 192.168.111.168 flowid 1:1

SFQ, he can prevent one IP in a segment from taking up the entire bandwidth.

Introduction to TC

In Linux, there are two ways to control TC CBQ and HTB.HTB are designed to replace CBQ. It is a layered filtering framework. The TC consists of three basic constituent blocks: the queue rules Qdisc (queueing discipline), classes (Class), and Classifiers (classifiers).

Queue (queueing discipline): used to control the speed of the network to send and receive. Through queues, Linux can cache network packets, and then smooth network traffic based on the user's settings, as far as possible without interrupting the connection (such as TCP). It should be noted that the Linux control of the receiving queue is not good enough, so we generally only use the send queue, that is, "control issued by the charge." It encapsulates the other two main TC components (classes and classifiers). If the kernel needs to send packets through a network interface, it will need to queue the packets according to the Qdisc (queuing rules) configured for the interface. The kernel then takes as many packets from the Qdisc as possible and hands them to the network adapter driver module.

The simplest qdisc is PFIFO it does not do any processing to the incoming packets, and the packet takes the first in first out way through the queue. However, it saves packets that the network interface cannot handle for a while.

Queue rules include FIFO (first-in first Out), RED (random early detection), SFQ (random fair queue) and Token bucket (Token Bucket), class base queue (CBQ), and CBQ is a super queue that can contain other queues (or even other CBQ).

Class is used to represent a control policy. Obviously, many times, we are likely to implement different traffic control strategies for different IPs, at which point we have to use different class to represent different control strategies.

Filter is used to delimit a user into a specific control policy (that is, in a different Class). For example, now we want to implement a different control strategy for the XXA,XXB two IP (a,b), at which point we can use filter to enter XXA into control Strategy A, the XXB into the control strategy B,filter division of the flag bit can be used u32 marking function or iptables Set-mark (mostly using iptables to mark) functions to achieve.

Currently, the TC can use the filter has: Fwmark classifier, u32 classifier, based on the routing classifier and RSVP classifier (respectively for IPV6, IPV4), etc., where the Fwmark classifier allows us to use the Linux netfilter code to select traffic, The U32 classifier allows us to select the traffic based on any headers. It should be noted that the filter (filter) is inside the Qdisc and cannot be used as a body.

Packet->iptables (when passing iptables, Iptables sets different mark)->TC (Class)->TC (queue) based on different IP.