Using TC to control server traffic under Linux

TC Introduction

In Linux, TC has two kinds of control methods CBQ and HTB. HTB is designed to replace CBQ. HTB is more flexible than CBQ, but CPU overhead is also greater, and usually high-speed links use CBQ, generally HTB used more broadly. HTB's rules are essentially a tree-like structure, consisting of three basic constituent blocks: The queue specifies Qdisc (queueing discipline), classes (Class), and Classifiers (classifiers).

Qdisc QueueRules (queueing discipline):

It is used to control the transmission speed of the network. Through the queue, Linux can cache network packets and then, based on the user's settings, smooth network traffic as far as possible without interrupting connections (such as TCP). It is important to note that Linux does not control the receiving queue well enough, so we generally use only the send queue, That is, "control of the non-control." It encapsulates the other two main TC components (classes and classifiers). If the kernel needs to send packets through a network interface, it needs to queue the packets according to the QDISC queue rules configured for that interface. Then, the kernel takes as much of the packets out of the qdisc as possible, Give them to the network adapter driver module.

The simplest qdisc is PFIFO it does not do any processing of incoming packets, the packets are queued in first-in, first-out way. However, it saves packets that the network interface cannot process for a while. Common queue rules include FIFO first-out, RED random early detection, SFQ Random Fair queue and Token bucket token bucket, class base queue CBQ,CBQ is a super queue, that is, it can contain other queues, and even other CBQ.

class

Class is used to represent a control strategy. Obviously, many times, we will probably have to implement different traffic control strategies for different IPs, when we have to use different classes to represent different control strategies.

Filter Rule

Filter is used to draw the user into a specific control strategy (that is, a different class). For example, now that we want to implement a different control strategy (a, B) for XXA,XXB Two IP, we can use the filter to xxa into control policy A, XXB into control policy B, The flag bits divided by the filter can be implemented by U32 marking function or IPtables Set-mark (mostly using IPtables to mark).
Currently, the TC can be used by the filter has: Fwmark classifier, u32 classifier, based on the routing of the Classifier and RSVP classifier (for IPV6, IPV4), etc., wherein the fwmark classifier allows us to use the Linux netfilter code to select traffic, and U32 The classifier allows us to select the traffic based on any header. It should be noted that filter (filter) is inside qdisc and they cannot be used as the main body.

Application flow of TC

If the HTB hierarchy is considered a tree, then each node is called a class, each class can be set to a Qdisc, the default is Tc-pfifo. In addition HTB can also set up some filters, through which the incoming package can be distributed to the specified class. Here the filter is usually mounted on the root node (Eth0 interface), but the match can only match the leaf node.
Packet->iptables (when passing iptables, Iptables sets different mark)->TC (Class)->TC (queue) based on different IP

Application
Assume that the eth0 bit is the external network interface of the server. Before you begin, clear the Eth0 all queue rules first

TC Qdisc del dev eth0 root 2>/dev/null >/dev/null

1) define the topmost (root) queue rule and specify the default category number

TC Qdisc Add dev eth0 root handle 1:HTB default 2

2) define the 1:1 categories (speed) of the first layer
It was supposed to be more of a second-level leaf category, but for the time being, it could be in this application.

TC class Add dev eth0 parent 1:1 classid 1:2 HTB rate 98mbit ceil 100mbit prio 2
TC class Add dev eth0 parent 1:1 classid 1:3 HTB rate 1mbit ceil 2mbit prio 2

Note: The above is our control output server speed, one for 98M, one for 2M.
Rate: is the bandwidth value guaranteed by a class. If there is more than one class, ensure that the sum of all child classes is less than or equal to the parent class.
Prio: Used to indicate the competitiveness of borrowing bandwidth, the smaller the prio, the higher the priority, the stronger the competitiveness.
Ceil:ceil is the maximum bandwidth value a class can get.

At the same time, in order not to cause a session to occupy bandwidth, add the fair queue Sfq immediately.

TC Qdisc Add dev eth0 parent 1:2 handle 2:SFQ perturb 10
TC Qdisc Add dev eth0 parent 1:3 handle 3:SFQ perturb 10

3) Set Filter
Filters can use their own u32 can also use Iptables to mark
Specify that in the root class 1:0, the filter for 192..168.0.2, using the 1:2 rule, to give him 98M speed, the writing is as follows

TC Filter Add dev eth0 protocol IP parent 1:0 u32 match ip src 192.168.0.2 flowid 1:2
TC Filter Add dev eth0 protocol IP parent 1:0 u32 match ip src 192.168.0.1 flowid 1:3

If all IP is written as:

TC Filter Add dev eth0 protocol IP parent 1:prio u32 match IP DST 0.0.0.0/0 flowid 1:10

Use Iptables to match filters
You can also use this method, but the following iptables commands are required to mark the

TC Filter Add dev eth0 parent 1:protocol IP prio 1 handle 2 FW flowid 1:2
TC Filter Add dev eth0 parent 1:protocol IP prio 1 handle 2 FW flowid 1:3

Iptables as long as the mark on the line, this time mostly used in the gateway, it is appropriate to use this method.

Iptables-t mangle-a postrouting-d 192.168.0.2-j MARK--set-mark 10
Iptables-t mangle-a postrouting-d 192.168.0.3-j MARK--set-mark 20

TC Control of High speed
Rate Ceiling speed limit
The parameter ceil specifies the maximum bandwidth a class can use to limit how much bandwidth a class can borrow. The default ceil is the same as the rate, which is useful for ISPs because they generally limit the total number of users being serviced even if other users do not request services. (ISPS very much want the user to pay more money to get better service), note Root class is not allowed to be borrowed, so did not specify Ceil.
Note: The value of ceil should be at least as high as the rate at which it resides, meaning that ceil should be at least as high as any one of its subclasses

Burst Burst
Network hardware can only send one package at a time this depends only on the rate of one hardware. Link sharing software can take advantage of this ability to dynamically generate multiple connections running at different speeds. So rates and ceil are not an immediate measure but a mean of sending packets in a single time. The real situation is how to make a class with a very small amount of traffic available to other classes at the maximum rate for a certain time class. The burst and Cburst parameters control how much data can be sent effortlessly to other classes of need at the maximum speed of the hardware.
If the Cburst is less than a theoretical packet, the burst does not exceed the ceil rate, and the same method TBF the highest rate.
You may ask why bursts is needed. Because it can be easily raised to the speed on a very congested link. For example, WWW traffic is burst. You visit the homepage. Bursts of acquisition and reading. In the idle time burst will again "charge" once.
Note: Burst and cburst are at least as large as the values of their subclasses.

TC Command Format:
TC Qdisc [Add | change | replace | link] Dev dev [parent Qdisc-id | root] [handle Qdisc-id] Qdisc [Qdisc specific P Arameters]

TC Qdisc ... Dev dev (parent classid | root) [handle major:] HTB [Default Minor-id]

parameters of the Qdisc:  

parent Major:minor or root. A qdisc is the root node, otherwise the parent is specified in the other case. Where Major:minor is the handle ID of class, each class specifies an ID for identification.

handle Major: , this syntax is a bit strange, is optional, if qdisc below also to classify (more than one class), you need to specify this Hanlde. For Root, it is usually "1:".

Note: For Qdiscs and classes in TC commands, the syntax for identity handle (CLASSID) is x:y, where x is an integer used to identify a qdisc,y is an integer that identifies the class that belongs to the Qdisc. The Y value of the Qdisc handle must be 0,class and the Y value of handle must be non-0. The "1:0" is usually abbreviated as "1:", which is the notation shown above.

Default Minor-id, the traffic that is not categorized (which cannot match the filter) is sent to the class specified by the minor (class ID is Major:minor-id).

TC class ... dev dev parent major:[minor] [classid Major:minor] HTB rate rate [ceil rate] Burst bytes [Cburst bytes] [Prio Priority]

parameters of class:

The parent Major:minor, which specifies the parents of the class, the parent node can be either Qdisc or class, and if it is qdisc, it is not necessary to specify minor, which is a required parameter.

classid Major:minor, ClassID as the class identifier, this is optional. If this class has no child nodes, it can be unspecified. Major is the handle of the Father Qdisc.

Prio Low-priority class will match first

rate of this class and all its subclasses

Ceil If the parent has free bandwidth, the maximum can be assigned to the value of the current class, and the default is the same as rate.

Burst allows the number of bytes sent at ceil rate, at least the same as the burst maximum for subclasses.  

Cburst allows the number of bytes sent at the highest rate of the port, at least the same as the Cburst maximum for the subclass. Functions similar to the peakrate in TBF, when this value is limited to a very small amount of time, the burst traffic can be avoided to avoid an instantaneous rate exceeding ceil.

Quantum the number of bytes that the current class can send per round, the default calculation quantum = rate/r2q. Quantum must be greater than 1500 less than 60000. Quantum is used only when the class traffic exceeds the rate but does not exceed ceil.  The smaller the quantum, the better the bandwidth sharing effect. R2q is used to calculate the default of quantum,r2q is 10.

TC Filter [Add | change | replace] Dev dev [parent Qdisc-id | root] Protocol protocol Prio Priority FilterType [Filte Rtype specific parameters] Flowid Flow-id

Show
TC [-S |-d] qdisc show [Dev Dev]
TC [-S |-d] class show Dev dev tc filter show Dev Dev

View the status of TC
Tc-s-D qdisc Show Dev eth0
Tc-s-D class show Dev eth0

Remove TC Rule

TC Qdisc del Dev eth0 root

Instance

Speed control of a single IP using TC

TC Qdisc Add dev eth0 root handle 1:htb r2q 1
TC class Add dev eth0 parent 1:classid 1:1 HTB rate 30mbit ceil 60mbit
TC Filter Add dev eth0 parent 1:protocol IP prio u32 match IP DST 192.168.1.2 flowid 1:1

You can limit the download speed of 192.168.1.2 to 30Mbit up to 60Mbit, where r2q refers to the root without default, so that the bandwidth of the entire network is unlimited

speed control of entire IP with TC

TC Qdisc Add dev eth0 root handle 1:htb r2q 1
TC class Add dev eth0 parent 1:classid 1:1 HTB rate 50mbit ceil 1000mbit
TC Filter Add dev eth0 parent 1:protocol IP prio u32 match IP DST 192.168.111.0/24 flowid 1:1

can limit 192.168.111.0 to 255 of the bandwidth of 3000k, the actual download speed of about 200k.
In this case, all the machines in this segment share the 200k bandwidth.
You can also add a SFQ (random fair queue)

TC Qdisc Add dev eth0 root handle 1:htb r2q 1
TC class Add dev eth0 parent 1:classid 1:1 HTB rate 3000kbit Burst 10k
TC Qdisc Add dev eth0 parent 1:1 handle 10:SFQ perturb 10
TC Filter Add dev eth0 parent 1:protocol IP prio u32 match IP DST 192.168.111.168 flowid 1:1

SFQ, he can prevent an IP in a segment from taking up the entire bandwidth.

Using the TC to control the server external speed is 10M

As below, I want to manage a server, only outgoing 10M of data

TC Qdisc del Dev eth0 root
TC Qdisc Add dev eth0 root handle 1:HTB
TC class Add dev eth0 parent 1:classid 1:1 HTB rate 100mbit ceil 100mbit
TC class Add dev eth0 parent 1:1 classid 1:10 HTB rate 10mbit ceil 10mbit
TC Qdisc Add dev eth0 parent 1:10 sfq perturb 10
TC Filter Add dev eth0 protocol IP parent 1:prio 2 u32 match IP dst 220.181.xxx.xx/32 flowid 1:1 # above this one, let 220.181.xxx.xx/ 32 This run is the default, mainly to let this IP connection come in without being controlled
TC Filter Add dev eth0 protocol IP parent 1:prio u32 match IP DST 0.0.0.0/0 flowid 1:10 # By default let all traffic pass from this

Reference: http://blog.chinaunix.net/u3/94771/showart_1906064.html

http://liuleijsjx.javaeye.com/blog/402152

Article excerpt from: http://www.php-oa.com/2009/06/23/linux_tc.html