TC Flow Control of QoS implementation

TC Introduction

In Linux, TC has two kinds of control methods CBQ and HTB. HTB is designed to replace the CBQ. It is a hierarchical filtering framework.

The TC consists of three basic constituent blocks:

Queue rules Qdisc (queueing discipline), Class (classes), and classifiers (classifiers)

Queue in TC (queueing discipline):
Used to control the network transmission and delivery speed. Through queues, Linux can cache network packets and then, depending on the user's settings, try not to interrupt connections (such as TCP) The premise of smoothing the network traffic. It is important to note that Linux does not control the receiving queue well enough, so we generally only use the send queue, that is, "no control." It encapsulates the other two main TC components (classes and classifiers). If the kernel needs to send packets through a network interface, it needs to queue the packets according to the Qdisc (queuing rules) configured for that interface. The kernel then extracts the packets from the Qdisc as much as possible, handing them over to the network adapter driver module.

The simplest qdisc is PFIFO it does not do any processing of incoming packets, and the packets are queued in first-in, first-out way. However, it saves packets that the network interface cannot handle for a while.
The queue rules include FIFO (first-in, pre-out), RED (random early detection), SFQ (random fair queue) and Token bucket (tokens bucket), class-base queue (CBQ), CBQ is a super queue, that is, it can contain other queues (and even other CBQ).

Class classes in TC
Class is used to represent a control strategy. Obviously, many times, we will probably have to implement different traffic control strategies for different IPs, when we have to use different classes to represent different control strategies.

Filter rule in TC
Filter is used to draw the user into a specific control strategy (that is, a different class). For example, now that we want to implement a different control strategy (a, B) for XXA,XXB Two IP, we can use the filter to xxa into control policy A, XXB into control policy B, The flag bits divided by the filter can be implemented by U32 marking function or IPtables Set-mark (mostly using IPtables to mark).
Currently, the TC can be used by the filter has: Fwmark classifier, u32 classifier, based on the routing of the Classifier and RSVP classifier (for IPV6, IPV4), etc., wherein the fwmark classifier allows us to use the Linux netfilter code to select traffic, and U32 The classifier allows us to select the traffic based on any header. It should be noted that filter (filter) is inside qdisc and they cannot be used as the main body.

Application flow of TC

Packet->iptables (when passing iptables, Iptables sets different mark)->TC (Class)->TC (queue) based on different IP

Application
Assume that the eth0 bit is the server's extranet network interface.
Before you begin, clear the Eth0 all queue rules first

TC Qdisc del dev eth0 root 2>/dev/null >/dev/null

1) define the topmost (root) queue rule and specify the default category number

TC Qdisc Add dev eth0 root handle 1:HTB default 2

2) define the 1:1 categories (speed) of the first layer
It was supposed to be more of a second-level leaf category, but for the time being, it could be in this application.

TC class Add dev eth0 parent 1:1 classid 1:2 HTB rate 98mbit ceil 100mbit prio 2

TC class Add dev eth0 parent 1:1 classid 1:3 HTB rate 1mbit ceil 2mbit prio 2

Note: The above is our control output server speed, one for 98M, one for 2M.
Rate: is the bandwidth value guaranteed by a class. If there is more than one class, ensure that the sum of all child classes is less than or equal to the parent class.
Prio: Used to indicate the competitiveness of borrowing bandwidth, the smaller the prio, the higher the priority, the stronger the competitiveness.
Ceil:ceil is the maximum bandwidth value a class can get.

At the same time, in order not to cause a session to occupy bandwidth, add the fair queue Sfq immediately.

TC Qdisc Add dev eth0 parent 1:2 handle 2:SFQ perturb 10

TC Qdisc Add dev eth0 parent 1:3 handle 3:SFQ perturb 10

3) Set Filter
Filters can use their own u32 can also use Iptables to mark
Specify that in the root class 1:0, the filter for 192..168.0.2, using the 1:2 rule, to give him 98M speed, the writing is as follows

TC Filter Add dev eth0 protocol IP parent 1:0 u32 match ip src 192.168.0.2 flowid 1:2

TC Filter Add dev eth0 protocol IP parent 1:0 u32 match ip src 192.168.0.1 flowid 1:3

If all IP is written as

TC Filter Add dev eth0 protocol IP parent 1:prio u32 match IP DST 0.0.0.0/0 flowid 1:10

Use Iptables to match filters
You can also use this method, but the following iptables commands are required to mark the

TC Filter Add dev eth0 parent 1:protocol IP prio 1 handle 2 FW flowid 1:2

TC Filter Add dev eth0 parent 1:protocol IP prio 1 handle 2 FW flowid 1:3

Iptables just tick the mark.

Iptables-t mangle-a postrouting-d 192.168.0.2-j MARK--set-mark 10

Iptables-t mangle-a postrouting-d 192.168.0.3-j MARK--set-mark 20

TC control of the most high speed
Rate Ceiling speed limit
The parameter ceil specifies the maximum bandwidth that a class can use to limit how much bandwidth a class can borrow. The default ceil is the same as the rate
This feature is useful for ISPs because they generally limit the total number of users being serviced even if other users do not request services. (ISPS very much want the user to pay more money to get better service), note Root class is not allowed to be borrowed, so there is no designation Ceil
Note: The value of ceil should be at least as high as the rate at which it resides, meaning that ceil should be at least as high as any one of its subclasses

Burst Burst
Network hardware can only send one package at a time this depends only on the rate of one hardware. Link sharing software can take advantage of this ability to dynamically generate multiple connections running at different speeds. So rates and ceil are not an immediate measure but a mean of sending packets in a single time. The real situation is how to make a class with a very small amount of traffic available to other classes at the maximum rate for a certain time class. The burst and Cburst parameters control how much data can be sent effortlessly to other classes of need at the maximum speed of the hardware.
If the Cburst is less than a theoretical packet, the burst does not exceed the ceil rate, and the same method TBF the highest rate.
You may ask why bursts is needed. Because it can be easily raised to the speed on a very congested link. For example, WWW traffic is burst. You visit the homepage. Bursts of acquisition and reading. In the idle time burst will again "charge" once.
Note: Burst and cburst are at least as large as the values of their subclasses.

TC Command Format:
Join
TC Qdisc [Add | change | replace | link] Dev dev [parent Qdisc-id | root] [handle Qdisc-id] Qdisc [Qdisc specific P Arameters]
TC class [Add | change | replace] Dev dev parent qdisc-id [classid Class-id] qdisc [qdisc specific parameters]
TC Filter [Add | change | replace] Dev dev [parent Qdisc-id | root] Protocol protocol Prio Priority FilterType [Filte Rtype specific parameters] Flowid Flow-id

Show
TC [-S |-d] qdisc show [Dev Dev]
TC [-S |-d] class show Dev dev tc filter show Dev Dev

View the status of TC
Tc-s-D qdisc Show Dev eth0
Tc-s-D class show Dev eth0

Remove TC Rule

TC Qdisc del Dev eth0 root

Instance

Limit the speed control of a single IP using TC download

TC Qdisc Add dev eth0 root handle 1:htb r2q 1

TC class Add dev eth0 parent 1:classid 1:1 HTB rate 30mbit ceil 60mbit

TC  Filter Add dev eth0 parent 1:protocol IP prio u32 match IP DST 192.168.1.2  flowid 1:1

You can limit the download speed of 192.168.1.2 to 30Mbit up to 60Mbit, where r2q refers to the root without default, so that the bandwidth of the entire network is unlimited

speed control of entire IP with TC

TC Qdisc Add dev eth0 root handle 1:htb r2q 1 TC class add dev eth0 parent 1:classid 1:1 HTB rate 50mbit ceil 1000mbit t C Filter Add dev eth0 parent 1:protocol IP prio u32 match IP DST 192.168.111.0/24 flowid 1:1

can limit 192.168.111.0 to 255 of the bandwidth of 3000k, the actual download speed of about 200k.
In this case, all the machines in this segment share the 200k bandwidth.
You can also add a SFQ (random fair queue)

TC Qdisc Add dev eth0 root handle 1:htb r2q 1 TC class add dev eth0 parent 1:classid 1:1 HTB rate 3000kbit Burst 10k TC Qdisc Add dev eth0 parent 1:1 handle 10:SFQ perturb TC filter Add dev eth0 parent 1:protocol IP prio u32 match IP DST 192.168.111.168 Flowid 1:1

SFQ, he can prevent an IP in a segment from taking up the entire bandwidth.

using the TC to control the server external speed is 10M

As below, I want to manage a server, only outgoing 10M of data

TC Qdisc del dev eth0 root tc qdisc add dev eth0 root handle 1:HTB TC class add dev  eth0 parent 1:classid 1:1 HTB RA Te  100mbit ceil 100mbit TC class Add dev  eth0 parent 1:1 classid 1:10 HTB rate 10mbit ceil 10mbit tc Qdisc Add dev  eth0 Parent 1:10 SFQ perturb TC Filter Add dev eth0 protocol IP parent 1:prio 2   u32 match IP DST 220.181.xxx. XX/32 Flowid 1:1 #  above this one, let 220.181.XXX.XX/32 this run default, mainly to let this IP connection come in not controlled TC filter add dev eth0 protocol IP parent 1:p Rio u32 match IP DST 0.0.0.0/0  flowid 1:10 # By default let all traffic pass from this

parameters of the Qdisc:

Parent Major:minor or root. A qdisc is the root node, otherwise the parent is specified in the other case. Where Major:minor is the handle ID of class, each class specifies an ID for identification.

Handle Major:, this syntax is a bit strange, is optional, if qdisc below also to classify (more than one class), you need to specify this Hanlde. For Root, it is usually "1:".

Note: For Qdiscs and classes in TC commands, the syntax for identity handle (CLASSID) is x:y, where x is an integer used to identify a qdisc,y is an integer that identifies the class that belongs to the Qdisc. The Y value of the Qdisc handle must be 0,class and the Y value of handle must be non-0. The "1:0" is usually abbreviated as "1:", which is the notation shown above.

Default Minor-id, the traffic that is not categorized (which cannot match the filter) is sent to the class specified by the minor (class ID is Major:minor-id). TC class ... dev dev parent major:[minor] [classid Major:minor] HTB rate rate [ceil rate] Burst bytes [Cburst bytes] [Prio Priority]

parameters of class:

The parent Major:minor, which specifies the parents of the class, the parent node can be either Qdisc or class, and if it is qdisc, it is not necessary to specify minor, which is a required parameter.

ClassID Major:minor,classid as the class identifier, this is optional. If this class has no child nodes, it can be unspecified. Major is the handle of the Father Qdisc.

Prio low-priority class will match first

Rate of this class and all its subclasses

Ceil if the parent has free bandwidth, the maximum can be assigned to the value of the current class, and the default is the same as rate.

Burst allows the number of bytes sent at ceil rate, at least the same as the burst maximum for subclasses.

Cburst allows the number of bytes sent at the highest rate of the port, at least the same as the Cburst maximum for the subclass. Functions similar to the peakrate in TBF, when this value is limited to a very small amount of time, the burst traffic can be avoided to avoid an instantaneous rate exceeding ceil.

Quantum the number of bytes that the current class can send per round, the default calculation quantum = rate/r2q. Quantum must be greater than 1500 less than 60000. Quantum is used only when the class traffic exceeds the rate but does not exceed ceil.  The smaller the quantum, the better the bandwidth sharing effect. R2q is used to calculate the default of quantum,r2q is 10.

TC Filter [Add | change | replace] Dev dev [parent Qdisc-id | root] Protocol protocol Prio Priority FilterType [Filte Rtype specific parameters] Flowid Flow-id

There are two kinds of QIDSC in Linux QoS, one is the Class (classful) Qdisc, the other is the Qdisc with no classification (classless), the qdisc of classification and non-classification can be the root qdisc of the device, But the classification of the QDISC through its classification (class) and can be grafted out sub-qdisc, sub-Qdisc can be categorized Qdisc can also be non-classified qdisc, the last leaf qdisc must be non-classified qdisc, generally used pfifo/ Bfifo as a leaf qdisc. Therefore, this combination can be used to combine complex scheduling methods with categorical qdisc and non-categorical qdisc.