General encoding mode for Linux TCP/IP protocol stack

From: http://blog.chinaunix.net/u/26185/showart_475934.html

 

Like other kernel functions, each network function is one of the kernel members. Therefore, it must use the memory reasonably and fairly,
CPU and other shared resources. The vast majority of features are not an independent program in the kernel, but are more or less affected by this function with other parts of the kernel. Therefore, they always try to use
Similar architecture to implement similar functions.
For many kernel components, some requirements are generic, such as allocating several instances to the same data structure, or tracking a Data Structure reference to avoid insecure memory redistribution. Next we will look at some common methods for Linux to solve these needs. We also talked about common coding techniques that may be encountered when viewing kernel encoding.

1. Cache

The kernel uses kmalloc and kfree to allocate and release memory. The usage of these two functions is similar to that of the user space functions malloc and free.
A kernel component usually needs to allocate multiple instances with one data structure. If allocation and release occur frequently, a special memory cache is usually allocated to the initialization function of the related kernel components (such as the fib_hash_init function in the routing subsystem) to accelerate memory allocation. When a memory block is released, it will be returned to the same memory cache for allocation.

The following are some network data structures that require the kernel to maintain the memory cache:

Socket buffer Descriptors

This cache is allocated by skb_init in net/CORE/sk_buff.c, which is used to allocate the sk_buff structure. The sk_buff structure may be the data structure with the highest distribution and release frequency in the network subsystem.

Neighboring protocol Mappings

The neighbor protocol uses the memory cache to allocate the neighbor structure, which stores the address ing between L3 and L2.

Routing tables

The routing code uses two memory caches to allocate two data structures, which define the route table.

The following are some functions used when using the memory cache:

Kmem_cache_create
Kmem_cache_destroy

Creates or destroys a cache.
Kmem_cache_alloc
Kmem_cache_free

Allocates or releases an object from the memory cache. They are usually called in a packaging function, which processes the allocation and release requests at a higher level.
. For example, the kfree_skb function processes the request to release the sk_buff, but only after all the references to this structure are released and the related subsystems (such as the firewall) have been cleared.
Call kmem_cache_free to release this sk_buff.

The limit on the number of instances that can be allocated from a given memory cache is usually specified in the kmem_cache_alloc function, but sometimes it can be adjusted through parameters in the/proc file system.

2. cache and hash table

It is common to use caching to improve performance. In network code, there are L3 to L2 ing caches (such as ARP cache in IPv4), route table caches, and so on.

Cache lookup functions usually have an input value to indicate whether a new element needs to be allocated and added to the cache if the cache lookup fails. Other types of lookup functions only add elements that do not have a hit.

Cache is usually implemented using a hash table. The kernel provides many data structures, such as one-way and two-way linked lists. These data structures can be directly used to implement simple hash tables.

The standard method for processing the same hash value is to put these elements into a linked list. However, traversing these linked lists usually takes longer time than searching for elements by using hash values. Therefore, use a hash function with a low probability of conflict.

If the query time of a hash table (whether or not it is used as a cache) is a key parameter of a sub-system, a mechanism should be implemented, increase the size of the hash table to reduce the average probability of conflict, which can reduce the average search time.

You can also go to other subsystems, such as neighboring.
Layer, you can see that by adding a random variable to the key value, the hash value can be evenly distributed in the cache bucket. This can reduce DoS (Denial
Service), because such DoS uses specific parameters to make the table items of the hash table concentrated on the same hash value.

3. Reference Counter

If a piece of code accesses a released data structure, the kernel will not be happy, and the user will not be happy with the reflection of the kernel. To avoid such annoyance
But also makes the garbage collection mechanism more convenient and efficient, and many data structures maintain a reference count. A good kernel programmer will increase or decrease this value each time he accesses a data structure.
Data Reference count. For those data structures that require reference counting, the corresponding kernel module that owns this data structure usually exports two functions to increase or decrease the reference counting. These functions are usually named
Xxx_hold (add reference count) and xxx_release (reduce reference count ). Sometimes, the function that reduces the reference count may also be named xxx_put (for example
Dev_put is used to reduce the reference count of the net_device structure ).

Although we assume that all kernel programmers are very serious, kernel programmers are also people, so they cannot always write code without bugs. Using reference counting is a simple but effective way to prevent the release of the data structures that are still in use. However, this method cannot solve all the problems. The following are the consequences of forgetting to increase or decrease the reference count:

• If you release a data structure but forget to call the xxx_release function, the kernel will never allow the release of this data structure (unless another bug code calls the function that reduces the reference count twice ). This will cause the memory to be gradually exhausted.

• If you reference a data structure but forget to call the xxx_hold function, at some point in time, you happen to be the only quote of this data structure.
  Data structures will be released in advance because you have not increased the reference count. This situation is more harmful than the previous one. If your subsequent operations attempt to reference this structure, other data will be damaged, or
  Cause the kernel to crash immediately.

If you want to release a data structure, you must first notify the referers of the structure to reduce the reference count of the structure. This can be achieved through notification chain.

In the following cases, you need to increase the reference count of the data structure:

• The two data structures are closely related. In this case, a structure contains a pointer that is initialized to another structure.

• A timer function needs to access a data structure. When the timer is running, the reference count of the data structure will be added. The reason for this is that you do not want the data structure to be released before the timer is executed.

• A matched table item is successfully found in a linked list or hash table. In most cases, this table item is used by the lookup. Therefore, in the lookup function, the reference count of the successfully matched data structure needs to be increased, and the reference count will be reduced by the lookup.

After the last reference of the data structure is released, the data structure can be released because it has no effect. Of course, this is not a required operation.

4. Garbage Collection

Memory is a shared and limited resource, so it should not be wasted, especially in the kernel. Because the kernel does not use virtual memory. Most kernel subsystems implement some type of garbage collection mechanism to collect memory occupied by useless or outdated data structures. From the existing implementation perspective, there are mainly two types of garbage collection mechanisms:
Asynchronous

This type of garbage collection mechanism has nothing to do with specific events. It uses a timer function to regularly check a set of data structures and release the data structures that can be released. The condition for determining whether a data structure can be released depends on the function and internal logic of the subsystem. However, a basic condition is that the reference count of the data structure is 0.
Synchronization

In case of memory shortage, if the asynchronous garbage collection function cannot be run regularly, the kernel can activate an immediate garbage collection function. In this function, the condition for determining whether a data structure can be released can be different from the asynchronous garbage collection function (for example, it can release some data structures with reference count not 0 ).

5. function pointers and virtual function tables (vfts)

Using function pointers is a good way to get clear C language code while taking advantage of some advantages of object-oriented languages. When defining data structures (objects), you can include a set of function pointers (methods ). Some or all of the data structure operations can be done through these functions. In C, the function pointer in the data structure is as follows:

struct sock {      
    ...      
    void    (*sk_state_change)(struct sock *sk);      
    void    (*sk_data_ready)(struct sock *sk, int bytes);
    ...
    };

The biggest advantage of using function pointers is that when initializing an object, function pointers can be assigned to different values based on different objects or different objects. In this way, you can call the sk_state_change function to activate different functions of different sock objects.
In network code, function pointers are widely used. Here are some examples:

• In the routing subsystem, when processing incoming or outgoing packets, it initializes two function pointers in sk_buff.

• When a package is ready to be sent to a network device, it will be passed to the hard_start_xmit function pointer in the net_device structure. This pointer is assigned a value when the network device driver is bound to the network device.

• When the L3 protocol sends a package, it calls one of a group of function pointers. These pointers are initialized in L3 Address Resolution Protocol processing. Which of the following statements is called?
  The function depends on which function it is initialized. The address resolution process from L3 to L2 is transparent (for example, IPv4 uses ARP ). If the address parsing process is unnecessary, these function pointers are assigned to them.
  Value.

In the above example, we can see that the function pointer can be used as an interface between different kernel components; or a sub-system can call different functions under different conditions; or it is used to allow different protocols. drivers or functions can use different methods.

Let's look at an example. When a device driver registers a network device with the kernel, the kernel executes functions unrelated to the device type. At some points, it calls
Use some function pointers in the net_device structure to let the device driver do something. Device drivers can initialize these function pointers as their own functions, or set them to null, which indicates
Run the default function on the core.

Before calling a function pointer, check the value of the function pointer to avoid referencing a null pointer. The following is an example of a snapshot taken from register_netdevice:

    if (dev->init && dev->init(dev) != 0) 
{         
 ...     
 }

Function pointers have a major drawback: It makes reading Source Code more difficult. When you read a given code path, you may pay attention to the function pointer calls. In this case, you need to first understand how the function pointer is initialized before you can read the subsequent code. Function pointer Initialization is related to many different factors:

• If the assignment of a function pointer is related to specific data, for example, the Protocol identifier or the packet received is a specific driver. In this case, it is easy to find the real function. For example, ifDrivers/NET/3c59x. c
  The driver receives a package. In the device initialization function, you can find the value of the function pointer in the net_device structure ..

• If the assignment of function pointers is related to more complex logic, such as the status values in L3 to L2 address ing. In this case, it is difficult to predict why the value assigned by the function pointer is related to an external event.

Put a group of function pointers in a data structure, which is usually called a virtual function table (VFT ). When a virtual function table is used as an interface between two subsystems, such as L3 and
Interfaces between L4, or interfaces exported as a kernel component (a group of objects), which may contain many different pointers, these pointers are used in different protocols or functions. Each function may only use
To a small part of function pointers. Of course, if the virtual function table is used too much, it will become very huge. In this case, you may need to redesign your data structure.

6. GOTO statement

No C programmer will like the GOTO statement. Without looking at the history of the GOTO statement (the longest and most famous debate in the history of computer programming), my conclusion is that the GOTO statement is outdated, but why is the Linux kernel still using it?

Any code that uses a GOTO statement can be rewritten with a code without a GOTO statement. Using the GOTO statement reduces the readability of the code and increases the difficulty of debugging. Because you cannot fully determine the conditions required for executing the GOTO statement.

Let's make an analogy: given any node in the tree, you can clearly know the path from the root to the node. But if it is a random vine, you cannot always get a unique path from the root to the node.

However, the C language does not provide an exception capture mechanism (in other languages. Exception capture is usually disabled,
Because the use of exception capture can cause performance degradation and increase Code complexity), careful use of the GOTO statement can easily jump the code to the exception handling code. In kernel programming, especially network generation
Code, exception events are very common, so the GOTO statement becomes a convenient tool. Although the GOTO statement is used in the kernel, I do not advocate that developers abuse it. Although there are more
30,000 goto statements, but they are mainly used to return different values in the same function, or to jump out of more than one layer of nesting.

7. Container vector Definition

In some cases, a data structure contains an optional data block at the end. As shown in the following example:

struct abc {      
 int age;     
 char *name[20];     
 ...     
 char    placeholder[0];
  }

Optional blocks start with placeholder. Note that placeholder is defined as a vector with a length of 0. This means that
When assigning space to ABC, an optional block is also allocated. placeholder is the block pointer. If you do not need to select a block, placeholder is just
Pointer, which does not occupy any space.

Therefore, if ABC is used in different codes, each code uses the same basic definition (avoid using different methods to do the same thing ), at the same time, you can expand ABC according to your own needs.

1.2.8. Conditional compilation (# ifdef and Family)

Conditional compilation of the compiler is sometimes necessary. Over-using Conditional compilation reduces code readability, But I can declare that the Linux kernel has not abused them.
Conditional compilation can be used in many cases, but we are interested in checking whether the kernel supports a certain feature. Make
The xconfig Configuration tool determines whether a specific feature is compiled to the kernel, or not supported at all, or compiled to the kernel module.

An example of using # ifdef or # If defined to check whether the kernel supports a certain feature is as follows:

A member that contains or does not contain a Data Structure

struct sk_buff {      
...  
#ifdef CONFIG_NETFILTER_DEBUG      
unsigned int nf_debug;  #endif      
...  
}  

In this example, the debugging function of Netfilter requires the nf_debug item in the sk_buff structure. If the kernel does not support netfilter
Debugging function (only a small number of developers need this function) does not need to include this function, otherwise it will only occupy more memory for each network package.

A function contains or does not contain some code.

int ip_route_input(...)  
{      
...          
if (rth->fl.fl4_dst == daddr &&
rth->fl.fl4_src == saddr &&              
rth->fl.iif == iif &&              
rth->fl.oif == 0 &&  
#ifndef CONFIG_IP_ROUTE_FWMARK              
rth->fl.fl4_fwmark == skb->nfmark &&  
#endif              
rth->fl.fl4_tos == tos) {              
...          
}  
}  

Select a proper prototype for a function

#ifdef CONFIG_IP_MULTIPLE_TABLES  
struct fib_table * fib_hash_init(int id)  
#else  
struct fib_table * _ _init fib_hash_init(int id)  
{      
...  
} 

Select the correct definition for the function

#ifndef CONFIG_IP_MULTIPLE_TABLES  ...  static inline struct fib_table *fib_get_table(int id)  
{      
if (id != RT_TABLE_LOCAL)          
return ip_fib_main_table;      
return ip_fib_local_table  
}  ...  
#else  
...  
static inline struct fib_table *fib_get_table(int id)  
{      
if (id == 0)          
id = RT_TABLE_MAIN;      
return fib_tables[id];  }  
...  
#endif

Note the difference between this example and the previous example. In the previous example, the function body is outside the # ifdef/# endif block. In this example, each block contains a complete function definition.

You can use Conditional compilation to define or initialize variables and macros.

It is very important to know that a function or macro has multiple definitions. The specific function or macro used is related to Conditional compilation. This is an example above. Otherwise, the definition of the function, variable, or macro you see may not be the one you want to see.

9. condition check compilation Optimization

Most of the time, the kernel compares a variable with an external value to see if a condition has been met. The comparison result is predictable to a large extent. Such examples are common, for example, code that checks validity. The kernel uses likely and
Unlikely indicates that the returned results are true (1) or false (0 ). These two macros use GCC to optimize the compilation results based on their return values to improve code performance.

Here is an example. Assume that the do_something function is called. In case of an error, handle_error is called to handle the error:

err = do_something(x,y,z);  
if (err)      
handle_error(err);

If do_something has few errors, we can rewrite the code:

err = do_something(x,y,z);  
if (unlikely(err))      
handle_error(err);

An example of using likely and unlikely macros to optimize code is to process IP Option. Because IP option only appears under certain circumstances, the kernel can assume that most IP packets do not have an IP option. When the kernel forwards an IP packet, it does not need to worry about
IP Option option. In ip_forward_finish, in the final phase of packet forwarding, this function uses the unlikely macro to check whether there are IP options to be processed.

10. mutex

Locks are widely used in network code, and you will find that every topic in this book involves it. Mutex, lock mechanism and synchronization are an interesting but complex topic in the programming field, especially in the kernel programming field. After years of development and optimization, the Linux kernel has included multiple ways to achieve code mutex. Here we will focus on the locks used in network code.

Each mutex mechanism has its applicable environment. The following describes the common mutex mechanism in network code:

Spin lock

This lock can only be occupied by one thread at a time. Other threads trying to get the lock will try again until the lock is released. Because the cycle will have a certain amount of elimination
Therefore, this lock is generally used in a multi-processor system, and developers want the thread to occupy the lock for a long time. Other threads trying to get the lock also consume a certain amount, so they occupy the lock thread,
The execution process cannot sleep.

Read/write spin lock

If the user of a lock can be clearly divided into read-only and read-write, read/write spin locks are recommended in this case. The difference between a spin lock and a read/write spin lock is that the latter can
Multiple readers occupy the lock at the same time. However, if the writer acquires the lock, the reader cannot obtain the lock. Because the reader has a higher priority than the writer, this lock applies to the write
The number of users is small (or the number of write lock requests is small.

If the lock is obtained by the reader, the writer cannot obtain the lock. Only when the reader releases the lock can the writer obtain the lock.

Read-copy-update read-Update (RCU)

RCU is the latest mutual exclusion mechanism provided by Linux. It performs well in the following situations:

• Many read requests and few read/write requests

• The code that gets the lock is automatically executed and does not sleep.

• Data Structures protected by locks are accessed through pointers

The first rule is related to performance. The second rule is the condition when RCU is used.

It is worth noting that, according to the first rule, the read/write spin lock is generally used as an alternative to RCU. To understand why the RCU performance is higher than the read/write spin lock in some cases, you need to consider other factors, such as the impact of the processor cache in the SMP system.

The example of using RCU in network code is the routing subsystem. The number of route searches is more than the number of Route updates, and the route query code is not blocked in the middle.

The kernel provides a semaphore, but it is rarely used in the network code described in this book.

11. Host byte sequence and network byte sequence Conversion

Data Structures of more than one byte are stored in memory in two different formats: little endian and big
Endian. The little endian format stores the lowest byte in the lowest address, while the big
The opposite is true for endian. The storage format used in Linux is related to the specific processor. For example, Intel processor uses little
The endian model, while the Motorola processor uses the big endian model.

Assume that our Linux host receives an IP package from the remote host. We do not know how the packet is sent on the remote host, so we do not know how to read the packet header. For this reason, each protocol cluster must define the storage format of its network package. For example, the TCP/IP protocol stack uses the big endian model.

However, kernel developers still face a problem: they must write code that can be used in different processors and storage modes. The storage mode of some Processors may be the same as that of the network package. In this case, you do not need to change the storage format of the Network Package.

However, each time the kernel reads or writes a variable that exceeds one byte in the IP header, it must first convert the network byte sequence to the host byte sequence or vice versa. This principle is equally appropriate
Used for other protocols in the TCP/IP protocol stack. If the network byte order is the same as the local byte order, the conversion function performs an empty operation because there is no need to convert between them. This improves code portability.
In this case, only the conversion functions are platform-related.

Table 1 lists the functions used to convert two-byte and four-byte variables:

Table 1. byte-ordering conversion routines

Macro	Meaning (short is 2 bytes, long is 4 bytes)
Htons	Host-to-network byte order (short)
Htonl	Host-to-network byte order (long)
Ntohs	Network-to-host byte order (short)
Ntohl	Network-to-host byte order (long)

These macro definitions are placed inInclude/Linux/byteorder/generic. h
Header file. The following describes how each platform associates the storage format of the platform with the definition of these macros:

• Directory related to each platformInclude/ASM-xxx/
  , All have a fileByteorder. h
  .

• This file containsInclude/Linux/byteorder/big_endian.h
  AndInclude/Linux/byteorder/little_endian.h
  One of the two files is related to the storage format of the processor.

• Little_endian.h
  AndBig_endian.h
  Both files contain common files.Include/Linux/byteorder/generic. h
  . The macro definition in Table 1 depends onLittle_endian.h
  AndBig_endian.h
  In this way, the storage formats of different platforms will affect the macros defined in Table 1.

Every macro XXX defined in Table 1 has a corresponding macro _ constant_xxx, which is used to convert the storage format of constants, for example, an enumeration element. It is worth noting that the macro in table 1 is a common macro, whether its input value is a constant or a variable.

As we have mentioned earlier, the storage format is very important for data items of more than one byte. The storage format is also very important for the definition of bit fields in more than one byte. Example
For example, IPv4 header definition. Kernel use _ little_endian_bitfield and _
_ Big_endian_bitfield: Two condition compilation parameters are used to control the definition of the data structure.
Little_endian.h
AndBig_endian.h
Two files.

12. Catching bugs

Tracking some functions can only be called under certain conditions, or cannot be called under certain conditions. The kernel uses the bug_on and bug_trap macros to capture function calls that do not meet the conditions. If the input value of bug_trap is false
The kernel prints an alarm message. Bug_on prints an error message and causes the kernel to crash.

13. Statistics

When implementing a function, it is a good habit to count the number of times a specific condition appears. For example, count the number of cache hits and failures, and the number of successful and failed memory allocations. This book lists and describes each statistical variable that appears in the network code.

14. Timing

The kernel often needs to measure how long it has elapsed since a given time point. For example, a task with a large CPU usage usually releases the CPU after a given period of time. If it is rescheduled, it continues to run. This is very important for Kernel programs, although the Linux kernel supports kernel preemption. In network code, a common example is the garbage collection routine.

The time in the kernel space is measured by the clock. The interval between two consecutive clock interruptions. The clock processes different tasks (here, we do not pay attention
And occurs Hz times per second. Hz is an architectural variable. For example, if you initialize it to 1000 on the i386 machine, it means that 1000 clock interruptions occur every second and
The interval between two consecutive interruptions is 1 ms.

The global variable jiffies is added to every clock interruption. This means that at any time, jiffies represents the number of clicks that have occurred since the boot, and the value of N * Hz generally represents n seconds.

If a function needs to measure the time interval, it can save the current jiffies value to a local variable, and then compare the value with the jiffies at the subsequent time to obtain the difference between them. Through this difference (the number of tick points between two moments), we can calculate the time from the start of the timer.

The following example shows a function. It needs to execute some tasks, but its CPU usage cannot exceed a single tick. When do_something is completed, it sets job_done to a non-zero value, and then the function can return:

unsigned long start_time = jiffies;  
int job_done = 0;  
do {      
do_something(&job_done);      
If (job_done)          
return;  
while (jiffies - start_time < 1);