Memory Management of embedded LwIP protocol stack

TCP/IP is a hierarchical network architecture based on the OSI reference model. It consists of the application layer, transport layer, network layer, data link layer, and physical layer. Messages are transmitted between layers in the form of datagram. Because the header lengths of different layers are different, when data is transferred between different protocol layers, data is encapsulated and deencapsulated, added, and deleted frequently.

Similar problems often occur in embedded system development. When user data is transmitted from a local embedded device to a remote host, it is inevitable to encapsulate, deencapsulate, and copy messages through various protocols. Generally, using a continuous memory area to store and transmit data has the following drawbacks:

(1)When data is transferred from the upper layer to the lower layer, the lower layer protocol needs to encapsulate the data, while the upper layer does not (or should not) consider the lower layer when applying for memory. This will cause the lower-layer protocol to re-apply for memory and copy the memory, thus affecting the program efficiency.

(2)When data is transmitted from the lower layer to the upper layer, the proprietary data structure of the lower layer protocol should be invisible to the upper layer protocol. Therefore, you also need to apply for memory again and copy the memory.

(3)As data is processed layer by layer, its content may increase or decrease, and continuous memory is difficult to process such dynamic data addition and deletion.

Therefore, there must be a data structure that can adapt to the dynamic addition and deletion of data, but logically presents continuity, so that data can be transferred between different protocol layers without memory copying. Embedded TCP/IP protocol stack requires simplicity and efficiency, and reduces memory requirements. These require the corresponding memory management mechanism.

Introduction to pbuf in 1LwIP protocol stack

LightweightIP is an open-source TCP/IP protocol stack developed by the Swiss Emy of computer sciences, such as AdamDunkels. LwIP can be transplanted to the operating system or run independently without an operating system. The focus of lwiptcp/ip implementation is to reduce RAM usage while maintaining the main functions of the TCP protocol. This makes the LwIP protocol stack suitable for low-end embedded systems.

LwIP uses the pbuf structure to transmit data, which is similar to Mbuf in BSD. Pbuf is mainly used to store user data transmitted between applications and network interfaces.

The internal structure of pbuf is:

Structpbuf {

Structpbuf * next;/* points to the next pbuf */

Void * payload;/* point to the start position in pbuf data */

U16_ttot_len;/* sum of the Data lengths of the pbuf and subsequent pbuf */

U16_tlen;/* length of data in the pbuf */

U16_tflags;/* pbuf type */

U16_tref;/* Number of times the pbuf is referenced */

}

Implementation of 2LwIP Memory Management

In an embedded system running the TCP/IP protocol stack, the storage area of the entire system can be divided into two parts: the memory managed by the protocol stack and the memory managed by the application.

2.1 protocol stack management memory

The protocol stack management memory refers to the memory area that the TCP/IP kernel can operate on. It is mainly used to load network data groups to be received and sent. When a group or group is sent, the TCP/IP protocol stack allocates cache for these groups. After the received group is delivered to the application or group, reclaim and reuse the allocated cache. The cache allocated by the protocol stack must be able to accommodate packets of various sizes, for example, from only a few bytes of ICMP response packets to several hundred bytes of TCP segment packets.

There are four types of pbuf in LwIP: PBUF_POOL, PBUF_RAM, PBUF_ROM, and PBUF_REF. These four types of pbuf are allocated from the memory managed by the TCP/IP protocol stack. PBUF_ROM and PBUF_REF are closely related to the storage areas managed by applications.

PBUF_POOL is a pbuf with a fixed capacity. It is mainly used by network device drivers to allocate cache for received data groups. A pbuf pool (PBUF_POOL) is initialized in the memory of the protocol stack management. pbuf with the same size is allocated from this pbuf pool. Generally, multiple PBUF_POOL links are used as a linked list to store data groups. 1.

LwIP defines the size of a PBUF_POOL with a macro. Several pbuf_pools need to be allocated to a group, and a PBUF_POOL can be allocated when there is little data. A pbuf of the PBUF_POOL type is very fast and suitable for interrupt processing. Therefore, PBUF_POOL is mainly used by network device drivers to allocate cache for received data groups.

When an application sends dynamically generated data, you can use PBUF_RAM-type pbuf. PBUF_RAM is allocated in the pre-defined memory heap. The operation on the memory heap is similar to malloc/free in C language. The structure 2 of the memory heap allocation is shown in. In Figure 2, each allocated storage block comes with a small structure with two pointers pointing to adjacent memory blocks. Used is used to indicate the allocation of the memory block. The shadow part indicates that the block has been allocated. At this time, used is 1. When a N-byte storage block is required, the entire storage heap is searched. If an unused area (used = 0) is found and the capacity is not less than N Bytes, the allocation is successful and the used is set to 1. The allocated memory block needs to be released after use. In order not to generate fragments, adjacent and unused memory blocks need to be merged.

Both PBUF_POOL and PBUF_RAM can be dynamically allocated from the memory as needed. This allocation mechanism is also called dynamic memory allocation mechanism. This allocation mechanism not only allocates storage space for application data, but also for protocol headers. When data is transmitted between layers, only the data format needs to be modified to conform to the specifications of each layer, and the data itself does not need to be changed. In fact, the data format reflects the header of each layer. when data is transmitted between layers, the corresponding header needs to be dynamically added and removed, which can be well implemented using the dynamic allocation mechanism.

2.2 Application-managed storage

Application-managed storage refers to the storage area where applications manage and operate. Generally, data is allocated to applications from this area. Although the storage area cannot be managed by the TCP/IP protocol stack, the storage area must work with the memory managed by TCP/IP in a protocol stack that is not strictly layered. To save memory, LwIP does not adopt hierarchical access mode, but accesses data through pointers. In this way, you do not need to allocate storage space for data transmission. After the application delivers the LwIP data, the LwIP considers that the data cannot be changed. Therefore, the application data is considered to exist forever and cannot be changed. This is similar to ROM, And the type name PBUF_ROM also comes from it.

As shown in 3, the Data Pointer payload of PBUF_ROM points to Externalmemory (External Store ). Externalmemory refers to the storage area not managed by the TCP/IP protocol stack. It can be the cache allocated by the memory managed by applications for user data, or the ROM region, for example, a String constant in a static webpage. Because the data delivered by the application cannot be changed, you need to dynamically allocate a PBUF_RAM to load the protocol header, and then add PBUF_RAM (header) to the front of PBUF_ROM (data. This constitutes a complete data group.

In Figure 3, PBUF_ROM can also be PBUF_REF. PBUF_REF and PBUF_ROM have very similar features and can both achieve zero data copy. However, when the sent data needs to be queued, The PBUF_REF feature is displayed. For example, when sending a group, the group to be sent needs to be queued in the ARP queue. If these groups contain PBUF_ROM pbuf, the data in this type of pbuf is located in the storage area of the application, the pointer is referenced by PBUF_ROM. In this way, the storage area of the referenced application cannot be used separately until the group is processed. In this case, pbuf of the PBUF_REF type is used. During queuing, LwIP allocates cache (PBUF_POOL or PBUF_RAM) to pbuf of the PBUF_REF type, and copies the data of the referenced application to the allocated cache. In this way, the storage area of referenced data in the application can be released.

The pbuf structure implements data transfer between layers, but it consumes a lot of memory and requires the TCP/IP protocol stack to allocate storage space for it, such as protocol control udp_pcb and tcp_pcb. Generally, the embedded TCP/IP protocol stack is not strictly layered. To minimize the need for memory is the focus of embedded TCP/IP, the memory management mechanism of the kernel is directly related to the performance of the embedded TCP/IP protocol stack.

Edit recommendations]