Network System Management-application layer packet capture solution and implementation

Article title: Network System Management-application layer packet capture solution and implementation. Linux is a technology channel of the IT lab in China. Includes basic categories such as desktop applications, Linux system management, kernel research, embedded systems, and open source.
　　 Introduction
Packet capture needs generally come from filtering, conversion protocol, and packet capture analysis.
Filters have many applications, typically Packet Filter Firewall.
The application of the conversion protocol is limited to some specific environments. For example, if a third party develops network protocol software and cannot integrate it with the original operating system software, it has to adopt the "embedded protocol stack block" (BITS) method. For example, the third-party implementation of IPSEC on Windows cannot be integrated with the IP software provided by the operating system vendor, so it must be implemented between the IP layer and the link layer as a layer of the protocol stack. Third-party PPPOE software is also implemented in this way.
The purpose of intercepting a packet is to use "packet capture" to describe more appropriate. "packet capture" generally indicates that the packet has the ability to be truncated. "packet capture" only needs to be obtained. The implementation is generally implemented as the protocol layer.
The "application layer packet capture" mentioned in this article refers to the working mode of intercepting packets in the driver and then sending them to the application layer for processing.
　　
　　 Packet capture mode
Network packet interception methods in user mode are as follows:
1. Winsock Layered Service Provider;
2. Windows 2000 package filtering interface;
3. replace the WINSOCK dynamic connection library that comes with the system;
　　
You can use the driver to intercept network data packets by using the following methods:
1. TDI Filter Driver)
2. NDIS Intermediate layer Driver (NDIS Intermediate Driver)
3. Win2k Filter-Hook Driver
4. NDIS Hook Driver
　　
The interception of data packets in the user mode has some limitations. "Obviously, the most fatal disadvantage of packet interception in the user mode is that it can only be performed at the Winsock level, however, data packets of the underlying protocol in the network protocol stack cannot be processed. For some Trojans and viruses, it is easy to avoid this layer of firewall ."
The "application layer packet capture" does not refer to the user-mode packet interception described above. It is intercepted in the driver and processed at the application layer. To obtain a common method, you must intercept it at the IP layer. This article uses the intermediate layer mode for comprehensive comparison.
　　
　　 Why do we need to process intercepted packets at the application layer?
Generally, network applications such as firewalls and protocol software work in the kernel. why should we, in turn, propose to process packets at the application layer? The reason can also be found out (even more far-fetched ):
As we all know, driver development is difficult. for an experienced programmer, there may be no technical problems in the development process, but for beginners, especially the first time I came into contact with programmers, it was simply a painful experience.
In addition, the development cycle is also an issue that must be considered. The program runs in the kernel, and the stability and compatibility all require a large number of tests, and the available function libraries are quite small compared to the application layer. In application layer development, debugging and modification are much easier.
Unfavorable factors include:
Performance impact: working at the application layer changes the working mode. whenever the driver captures data, it is sent to the application layer for processing and then sent back to the kernel, and then to the IP protocol. Therefore, the performance is greatly affected and the efficiency is very low. on a 80% Mbps network, only of the performance is achieved.
In summary, it is suitable for specific applications:
When used on a desktop, the network load on the desktop is quite small, and less than 512 Mbps is enough to meet the requirements, especially for Internet access and other environments. The network connection traffic is less than Kbps, so there is no need to consider performance factors. As a single-host firewall or some other protocols, analysis and so on can be easily implemented based on this method.
　　
　　 Solution
Model
　　
Describes the model of packet capture at the application layer. The main process is as follows:
　　
Packet receiving process:
1. the network interface receives the packet, the middle layer intercepts the packet, and 2 sends the packet to the application layer for processing;
2. after processing at the application layer, return the intermediate layer processing result;
3. the middle layer discards the packet based on the processing result, or sends the processed packet to the IP protocol through 1;
4. the IP protocol and upper-layer applications receive packets;
　　
Packet sending process:
1. upper-layer applications send data to send packets through IP protocol;
2. the message is intercepted by the middle layer and sent to the application layer for processing through 2;
3. after processing at the application layer, return the intermediate layer processing result;
4. the middle layer discards the message based on the processing result or sends the processed message to the network;
　　
　　 Implementation details
IO and communication
　　
There is an easy way to use an event between the driver and the application.
When the application CreateFile is used, the driver IoCreateSynchronizationEvent is a famous event, and then the application CreateEvent/OpenEvent is a famous event.
Note:
1. do not create events during driver initialization. at this time, most events cannot be created successfully;
2. if the driver is created first, the application can only read (Waitxxxx) but not write (SetEvent/ResetEvent) when it is opened later ). If an application is created first, both the application and the driver have the read and write permissions;
3. the name is ideal. Note that the driver name is under \ BaseNamedObjects \. for example, if the application uses "xxxEvent", the driver is "\ BaseNamedObjects \ xxxEvent ";
4. you can use HANDLE, but it is unknown whether it is feasible in WIN98.
5. after that, the driver should immediately return the read request; otherwise, the return fails. Otherwise, the meaning of event notification will be lost (read is not waiting to be completed, but will be read only when there is a need (notification event );
6. When an application finds an event, it should read it in a loop until the reading fails, indicating that no data is readable; otherwise, subsequent data is missed and not read in time;
　　
　　 Processing thread priority
The application layer processing thread should increase the priority because the thread serves other upper-layer applications. if the priority is lower than the priority of other threads, a wait state similar to a deadlock will occur.
In addition, when raising the priority, you must note that the running time of the thread should be shortened as much as possible, and the CPU should not be used for a long time; otherwise, other threads will not be able to get the service. The priority does not need to be raised to the REALTIME_PRIORITY_CLASS level. in this case, the thread cannot perform operations such as disk IO, but also affects the mouse, keyboard, and other work.
The driver can also dynamically increase the thread priority.
　　
　　 Cache
After the driver receives the packet, there should be at least one buffer for temporary storage, waiting for processing at the application layer. The buffer does not need to be large. as long as the buffer does not overflow before the application layer obtains the time slice, it is enough to store dozens of packets in practice.
The buffer usage method is a first-in-first-out queue. To facilitate the implementation of a ring queue for static storage, you do not need to allocate memory each time. Instead, you need to allocate a large block of memory at a time to use the ring.
Initial, head = tail = 0;
Both tail and head are infinitely increasing.
Tail? Head <= size;
　　
When a packet is placed, tail = tail + packetlen;
Head = head + packetlen;
　　
Tail = head indicates null;
Tail> head indicates that there is data;
Tail + input packet length-head> size indicates full;
　　
When retrieving data:
Ppacket GetPacket ()
{
ASSERT (tail> = head );
　　
If (tail = head)
Return NULL;
　　
// Else
Ppacket = & start [head % SIZE];
If (head % size + ppacket-> length> size)
// Data is not continuous (some are at the end and some are at the header );
Else
// The data is continuous.
　　
Return ppacket;
}
　　
Put data:
Bool InputPacket (ppacket)
{
If (tail + input packet length-head> size) // Full
Return false;
　　
// Copy packet to & start [tail % SIZE]
// If (tail % SIZE + packet length> SIZE)
// Data is not continuous (some are at the end and some are at the header );
// Else
// The data is continuous.
　　
Tail = tail + packet length;
Return true;
}
　　
The preceding method uses arrays to provide a maximum packet length space for each packet. Because the number of buffers is limited, this method can meet the needs. If you want to reduce the waste of space, you can store the data according to the actual length of each message. the above algorithm cannot adapt to this method.
　　
　　 Communication between application layer and driver
When the Nic receives or sends an IP address, the driver caches the message and notifies the application layer of the message to be processed. Then, the application layer can obtain this message through IO or shared memory.
Practice shows that the above two methods can meet the needs at a rate of Mbps, and the easiest way is to use the buffer I/O method.
After the application layer completes processing, you can use either of the above two methods to submit the results to the driver. However, because I/O can only send one packet at a time, the network speed of 70% Mbps is reduced ~ 80% network speed, 10 Mbps will not be affected. That is to say, the maximum speed sent by the host is only 70% of the network speed, which is the same as the speed at which the application sends UDP datagram up to MTU. For TCP, due to two-way communication, the loss is greater, about 40% ~ 60% speed.
In this case, the shared memory mode is used to reduce the overhead of system calls and avoid the speed decrease.
　　
　　 Speed control of message transmission
When the IP protocol sends packets, in general, our middle-layer driver must cache these packets, tell the IP software to send successfully, and then let the application layer make a decision after processing. Obviously, the speed of storing packets far exceeds the speed that the network adapter can send. However, IP software (especially UDP) will send packets at the speed that we store. The cache is quickly exhausted. The subsequent messages have to be discarded. In this way, UDP transmission will not work properly. Because TCP can adapt to the network conditions on its own, it can still work in this case, and the speed is around 70%. In Passthru, it can be forwarded to the low-layer driver and then returned asynchronously or synchronously, so as to achieve the same sending speed of the NIC.
Therefore, there must be a way to avoid this situation. The middle layer driver caches these packets and tells the IP software that the sending status is Pending (Pending ). After the final processing is completed, the IP address software will be notified of the completion of the sending. This coordinates the sending speed. This approach brings about a problem that the driver must give up ownership of these buffered packets when the sending times out. Specifically, when the MiniportReset is called, it may be that NDIS detects the sending timeout, thus giving up all the unfinished sending operations. if this situation is not properly handled, this will cause serious problems. If the middle layer sets the NDIS_ATTRIBUTE_IGNORE_PACKET_TIMEOUT flag by calling the NdisMSetAttributesEx function during Miniport initialization, the middle layer driver will not receive message timeout notifications, and the middle layer must process cached packets by itself.
　　
　　 Work with Passthru
When upper-layer applications no longer require packet capture, the driver should be completely Passthru. This requires