Libpcap, tcpdump, ntop user experience

In the past two years, the development of network applications has a high demand for Qos. many companies are doing this. even Xaar has set up a QQos. However, we can see that we still use ports to determine the protocol, they do not have the technical strength to be more advanced. However, the Cisco devices used in the last two days are indeed different. the protocol is analyzed to Layer 7, BT, pplive, and thunder, which can all be determined. The development of network applications over the past two years is very demanding on Qos, many companies are doing this. even Xaar has a QQos. However, it seems that the port is used to determine the protocol. if they want to come, they will not be more powerful.
However, the Cisco devices used in the previous two days are indeed different. the protocol is analyzed to Layer 7, BT, pplive, and thunder, and their structure is also complicated, another machine is required to install a virtual machine and run a program for statistical analysis. of course, this is also normal. my original analysis is also like this. one machine cannot do everything at all. A vSwitch performs data collection, basic analysis, and link control on the link. data is sent to another pc or server, using large memory and high-speed cpu, filtering and statistics of massive amounts of data, especially because the current p2p protocol has to be resolved to Layer 7 to determine what it is, so it consumes a lot of resources. Determine and control in real time. Then the historical data is stored in the database. MysqlAnd then use a client software analysis to show that their program is the rich client of eclipse, and all the components are free of charge for eclipse...

Cisco is said to have acquired a company named SCE dedicated to this.
Two days ago, I started an experiment under linux to capture data packets. TcpdumpPacket loss is always performed, and a lot of data is lost by default. domain name anti-analysis is removed and the timestamp is removed. this is a lot better, but it is still lost. if the result is directly output to the file, it is a little less lost, but it is just a packet capture and has not been processed yet... Checked TcpdumpManual of, searched for half a day, found that pcap is missing, the main reason is probably that the buffer is not big enough, the old package has not been removed, and the new one has come, so I had to rush out the old one. Then I found several ntop papers written by the author www.ntop.org. the main reason for the analysis is the way to process the network card in linux. because of the interruption, each packet is interrupted and then processed, which consumes a lot of cpu and is inefficient, therefore, we can try to reduce the number of interruptions. the ideal solution is to allow the application layer to directly exclusively occupy the NIC, so that all Nic data is directly sent to the application .. The author has spent a lot of effort in this aspect to experiment and summarize, so that we plan to use ntop to collect data, but the ntop program has done too much... So that... It seems that it is not suitable for our requirements. our egress bandwidth is 1G + 120 M, and the average data is more than 100 M. ntop makes protocol judgments and records all records, the number of out-of-school ip addresses is tens of millions, and various data quickly fills up 2 GB of memory. because there are too many chores and the speed of ntop is affected, data cannot be processed in a timely manner, memory accumulation is also a reason for the consumption of light memory. I have tried my best to reduce the chores on the configuration .. In addition, ntop has always encountered inexplicable problems under such high loads, such as changing the settings to avoid school failures and making judgments confusing .. I wrote the configuration in the startup script/etc/init. d/ntop.
-B-o-n-z-M-x 65535-X 131072-D "xxxx.edu.cn"-C "network "\
-M "x. x/20 "\

Disable decode, reverse analyze the domain name, and expand the number of hash entries. or, if you do not care about the Internet IP address, you can add-g to track the intranet only.
In this case, it still does not work... It seems that if you want to capture packets, you can handle them .. It also requires a very professional solution .. Cisco's hardware technology can provide good equipment for such a requirement. if you want to solve it yourself, you must make major changes to the operating system processing to remove anything you don't need, the working mode is changed to real-time or at least everything is aimed at increasing the speed and reducing resource consumption and waste. Some linux route or switch professional solutions should be available for reference. But the current server performance... In fact, it can also be done together .. When the situation is good, only 0.2% is lost. if we make some improvements based on the PF_RING method of ntop author, we should be able to bear it. but the best thing is that this server only performs packet capture and collection, then, the data is immediately sent to other servers for further processing using the most resource-saving method.
Bittorrent and some other p2p protocols cannot be identified by port alone. because the server and client can set ports freely, data must be recorded during their handshake, and maintain this status. because the data transmitted later cannot be identified at all, it is only a TCP packet. Therefore, you must first adapt the content of each data and record the session, the subsequent data does not need to be unwrapped, because all data packets are TCP packets.