How to write a grab program that accesses a specific URL based on HTTP

Grabbing a packet is a simple and easy thing to do, and it can help you analyze the behavior of your network. As early as 2004, I remember the teacher told me how important it was to grab a bag.
As programmers, in addition to grasping the packet almost no means to detect network behavior, the programmer does not have the opportunity to touch the network equipment, there is no ability to understand the network details, the only thing the programmer can touch is his terminal, the only thing on this terminal is to grab the packet. I'm a half-programmer, so no matter who I stand, I think it's an important thing to grab a bag, although in my mind, grabbing and analyzing the packet doesn't solve most of the problems.
...
How to catch a packet on a machine that carries a lot of business is a problem. Because of the existence of a virtual host, a machine can host a large number of URLs, the PPS of these machines to tens of millions of minutes to remember, if we use tcpdump grab packet, a few seconds will be caught on the G data, which is a huge pressure on the disk. In these captured data, there may be no data we want at all, such as I point to crawl access to WWW.A.B/ABC packets, what should I do?! At the level of grasping the bag, you simply can't tell the difference between www.a.b/abc,www.a.b/def,www.123.abc/sdf ... Because they are local 80 port traffic, what you have to do is to grab all the traffic, and then use the tools like Wireshark/tshark to divert,
I have also written some articles about this shunt operation:
"How to crawl packets that access HTTP streams for a specific URL"
"Using Python to detach or crawl the HTTP stream in the Pcap capture package file"
Python implements crawling of packets that access a specific URL
The question now is, if time and space do not allow me to do so, how do I crawl the packets that access a particular URL at the first time?

A GET request to access a specific URL is sent after the TCP three handshake (HTTP can run over UDP, but this article only considers TCP), when a data flow is already established! Before deciding whether or not to crawl a stream based on a URL, the stream is already established, so when you see a TCP handshake process, it's not a stream of interest at all! This is where the problem lies. Therefore, in order to make a flow grab complete, all packets starting with the handshake packet must be cached before the URL is encountered. When a URL is found that is of interest, all packets cached before the output are released, and cached packets are freed when the URL of interest is not found within the tolerable time frame. This is the basic idea. If there is a pcap package, according to the above thinking to separate the data stream of interest is a relatively simple thing, with Python can be done in seconds, this in my previous article also mentioned. This article, however, is about capturing such packets in real time, rather than analyzing them afterwards.

I initially wanted to do it in the Tcpdump program, creating a hash table to keep every five-tuple stream, similar to NetFilter's nf_conntrack, in fact it's easy to do so. But not elegant enough, because this hash table has redundancy! With whom is redundancy? The answer is redundancy with the kernel-maintained socket hash table. The Socket hash table is a natural connection tracking information pool ah. I do not despise the Python dictionary, tuples, List of the implementation efficiency, but I think the direct use of the kernel socket hash table will be better! In other words, if I can record this socket's packet in this socket structure to not crawl, then as long as I can be a SKB (that is, the packet) corresponding to a socket, it is all right. Since there is such a well-organized hash table in the system, why do I have to create one myself?
... I didn't even want to create it myself, I just took advantage of Python's built-in data type! But I don't even want to use it, so I use socket! directly. The socket corresponds to a five-tuple, corresponding to a data stream, if I just want to be in the service of the local capture package, this socket is the use of the packet is the state of the must kill technology!
In order to realize the above idea, do I need to add a field in the sock structure? It seems to be true. However, this requires recompiling the kernel! In order not to recompile, I found that this requirement can be done using the Sk_flags field of the sock struct. The high 4 bits in the Sk_flags field are not used because I am ready to borrow them!

Thus, the high 4 bits of the sock structure form a state machine:

Implementation of this state machine flowchart I will not draw, no time ... I think the comments on the code are fairly clear. The above-mentioned state machine and flowchart programming implementation, is the following NetFilter module:

#include <linux/module.h> #include <linux/init.h> #include <linux/kernel.h> #include <linux/ skbuff.h> #include <linux/list.h> #include <linux/ip.h> #include <net/tcp.h> module_author ("Mary  Wangran ");  Module_license ("GPL"); #define cap_wait30#define cap_ign29#define cap_del28#define max_cache8unsigned char *url = "Test"; struct Wait_entry {struct list_head list;u16cnt;__be32saddr;__be32 daddr;u16sport;u16dport;unsigned longsave_flags; struct Sk_buff *skb[max_cache];}; Static Define_spinlock (Caplist_lock), Static List_head (wait_list), static struct wait_entry * Find_add_entry (struct SK_ Buff *skb) {struct List_head *lh, *n;struct wait_entry *wn;struct iphdr *iph = IP_HDR (SKB); struct TCPHDR *th = (void *) IPH + iph->ihl*4;u32 cmp_saddr, cmp_daddr;u16 cmp_sport, cmp_dport;cmp_saddr = iph->saddr > iph->daddr? IPH-&GT;SADDR:IPH-&GT;DADDR;CMP_DADDR = iph->saddr > iph->daddr? Iph->daddr:iph->saddr;cmp_sport = tH->source > Th->dest? Th->source:th->dest;cmp_dport = th->source > th->dest? Th->dest:th->source;spin_lock (&caplist_lock); List_for_each_safe (LH, N, &wait_list) {WN = List_entry ( LH, struct wait_entry, list); if (cmp_saddr = = Wn->saddr && cmp_daddr = = wn->daddr && cmp_spo RT = = Wn->sport && Cmp_dport = = Wn->dport) {if (wn->cnt < Max_cache) {wn->skb[wn->cnt] = SKB _clone (SKB, gfp_atomic); wn->cnt + = 1;spin_unlock (&caplist_lock); return WN;} else {int i = 0;for (i = 0; i < wn->cnt; i++) {if (Wn->skb[i]) {KFREE_SKB (wn->skb[i]);}} List_del (LH); Kfree (WN); Spin_unlock (&caplist_lock); return NULL;}} WN = (struct wait_entry *) kzalloc (sizeof (struct wait_entry), gfp_atomic); wn->saddr = iph->saddr > Iph->daddr ? iph->saddr:iph->daddr; WN-&GT;DADDR = iph->saddr > iph->daddr? iph->daddr:iph->saddr;; Wn->sport = th->source > th->dest? Th-> source:th->dest;wn->dport = th->source > th->dest? Th->dest:th->source;wn->skb[0] = Skb_clone (SKB, gfp_atomic);; wn->cnt = 1;__set_bit (cap_wait, &wn->save_flags); List_add (&wn->list, &wait_list); Spin_unlock ( &caplist_lock); return WN;} Char *findstr (const char *S1, const char *S2, unsigned int len) {int L1, l2;l2 = strlen (s2); if (!L2) return (char *) S1;L1 = Len;while (L1 >= L2) {l1--;if (!memcmp (S1, S2, L2)) return (char *) s1;s1++;} return NULL;} static int String_match (struct sk_buff *skb, char *str) {char *ret = Null;ret = findstr (skb->data, str, MB); if (ret) {R Eturn 1;} return 0;} static void Capture_skb (struct sk_buff *skb, const struct Net_device *dev) {struct IPHDR *iph = IP_HDR (SKB); struct TCPHDR * th = (void *) Iph + iph->ihl*4;u16 sport = 0, Dport = 0;u32 saddr = 0, daddr = 0;saddr = Iph->saddr;daddr = Iph->d Addr;sport = Th->source;dport = th->dest;//Simple printing just Printk ("# # #print%0x%0x%0x%0x s:%u a:%u len:%u\n",SADDR, Daddr, Sport, Dport, Ntohl (TH-&GT;SEQ), Ntohl (TH-&GT;ACK_SEQ), Skb->len);} static void Check_pcap (struct sock *sk, struct sk_buff *skb, char *url, int hook, const struct Net_device *dev) {struct WAI T_entry *entry = null;if (sk->sk_state = = Tcp_listen) {//Pay attention to the half-connection attack here! Therefore, the time-out mechanism for Entry table entries is required. Entry = Find_add_entry (SKB); if (!entry) {goto out;} Note the Tcp_defer_accept option, which allows you to receive a GET request in the Listen state! if (URL && string_match (SKB, url)) {int i = 0;spin_lock (&caplist_lock);//If a string is matched, then a maximum of 8 packets previously cached are exported, If you want to do something better, take a timestamp when you cache the packets, or there will be a burst. for (i = 0; i < entry->cnt; i++) {CAPTURE_SKB (entry->skb[i], dev);} Spin_unlock (&caplist_lock);//Because at this time the listen state socket does not correspond to a five-tuple, so entry as a five-tuple substitution to save the flags information, Finally, this flags will be mapped to the established establish socket! Match success, the entry corresponding to the final socket flag to have flag, indicating that the packet on the socket needs to crawl. __set_bit (&entry->save_flags);//Match succeeded, when this entry represents the tuple created a establish socket, after the flags are forwarded to the socket's flags, Delete it, because it's not needed anymore. __set_bit (Cap_del, &entry->save_flags);//Match success, do not continue to wait for GET,Clear the WAIT identity __clear_bit (cap_wait, &entry->save_flags);}} else if (sk->sk_state = = tcp_time_wait) {//todo} else {int add = 0;if (!sock_flag (SK, Cap_ign) &&!sock_flag (SK , cap_wait) &&!sock_flag (SK,)) {//here represents this for the first time from the listen state into the establish state entry = Find_add_entry (SKB); if ( Entry) {//Entry flags to socket (note that only high 4 bits are used) Sk->sk_flags |= (Entry->save_flags & 0xf0000000); if (Test_bit (CAP _del, &entry->save_flags)) {//If the DEL bit is set, the description has been successfully matched, no need for this entry, direct deletion//attention, at this time the flags also have bit int i = 0;spin_lock ( &caplist_lock); for (i = 0; i < entry->cnt; i++) {if (Entry->skb[i]) {KFREE_SKB (entry->skb[i]);}} List_del (&entry->list); Kfree (entry); entry = Null;spin_unlock (&caplist_lock);}} else {//If there is no listen at all, or in the listen phase is deleted entry, directly ignored, about this socket, never grab packet sock_set_flag (SK, cap_ign); goto out;} add = 1;} if (Sock_flag grab the bag. CAPTURE_SKB (SKB, dev);} else if (Sock_flag (SK, cap_wait)) {//carry the wait flag, continue to wait for the packet, expecting to match within 8 packets to a specific urlif (add = = 0{entry = Find_add_entry (SKB);} if (!entry) {Sock_set_flag (SK, cap_ign); Sock_reset_flag (SK, cap_wait); goto out;} if (URL && string_match (SKB, url)) {int i = 0;spin_lock (&caplist_lock);//If a string is matched, then a maximum of 8 packets previously cached are exported, If you want to do something better, take a timestamp when you cache the packets, or there will be a burst. for (i = 0; i < entry->cnt; i++) {CAPTURE_SKB (entry->skb[i], dev);} Match succeeds, the bit will enter the socket's flags, no longer need to wait for a match, no cache pending packets, delete entryfor (i = 0; i < entry->cnt; i++) {if (Entry->skb[i]) {k FREE_SKB (Entry->skb[i]);}} List_del (&entry->list); Kfree (entry); Spin_unlock (&caplist_lock); Sock_set_flag (SK,sock_reset _flag (SK, cap_wait);} Ignore}}out:return;}                                        static unsigned int ipv4_tcp_urlcap_in (unsigned int hooknum, struct sk_buff *skb, const struct Net_device *in, const struct NE T_device *out, Int (*OKFN) (struct Sk_buff *)) {struct sock *sk;strUCT Iphdr *iph = Ip_hdr (SKB), struct tcphdr *th = (void *) Iph + iph->ihl*4;if (iph->protocol! = ipproto_tcp) {return Nf_accept;} SK = __INET_LOOKUP_SKB (&tcp_hashinfo, SKB, Th->source, th->dest); if (!sk) {goto out;} Skb->sk = Sk;check_pcap (sk, SKB, URL, Hooknum, in); Out:return nf_accept;}                                         static unsigned int ipv4_tcp_urlcap_out (unsigned int hooknum, struct sk_buff *skb, const struct Net_device *in, const struct n Et_device *out, Int (*OKFN) (struct Sk_buff *)) {struct sock *sk;struct iphdr *ip h = IP_HDR (SKB); if (iph->protocol! = ipproto_tcp) {return nf_accept;} SK = Skb->sk;if (!sk) {goto out;}  Check_pcap (SK, SKB, URL, hooknum, out); Out:return nf_accept;          } static struct Nf_hook_ops ipv4_urlcap_ops[] __read_mostly = {. Hook = ipv4_tcp_urlcap_in,. Owner = This_module,. PF =Nfproto_ipv4,. hooknum = nf_inet_local_in,. Priority = -199,}, {. Hook = ipv4_tcp_urlcap_out,  . Owner = This_module,. PF = Nfproto_ipv4,. hooknum = Nf_inet_local_out,. Priority =    -199,},};  static int __init url_cap_init (void) {int ret;  ret = Nf_register_hooks (Ipv4_urlcap_ops, Array_size (ipv4_urlcap_ops));  if (ret) {goto out;;  }return 0;out:return ret;  } static void __exit Url_cap_fini (void) {Nf_unregister_hooks (Ipv4_urlcap_ops, Array_size (ipv4_urlcap_ops));  } module_init (Url_cap_init);   Module_exit (Url_cap_fini);

Then, let's test it out. Load the module on a host running the Apache Web server 1.1.1.2,/var/www/html directory Two small files, one is Big10, the other is test, the module's URL matching keyword set to "test", then when I do curl HTTP://1.1.1.2/BIG10, DMESG doesn't have any output, but when I do curl http://1.1.1.2/test, DMESG outputs the five-tuple and serial number information for all the interaction packs.

The problem 0. The ability to write code is entirely a precondition, the premise is that the GET request for HTTP will arrive in a fixed number of packets after the TCP three handshake, for example, I take 8! In general, three handshake total 3 packets, and then expect the client will soon have a GET request arrival, plus retransmission and other non-fixed behavior, will also be within the expected 8 packets arrived. However, if there is a Web implementation, no get requests within 10 packages, or multiple get requests within a TCP connection, the program is powerless.
1. Unable to process native as client do not assume that SKB is a simple struct, its data field does not necessarily hold the data for the packet. For the SKB of the receive path, the data is generally represented in the content, but for the sending path, in order to deal with the efficient, in general, do not make a merged copy of the memory, but rather a similar decentralized/clustered IO, "Let the data stay in the application layer in place" and then let the driver to self-can claim. So in order to match a string, you have to parse the SKB frag field, touch the page structure, processing in order to read data and temporary memory mapping, etc... Therefore, for the transmission path matching, this program is not supported!
2.IP layer NetFilter grab packet and packet socket grab packet There are differences note that the above code works in the IP layer of the NetFilter, while the real grab is working on the network card above the position, in which there is a "queue module", For Linux it is qdisc. We can take a look at the comparison between the data exported from the module and the actual capture, this difference is mainly manifested in the time stamp and packet sequence, once I used the Tcpprobe tool to export data and use tcpdump crawl data serious inconsistency, this is separated by this qdisc caused by, For details, see "Traffic shaping, latency, and the impact of ACK loss on TCP send timing." For this article, because I did not configure any TC rules, the two appear to be consistent:

To maintain consistency with the tcpdump results, it is necessary to maintain the position of the packet socket capture package, NetFilter module just to give a SKB a specific label, for the Send path, it is easy, NetFilter module for SKB to sign, And then in the network card XMit function grab the packet when grabbed the SKB can, but for the receive path, because NetFilter already past NETIF_RECEIVE_SKB this underlying function, so that need in netfilter module for SKB tag, Re-inject its clone into a virtual device to receive the analog packet, so that the signature of the SKB once again through the NETIF_RECEIVE_SKB to complete the clutch!
Do not think this way will lose performance, in the actual work before the consideration of performance, the resulting is a cloud! Moreover, Linux's bridge,vlan,bonding and IMQ are all doing the same.
3. Coding to be optimized as you can see, I use List_head and spin_lock instead of using hlist and RCU, which means there's still a lot of room for optimization in terms of overhead. In addition, I just printed the metadata of the packet in the kernel, and did not really go to use packet Socket capture package, I once wanted to let Wenzhou leather shoes factory boss help me, but was the shoe boss refused.

How to write a grab program that accesses a specific URL based on HTTP