How to crawl HTTP traffic packets under Linux (Httpry)

For some reason you need to sniff HTTP web traffic (that is, HTTP requests and responses). For example, you might test the performance of a Web server, or X-Uy debug a Web application or restful service, or try to resolve a PAC (Proxy auto-configuration) problem, or check for any malicious files downloaded from a Web site. Whatever the reason, sniffing HTTP traffic is very helpful for system administrators, developers, and even end users.

Packet sniffing tools such as tcpdump are commonly used for real-time packet dumps, need to set some filtering rules to capture only HTTP traffic, even if so, its output is difficult to understand, requires a certain protocol basics. Real-time Web server log analysis tools such as Ngxtop provide a trace of readable real-time network traffic, but only for logs with a fully-accessed Web server.

Is there a powerful tool for HTTP traffic only? That's the Httpry,http packet sniffing tool. Captures HTTP packets and displays content at the HTTP protocol level in a readable format.

Installing Httpry

In a Debian-based system such as Ubuntu,httpry is not included in the base warehouse.

12345

$ sudo apt-get install gcc make git libpcap0. 8-dev $ git clone HTTPS://github.com/jbittel/httpry.git $ cd httpry $ make$ sudo make install

Fedora, CentOS, Rhel system need to install Epel source

1	$ sudo yum install httpry

You can also compile the source code

12345

$ sudo yum install gcc make git libpcap-devel $ git clone HTTPS://github.com/jbittel/httpry.git $ cd httpry $ make$ sudo make install

Httpry Basic Usage

1	$ sudo httpry -i <network-interface>

Httpry monitor captures and displays HTTP request and response packets in real time under the specified NIC

In most cases, the output scrolls very fast and requires that the captured HTTP packets be saved for offline analysis. You can use the-B or-o option. The "-B" option saves the original HTTP packet to a binary file, which can then be replayed with Httpry. The "-o" option saves the readable output to a text file.

Save to a binary file:

1	$ sudo httpry -i eth0 -b output. Dump

Replay:

1	$ httpry -R output. Dump

Save to text file:

1	$ sudo httpry -i eth0 -o output. TXT

Httpry Advanced Usage

If you want to capture specific HTTP methods, such as GET, POST, PUT, HEAD, connect, and so on, you can use the '-m ' option:

1	$ sudo httpry -i eth0 -m get,head

If you download Httpry source code, in the source directory, there is a Perl script to help us analyze the httpry output. The script is under the Httpry/scripts/plugins directory. These scripts are a good example if you want to write a custom parser for httpry output. Features are:

• Hostname: Show some column unique host names
• Find_proxies: Detecting Web Proxy
• Search_terms: Find and calculate enter search terms in the search service
• Content_analysis: Finding a URI that contains a specific keyword
• Xml_output: output in XML format
• Log_summary: Generate Log Summary
• Db_dump: Dump logs into MySQL database

Use the '-o ' option to run a period of time before using these scripts. Once you get the output, run these script analyses:

12	$ cd < Span class= "crayon-v" >httpry/scripts $ perl parse_log. Pl -d < Span class= "Crayon-sy". /plugins < httpry-output-< Span class= "crayon-v" >file>

After the parse_log.pl executes, some analysis results files (*.txt/xml) are generated in the Httpry/scripts directory. For example, Log_summary.txt looks like this:

How to crawl HTTP traffic packets under Linux (Httpry)