How to crawl HTTP traffic packets under Linux (Httpry)

How to crawl HTTP traffic packets under Linux (Httpry)September 5, 2014 North

For some reason you need to sniff HTTP web traffic (that is, HTTP requests and responses). For example, you might test the performance of a Web server, or X-Uy debug a Web application or restful service, or try to resolve a PAC (Proxy auto-configuration) problem, or check for any malicious files downloaded from a Web site. Whatever the reason, sniffing HTTP traffic is very helpful for system administrators, developers, and even end users.

Packet sniffing tools such as tcpdump are commonly used for real-time packet dumps, need to set some filtering rules to capture only HTTP traffic, even if so, its output is difficult to understand, requires a certain protocol basics. Real-time Web server log analysis tools such as Ngxtop provide a trace of readable real-time network traffic, but only for logs with a fully-accessed Web server.

Is there a powerful tool for HTTP traffic only? That's the Httpry,http packet sniffing tool. Captures HTTP packets and displays content at the HTTP protocol level in a readable format.

Installing Httpry

In a Debian-based system such as Ubuntu,httpry is not included in the base warehouse.

$ sudoapt-get install gcc makegit libpcap0.8-dev

$ git clone https://github.com/jbittel/httpry.git

$ cdhttpry

$ make

$ sudomakeinstall

Fedora, CentOS, Rhel system need to install Epel source

$ sudoyum installhttpry

You can also compile the source code

$ sudoyum install gcc makegit libpcap-devel

$ git clone https://github.com/jbittel/httpry.git

$ cdhttpry

$ make

$ sudomakeinstall

Httpry Basic Usage

$ sudohttpry -i <network-interface>

Httpry monitor captures and displays HTTP request and response packets in real time under the specified NIC

In most cases, the output scrolls very fast and requires that the captured HTTP packets be saved for offline analysis. You can use the-B or-o option. The "-B" option saves the original HTTP packet to a binary file, which can then be replayed with Httpry. The "-o" option saves the readable output to a text file.

Save to a binary file:

$ sudohttpry -i eth0 -b output.dump

Replay:

$ httpry -r output.dump

Save to text file:

$ sudohttpry -i eth0 -o output.txt

Httpry Advanced Usage

If you want to capture specific HTTP methods, such as GET, POST, PUT, HEAD, connect, and so on, you can use the '-m ' option:

$ sudohttpry -i eth0 -m get,head

If you download Httpry source code, in the source directory, there is a Perl script to help us analyze the httpry output. The script is under the Httpry/scripts/plugins directory. These scripts are a good example if you want to write a custom parser for httpry output. Features are:

• Hostname: Show some column unique host names
• Find_proxies: Detecting Web Proxy
• Search_terms: Find and calculate enter search terms in the search service
• Content_analysis: Finding a URI that contains a specific keyword
• Xml_output: output in XML format
• Log_summary: Generate Log Summary
• Db_dump: Dump logs into MySQL database

Use the '-o ' option to run a period of time before using these scripts. Once you get the output, run these script analyses:

$ cdhttpry/scripts

$ perl parse_log.pl -d ./plugins file>

After the parse_log.pl executes, some analysis results files (*.txt/xml) are generated in the Httpry/scripts directory. For example, Log_summary.txt looks like this:

Copyright notice

Article title: How to crawl HTTP traffic packets under Linux (Httpry)
This address: http://www.ttlsa.com/web/how-to-sniff-http-traffic-from-the-command-line-on-linux/

How to crawl HTTP traffic packets under Linux (Httpry)