Wappalyzer API for Web applications fingerprint recognition example

Web application fingerprint identification, is the Web penetration information collection is the most critical step, this aspect of open source tools are also very many, like Blindelephant,whatweb and in the security circle are very hot wappalyzer. This article mainly describes how to use the Wappalyzer perl and PHP interface for fingerprint identification.

The function of the wappalyzer is to identify the fingerprint of a single URI by sending an HTTP request to the specified URI, obtaining the response header and the response body, and matching the fingerprint rule. This is also the most basic part of Web application fingerprint recognition, in addition, there are fingerprint confidence calculation (how to fake fingerprints, a variety of fingerprint characteristics of how to comprehensively judge, hide the fingerprint information how to extract), the entire site Fingerprint identification also involves effective crawler crawling, distributed computing and other issues, these are not in this text.

FREEBUF Small Science: Understanding Wappalyzer

Wappalyzer is a browser plug-in that can be used to identify a Web site using the Wappalyzer Web technology. It detects CMS and e-business systems, message boards, JavaScript frames, host panels, analytics statistics tools, and other web systems. The company behind Wappalyzer also collects information from Web applications for statistical analysis, revealing the growth of usage of various web systems. The actual wappalyzer is a fingerprint identification tool. For more information please click: http://www.freebuf.com/tools/7391.html

Official address: https://wappalyzer.com/

A Perl version

Principle:

Sends an HTTP request to the specified URI to determine the fingerprint by parsing the corresponding part of the HTTP

(1) Headers characteristics

Response header Key:value pairs, multiple key:value separated by commas, for example

"Headers": {"x-amp-version": "([D.] +); version: "," Set-cookie ":" ^amp= "},

(2) HTML features

Response body content, with multiple rules separated by commas, such as

"HTML": ["

Special Note: HTML Full-text matching rules must be carefully written

(3) URL characteristics

URL content, such as

"url": "/cgi-bin/uceditor?" (?:.*&) merchantid=. "

(4) Meta features

In response to an HTML page, such as the name in: Content contents pairs, multiple rules are separated by commas, for example

"Meta": {"generator": "Webedition", "Dc.title": "Webedition"}

(5) Script features

In response to HTML pages such as URL content in SRC, multiple rules are separated by commas, such as

"Script": [Angular:-|.) ([D.] *D) [^/]*.js;version: ","/([D.] + (-?rc[.d]*) *)/angular (. min)? Js;version: "," Angular.*.js "]

Limitations:

Access to app version number and confidence in rule files is not supported

There may be a problem with the Chinese encoding for UTF8

Advantages:

Compared to the PHP version, using QR regular precompiled processing, you can find the regular problem in advance, which is the main reason why I chose the language version.

(The next version of PHP will let you know that there are many holes in the precompiled process!)

Functions of the script:

(1) Fingerprint identification results are returned in JSON format so that the subsequent fingerprint information can be processed in storage.

(2) Support batch URI query

(3) Support for specifying a custom JSON format fingerprint rule file

(The default fingerprint file is placed in/usr/lib/perl5/www/apps.json, the path will vary depending on the installation path of the CPAN module.)

You can use Perl-v to see the @inc variable to determine the path, or more violent find.

Installation:

Cpan-i Www::wappalyzer
Clone Https://github.com/tanjiti/FingerPrint.git

Run:

(1) Obtaining a fingerprint of a single URI

Perl fingerprint.pl www.xxx.com tanjiti.json[fingerprint rules file, optional]

return results

{
"Www.xxx.com": {
"Blogs": [
"WordPress"
],
"Web-servers": [
"Nginx"
],
"CDN": [
"CloudFlare"
],
"CMS": [
"WordPress"
],
"Font-scripts": [
"Google Font API"
],
"Javascript-frameworks": [
"JQuery"
],
"Javascript-graphics": [
"Javascript Infovis Toolkit"
]
}
}

(2) Read the URL list from the file for bulk fingerprint identification, and output the results to the file

Perl fingerprint.pl url.txt tanjiti.json[fingerprint rules file, optional]

The results are exported to the Url.txt__fingerprint file.

Fingerprint rule File Writing example:

More Tanjiti.json

"Apps": {

"discuz!": {
"Website": "www.discuz.net/forum.php",
"Cats": [1],
"Meta": {"generator": "Discuz"},
"Headers": {"Set-cookie": "_lastact.*_sid|_sid.*_lastact|_sid.*smile|smile.*_sid"},
"url": "/uc_server[/$]|uc_client[/$]",
"HTML": "Powered by" (?:D iscuz!| <a href= "http://www.discuz.net/" | Ucenter) ",
"implies": "PHP"
},
"PHP": {
"Website": "Php.net",
"Cats": [27],
"Headers": {"Server": "Php/?" ( [D.] +)?; Confidence:40;version: "x-powered-by": "php/?" ([D.] +)?; Confidence:40;version: "," Set-cookie ":" Phpsessid "},
"url": ". php (?: $|?)"
}

}

Second, PHP version

Principle

With the Perl version, the difference has two points:

1. HTTP request section: Using LWP to send HTTP requests than Perl, PHP uses Curl to send HTTP requests

2. Rule matching part: The matching part of fingerprint rule uses JavaScript syntax, and then through PHP's V8js module to parse. (Why do you do this?) when the response body content is very much, the fingerprint is written very bad time, will be stuck in the rule matching the process, now under the rules of sina,163 such as the basic card death!

Run

PHP version of the interface has been able to use directly, just need to replace the rules file,?? /p>

First step: Install PHP, curl and V8JS

Apt-get Install Php5-dev php-pear build-essential Libv8-dev
PECL Install channel://pecl.php.net/v8js-0.1.3
echo extension=v8js.so >>/etc/php5/cli/php.ini

Verify that the installation is successful

php-m | grep v8js
V8js-----------OK

Step two: Download the Wappalyzer script

git clone https://github.com/ElbertF/Wappalyzer.git
Cp-r wappalyzer/drivers/php/*.
CP Wappalyzer/share/js/wappalyzer.js js/
CP Wappalyzer/share/apps.json. (Fingerprint rules file) The rules file is written with a problem, so I used the previous version of the rule file Cp/usr/lib/perl5/www/apps.json.

The PHP print fingerprint Identification program is structured as follows:

index.php Main Program
wappalyzer.php
wappalyzerexception.php
Js/driver.js Js/wappalyzer.js

Apps.json Fingerprint Rules file

Step Three: Verify success

PHP index.php www.tanjiti.com

Output format is

Application name, version number, reliability, app type (more than Perl version to version number, gain of reliability)

CloudFlare, 100%, CDN
Javascript Infovis Toolkit, 100%, Javascript-graphics
JQuery, 100%, Javascript-frameworks
Nginx, 100%, Web-servers.
PHP, 5.5.9, 100%, programming-languages
Ubuntu, 100%, Operating-systems
WordPress, 4.0, 100%, CMS, blogs

The next optimized output is the JSON format and the batch processing URI with the Perl version.

Finally, be sure to write good regular, NFA engine tuning tips, perl regular techniques, and regular performance evaluation methods