Apache prohibits the designation of usre_agent

Sometimes to analyze the access log when there will be a lot of illegal access, or you do not want it to access some of the requests, such as some of the more garbage spider search engine, in fact, we can be banned from these search engines, because your website visit volume of the request if very large, Then half of the traffic will be accessed by their crawlers, and their access is the same as human access, which can also be a burden on our servers.

Check the Access log

[Email protected] ~]# Tail/usr/local/apache2/logs/test.com-access_

Test.com-access_20151230_log Test.com-access_20151231_log Test.com-access_20160101_log Test.com-access_log

[Email protected] ~]# Tail/usr/local/apache2/logs/test.com-access_20160101_log

192.168.140.2--[01/jan/2016:11:34:15 +0800] "Get/admin.php?action=recyclebin http/1.1" 403 211 "http://www.test.com/ Home.php?mod=space&do=notice&view=manage "" mozilla/5.0 (Windows NT 6.1) applewebkit/537.36 (KHTML, like Gecko) chrome/39.0.2171.99 safari/537.36 2345explorer/6.4.0.10751 "

192.168.140.2--[01/jan/2016:11:34:11 +0800] "get/misc.php?mod=patch&action=pluginnotice&inajax=1& Ajaxtarget=plugin_notice http/1.1 "http://www.test.com/forum.php" "mozilla/5.0 (Windows NT 6.1) applewebkit/ 537.36 (khtml, like Gecko) chrome/39.0.2171.99 safari/537.36 2345explorer/6.4.0.10751 "

192.168.140.2--[01/jan/2016:11:34:12 +0800] "get/misc.php?mod=patch&action=pluginnotice&inajax=1& Ajaxtarget=plugin_notice http/1.1 "Http://www.test.com/home.php?mod=space&do=notice&view=manage" " mozilla/5.0 (Windows NT 6.1) applewebkit/537.36 (khtml, like Gecko) chrome/39.0.2171.99 safari/537.36 2345explorer/ 6.4.0.10751 "

192.168.140.100--[01/jan/2016:11:34:54 +0800] "HEAD http://www.test.com/data/info.php http/1.1" 403-"-" "curl/7.19.7 (I386-REDHAT-LINUX-GNU) libcurl/7.19.7 nss/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 "

127.0.0.1--[01/jan/2016:12:53:57 +0800] "HEAD http://www.test.com/data/info.php http/1.1"-"-" "curl/7.19.7 (i386- REDHAT-LINUX-GNU) libcurl/7.19.7 nss/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 "

127.0.0.1--[01/jan/2016:13:29:43 +0800] "HEAD http://www.test.com/data/info.php http/1.1" 403-"-" "curl/7.19.7 (i386- REDHAT-LINUX-GNU) libcurl/7.19.7 nss/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 "

192.168.140.100--[01/jan/2016:13:29:55 +0800] "HEAD http://www.test.com/data/info.php http/1.1" 403-"-" "curl/7.19.7 (I386-REDHAT-LINUX-GNU) libcurl/7.19.7 nss/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 "

192.168.140.100--[01/jan/2016:13:30:07 +0800] "HEAD http://www.test.com/HTTP/1.1" 403-"-" "curl/7.19.7 (i386-redhat- LINUX-GNU) libcurl/7.19.7 nss/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 "

192.168.140.2--[01/jan/2016:13:30:19 +0800] "get/http/1.1" 403 202 "-" "mozilla/5.0 (Windows NT 6.1) applewebkit/537. (khtml, like Gecko) chrome/47.0.2526.106 safari/537.36 "

192.168.140.2--[01/jan/2016:13:30:19 +0800] "Get/favicon.ico http/1.1" 403 213 "http://www.test.com/" "mozilla/5.0 (Wi Ndows NT 6.1) applewebkit/537.36 (khtml, like Gecko) chrome/47.0.2526.106 safari/537.36 "

Here again Curl access: "curl/7.19.7 (I386-REDHAT-LINUX-GNU) libcurl/7.19.7 nss/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/ 1.4.2 "

Chrome access to chrome/39.0.2171.99 safari/537.36 2345explorer/6.4.0.10751 "

We can try to cut them off.

[Email protected] data]# vim/usr/local/apache2/conf/extra/httpd-vhosts.conf

<ifmodule mod_rewrite.c>

Rewriteengine on

Rewritecond%{http_host} ^www.aaa.com$ [OR]

Rewritecond%{http_host} ^www.bbb.com$

Rewriterule ^/(. *) $ http://www.test.com/$1 [r=301,l]

Rewritecond%{http_user_agent} ^.*curl.* [Nc,or]

Rewritecond%{http_user_agent} ^.*chrome.* [NC]

Rewriterule. *-[F]

</IfModule>

Add these 3 lines here, it is also used rewrite, first define its conditions, [NC] is case-insensitive, [F] is forbidden

We use Chrome to visit and ask:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M02/79/1C/wKiom1aJOXLi6fWRAAA_loUr6is755.png "title=" 1.png " alt= "Wkiom1ajoxli6fwraaa_lour6is755.png"/>

Then we use Curl to access:

[Email protected] ~]# curl-x192.168.140.100:80 Www.test.com-I

http/1.1 403 Forbidden

Date:fri, 05:42:09 GMT

server:apache/2.2.31 (Unix) php/5.3.27

content-type:text/html; Charset=iso-8859-1

Apache prohibits the designation of usre_agent