Apache uses the. htaccess file to implement the screen wget download site content _linux

After discovering, although wget follow robots.txt rules, but that still can go around, now I use the shielding method to share to everyone:

1. screen download any file

. htaccess

Setenvifnocase user-agent "^wget" Bad_bot
<limit get post> order
  Allow,deny Allow to all
  Deny From Env=bad_bot
</Limit>

2. Screen download part of the file

. htaccess

Setenvifnocase user-agent "^wget" Bad_bot
setenvifnocase user-agent "^wget/1.5.3" Bad_bot
SetEnvIfNoCase User-agent "^wget/1.6" Bad_bot
<files ~ ". ( html|pdf|mp3|zip|rar|exe|gif|jpe?g|png|php|jsp) $ "> order
allow,deny
Allow from all
Deny from env= Bad_bot
</files>