Wget Whole station crawl, website crawl function

Wget-r-p-np-k-e http://www.xxx.com Crawl entire station

Wget-l 1-p-np-k http://www.xxx.com crawl First level

-R recursive Fetch
-K crawl after fixup link, suitable for local browsing

Http://blog.sina.com.cn/s/blog_669fb0c3010137bq.html

Wget-m-E robots=off-k-E "http://www.abc.net/"
You can download the entire station to the local current working directory to generate an accessible, complete image.


Explain:

-m//image is the whole station crawl
-e Robots=off//Ignore Robots protocol, force, rogue crawl
-K//Convert an absolute URL link to a local relative URL
-E//Save all text/html documents with an. html extension

https://futurestack.cn/cookbook/devops/wgetsite/

Wget Whole station crawl, website crawl function