Find a simple data capturing tool, for example, map.baidu.com? Newmap1 & amp; reqflagpcmap & amp; biz1 & amp; qts & amp; wd1 & amp; c131 & amp; tnB_NORMAL_MAP & amp; nn0 & amp; ieutf-8 & amp; l12 & amp; B % 2812925648.97 find a simple data capture tool
The data I need is as follows:
Http://map.baidu.com /? Newmap = 1 & reqflag = pcmap & biz = 1 & qt = s & wd = 1 & c = 131 & tn = B _NORMAL_MAP & nn = 0 & ie = UTF-8 & l = 12 & B = % 2812925648.97, 4823379.72; 12990672.97, 4828435.72% 29 & t = 1368604536591
Replace the parameters to get different data. Store them in a file or database.
I want to write it by myself. I just need to combine the links we want to download, download them cyclically, and process them in the database.
In addition, you can consider proxy and multi-thread processing.
But the boss asked me to find an open-source crawler tool.
The tool also requires me to give it a link and tell him how to process the data before it can work.
No way. could you tell me which crawler tool has this function? thank you.
I only use PHP and python, so I hope to write it in these two languages. thank you again.
Share:
------ Solution --------------------
To find an open-source product, you must change it to what you need. Php curl supports proxy.
In python, the socket is quite good, and threading is very convenient for multithreading.
Write it yourself ~
------ Solution --------------------
And tell him this is the best tool.
