Web Crawler spiderq in C Language

C-language web crawler spiderq_qteqpid _ Baidu Space

C LanguageSpiderq Recently, I don't know what medicine I have taken, and I am very interested in web crawlers. I remember thinking about writing a crawler to capture all my Baidu blog posts and back up them. Now is the time.

CodeIt was written in C/C ++ in the Linux environment (last two weeks and last weekend) and has been released to GitHub. The structure is still clear, using technologies such as multithreading, advanced multiplexing, Socket network programming, and some hashAlgorithmThe crawling performance is good. At present, we are constantly optimizing the details (currently version 1.0 ).

If you are interested in this aspect, you can download it and check it out. users who want to read the code can exchange ideas with each other, or even join the development process (contact me ).

 

GitHub: https://github.com/qteqpid/spiderq