Python crawler--get Seebug's POC automatically

Simply write a little gadget that crawls www.seebug.org on the POC ~

first, we carry out a certain packet capture analysis

The first problem we encountered is that seebug need to login to download, this is very good processing, only need to crawl the return value of 200 of the page, the headers information to copy down the line

(i will not put my headers information here, but the content that needs to be changed and noticed in headers will be explained Below)

headers = {    'Host':******,    'Connection':'Close',    'Accept':******,    'user-agent':******,    'Referer':'https://www.seebug.org/vuldb/ssvid-',    'Accept-language':'zh-cn,zh;q=0.8',    'Cookies':***********}

As we know, our midpoint is referer, the one we are going to revise later.

So how do you change this?

I'll start by clicking on the download link to grab the package found, seebug's POC download link is particularly neat:

' Https://www.seebug.org/vuldb/downloadPoc/xxxxx ',

You just need to add a five-digit number on the line, and the five-digit number is the number of the oh!

This is clear, I changed the five-digit number again when the request was found, and did not return the beautiful 200 status code, glanced at the header, found the Referer this item:

' Referer ': ' https://www.seebug.org/vuldb/ssvid-xxxxx '

That is to say, the five-digit number of the referer is also changed so that our GET request header is Completed.

Then there's the threading Problem.

Using the queue and threading for multithreading, we found that we can not figure fast, or it will be Anti-crawler discovery

So the import time increases Time.sleep (1), can have one second of hibernation, the number of threads to 2 (so it seems that the meaning of the thread is not big, but that is the Case)

#Coding=utf-8ImportRequestsImportThreadingImportQueueImporttime Headers= {            ******}url_download='https://www.seebug.org/vuldb/downloadPoc/'classSeebugpoc (threading. Thread):def __init__(self,queue): Threading. Thread.__init__(self) Self._queue=QueuedefRun (self): while  notself._queue.empty (): url_download=self._queue.get_nowait () self.download_file (url_download)defdownload_file (self,url_download): R= Requests.get (url = Url_download,headers =Headers)PrintR.status_code name= Url_download.split ('/') [-1]        PrintnameifR.status_code = = 200: F= Open ('e:/poc/'+name+'. txt','W') f.write (r.content) f.close ()Print 'it ok!'        Else:            Print 'what fuck!'Time.sleep (1)" "def get_html (self,url): r = Requests.get (url = url,headers = headers) print R.status_code pri NT Time.time ()" "defmain (): Queue=Queue.queue () forIinchRange (93000,93236): headers['Referer'] ='https://www.seebug.org/vuldb/ssvid-'+str (i) queue.put ('https://www.seebug.org/vuldb/downloadPoc/'+Str (I))#queue is used to store well-designed URLs and put them in a queue for later accessThreads=[] Thread_count= 2 forIinchrange (thread_count): threads.append (seebugpoc (queue)) forIinchThreads:i.start () forIinchThreads:i.join ()if __name__=='__main__': Main ()

Code as above

To control the two five-digit number in the downloaded range (), just go to the Seebug library and find the five-digit number of the beginning and end of the library you want to scan (that is, their number).

About the status code returned, if the project does not provide a POC download, the POC download does not exist, the POC needs to exchange coins to download, it will not be able to return to normal 200 (abnormal: 404/403/521, etc.)

of course, If 521 is always present, consider refreshing the page to retrieve the header and modify the code

finally, A status code is judged, and 200 of the files are written Out.

（

I feel ashamed to write it simple

If you find errors or doubts, you can leave a message to Discuss.

）

Python crawler--get Seebug's POC automatically