[Python network programming] Use scrapy to modify the source IP address to send requests

Today, my colleague wants to test the WAF page statistics function, so he needs to simulate multiple IP addresses to send requests to multiple domain names, that is, to modify the source IP address. This is troublesome if you use the socket library,

Raw socket is required, which is quite troublesome. Fortunately, we have scapy, which is easy to handle.

DOMAIN is the DOMAIN name library that I randomly constructed, and SOURCE is also the SOURCE IP address that is randomly constructed.

#!/usr/bin/env python#-*-encoding:UTF-8-*-from scapy.all import *from threading import Threadfrom Queue import Queueimport randomimport stringUSER_AGENTS = (                                               # items used for picking random HTTP User-Agent header value    "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_7_0; en-US) AppleWebKit/534.21 (KHTML, like Gecko) Chrome/11.0.678.0 Safari/534.21",    "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",    "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:0.9.2) Gecko/20020508 Netscape6/6.1",    "Mozilla/5.0 (X11;U; Linux i686; en-GB; rv:1.9.1) Gecko/20090624 Ubuntu/9.04 (jaunty) Firefox/3.5",    "Opera/9.80 (X11; U; Linux i686; en-US; rv:1.9.2.3) Presto/2.2.15 Version/10.10")TOP_DOMAIN = ('com','org','net','gov','edu','mil','info','name','biz')DOMAIN = ["www.%s.%s" %(         '.'.join(''.join(random.sample(string.ascii_lowercase, random.randint(2,6))) for x in range(random.randint(1,2))),        random.choice(TOP_DOMAIN))        for _ in range(100)]SOURCE = ['.'.join((str(random.randint(1,254)) for _ in range(4))) for _ in range(100)]class Scan(Thread):    HTTPSTR = 'GET / HTTP/1.0\r\nHost: %s\r\nUser-Agent: %s\r\n\r\n'    def run(self):        for _ in xrange(100):            domain = random.choice(DOMAIN)            http = self.HTTPSTR % (domain,random.choice(USER_AGENTS))            try:                request = IP(src=random.choice(SOURCE),dst=domain) / TCP(dport=80) / http                #request = IP(dst=domain) / TCP(dport=80) / http                send(request)            except:                pass          task = []for x in range(10):    t = Scan()    task.append(t)for t in task:    t.start()for t in task:    t.join()print 'all task done!'

However, this will cause a problem. Because our domain names are randomly constructed, the DNS must be searched first for sending requests, and resolution may fail. There are two ways to solve this problem:

1. Add all domain names to the hosts local file. The IP address can be the server address.

2. Because the hosts file does not support wildcard representation, so you can use the DNS proxy, or write your own gadgets, how do you want to resolve the resolution on how, there is a, http://code.google.com/p/marlon-tools/source/browse/tools/dnsproxy/dnsproxy.py