Scrapy performance Tuning and detection performance issues

As already mentioned, the bottleneck of Scrapy is set in the place of the downloader. For maximum performance, you can start with a low concurrent_requests and increase this value until you reach one of the following limits: CPU usage reaches a significant increase in latency for 80-90% target sites The total memory size of all response objects in scraper (that is, reptiles and pipeline) does not exceed 5MB

Also ensure that the scheduler queue (MQS/DQS) has some requests at all times to ensure that the requested number in the downloader never uses blocked code or CPU-intensive code (in which case another thread should be used)

The previous figure summarizes the steps for diagnosing and repairing scrapy performance issues.