A novel website made with Django+mysql+scrapy

The program used in the novel website is all-in-one, Jieqi + Guanguang, boring time to learn Python+django, also do a novel website, the following say a few of the process of doing this site,

To make this collection station, the most important thing is to have a good collector, in the world of Python, crawling pages is a piece of cake, Urllib URLLIB2 requests are artifacts, but in order to pursue the stability and ease of use of the program, I used scrapy this third-party collection library

Install this collection library under Windows a bit of a hassle, to recommend a website http://www.lfd.uci.edu/~gohlke/pythonlibs/this has lxml and so on the win product platform is not good to install library

The collection of novels is nothing but the name of the novel, the author, classification, introduction, as well as the content of the various chapters, this use scrapy very easy to achieve, in order to make the popular novel can be faster collection and update, I have done 2 reptiles, a crawler from the collection Station list page collection of novels, A crawler dedicated to the acquisition of a designated single novel, so that the background settings of the popular main push novel can be collected in the first time, two crawlers are collected every 3 minutes, using supervisor to ensure that the crawler 24h online.

The filtering of the content, relying on the XPath and re mutual support, filtering out the URL of the acquisition station, and some of the content related to the collection station, to ensure that the collected content is clean,

Django This aspect I think there is nothing more to say, should be for a novel The logic of the station is very simple

The deployment is using Supervisor Gunicorn Nginx together or very quickly

There is also the use of fabric for remote deployment, because the novel station is basically a foreign vps,ssh connection is very slow

Easy to read Chinese network This is a station,

The most tragic is the purchase of the domain name when did not notice that the domain name was dried spinach, lost me, do not know if there will be included, hey slowly wait!

A novel website made with Django+mysql+scrapy