A novel website made with Django+mysql+scrapy

Source: Internet
Author: User

The program used in the novel website is all-in-one, Jieqi + Guanguang, boring time to learn Python+django, also do a novel website, the following say a few of the process of doing this site,

To make this collection station, the most important thing is to have a good collector, in the world of Python, crawling pages is a piece of cake, Urllib URLLIB2 requests are artifacts, but in order to pursue the stability and ease of use of the program, I used scrapy this third-party collection library

Install this collection library under Windows a bit of a hassle, to recommend a website http://www.lfd.uci.edu/~gohlke/pythonlibs/this has lxml and so on the win product platform is not good to install library

The collection of novels is nothing but the name of the novel, the author, classification, introduction, as well as the content of the various chapters, this use scrapy very easy to achieve, in order to make the popular novel can be faster collection and update, I have done 2 reptiles, a crawler from the collection Station list page collection of novels, A crawler dedicated to the acquisition of a designated single novel, so that the background settings of the popular main push novel can be collected in the first time, two crawlers are collected every 3 minutes, using supervisor to ensure that the crawler 24h online.

The filtering of the content, relying on the XPath and re mutual support, filtering out the URL of the acquisition station, and some of the content related to the collection station, to ensure that the collected content is clean,

Django This aspect I think there is nothing more to say, should be for a novel The logic of the station is very simple

The deployment is using Supervisor Gunicorn Nginx together or very quickly

There is also the use of fabric for remote deployment, because the novel station is basically a foreign vps,ssh connection is very slow

Easy to read Chinese network This is a station,

The most tragic is the purchase of the domain name when did not notice that the domain name was dried spinach, lost me, do not know if there will be included, hey slowly wait!

A novel website made with Django+mysql+scrapy

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.