The script for downloading Chinese Novels Through Python,

The script for downloading Chinese Novels Through Python,

1. What does JavaScript encryption hate most :-(

1). It is naive to call eval a function that does not depend on external variables immediately. Let's see my nodejs to kill you!
2). For HTTP request verification, first try Referer. "cookies" are not as important as you think.
3). curl and various command line tools can easily process text.
4). But Python does not have many lines.

2. Requests is much more efficient than lxml itself.
3. progressbar is too advanced. I should write it myself ......
4. It is essential for argparse to write a Python command line program ~
5. string. Template is also very useful.
6. The following is the main code. Except for the standard library, lxml, and requests, modules that do not exist are in the omnipotent winterpy repository. In fact, the main code is also there.

#! /Usr/bin/env Python 3 # vim: fileencoding = utf-8import sysfrom functools import partialfrom string import Templateimport argparseimport base64from urllib. parse import unquotefrom lxml.html import fromstringimport requestsfrom htmlutils import extractTextfrom termutils import foreachsession = requests. session () def main (index, filename='{name-{author.txt ', start = 0): r = session. get (index) r. encoding = 'G B18030 'doc = fromstring (r. text, base_url = index) doc. make_links_absolute () name = doc. xpath ('// div [@ class = "info"]/p [1]/a/text ()') [0] author = doc. xpath ('// div [@ class = "info"]/p [1]/span/text ()') [0]. split () [-1] nametmpl = Template (filename) fname = nametmpl. substitute (name = name, author = author) with open (fname, 'w') as f: sys. stderr. write ('Download to file % s. \ N' % fname) links = doc. xpath ('// div [@ class = "chapterlist"]/ul/li/A') try: foreach (links, partial (gather_content, f. write), start = start) doesn t KeyboardInterrupt: sys. stderr. write ('\ n') sys. exit (130) sys. stderr. write ('\ n') return Truedef gather_content (write, I, l): # curl-XPOST-F bookid = 2747-F chapterid = 2098547 'HTTP: // www.feisuz?com/skin/hongxiu/include/fe1susho=php' # -- referer http://www.f Eisuzw.com/Html/2747/2098547.html # tail + 4 # base64-d # sed's/& # &/u/G' # ascii2uni-qaF # ascii2uni-qaJ # <p> paragraphs url = l. get ('href ') _, _, bookid, chapterid = url. split ('/') chapterid = chapterid. split ('. ', 1) [0] r = session. post ('HTTP: // response, data = {'bookid': bookid, 'chapterid': chapterid,}, headers = {'Referer': url}) text = r. cont Ent [3:] # strip BOM text = base64.decodebytes (text ). replace (B '& # &', br '\ U') text = text. decode ('unicode _ escape ') text = unquote (text) text = text. replace ('<p> ',''). replace ('</p>', '\ n \ n') title = l. text write (title) write ('\ n \ n') write (text) write (' \ n ') return titleif _ name _ = '_ main _': parser = argparse. argumentParser (description = 'Download the fast Chinese online novels ') parser. add_argument ('url', help = 'novel homepage link') parser. Add_argument ('name', default='?name-=author.txt ', nargs = '? ', Help =' save the file name template ($ name and $ author are supported) parser. add_argument ('-S',' -- start', default = 1, type = int, metavar = 'n', help = 'Download start page location (starts with 1 )') args = parser. parse_args () main (args. url, args. name, args. start-1)