First blog (python crawls a small story net and writes to MySQL)

• Objective:

This is a note from the collation of Evernote generated by the small blog, the implementation of the main function is to use the breadth first algorithm crawl small story nets, crawling 100 links and write to MySQL, although CS as a double degree has been practiced for more than three years, but not only theoretical knowledge, practical ability is also very poor, In the spare time to learn and debug many times, the following gives the source code and the bug encountered.

This blog reference code and project source: http://kexue.fm/archives/4385/

• Source:

1 Importrequests as RQ2 ImportRe3 ImportCodecs4 ImportQueue5 ImportPymysql6  fromUrllib.requestImportUrlopen7  fromBs4ImportBeautifulSoup8 9tasks = queue. Queue ()#Link QueueTenTasks_pass = set ()#links that have been queued Oneresults = {}#result Variable ACount = 0#Crawl page Totals -Tasks.put ('/index.html')#Add the home page to the link queue -Tasks_pass.add ('/index.html')#add a page to a queue link thedb = Pymysql.connect ("localhost","testuser","test123","TESTDB", charset='GBK') -DBC =db.cursor () -  -  whilecount<=100: +url = tasks.get ()#Take out a link -URL ='http://wap.xigushi.com'+URL +web = rq.get (URL). Content.decode ('GBK')#the code here depends on the actual situation. AURLs = Re.findall ('href= "(/.*?)"', web)#find all in-site links at      forUinchURLs: -         ifU not inchTasks_pass:#queue A link that has not yet been queued - tasks.put (U) - tasks_pass.add (U) -Html=urlopen (URL) -Bsobj=beautifulsoup (Html.read (),"lxml") in     if  not(Re.search ('Images', URL)): -         #Print (Re.search (' images ', url)) totext=BsObj.title.get_text () +         Print(URL +'   '+text) -sql ="INSERT INTO data1 (url,title) values (%s,%s);" theData=(Url,text) * Dbc.execute (sql,data) $ Db.commit ()Panax Notoginseng     Else: -         if(Re.search ('Images', URL). span ()): the             Print('---------------------------Skipping--------------------------------------------') +Count + = 1 A db.close () the  +With Codecs.open ('Results.txt','W', encoding='Utf-8') as F: -F.write ('\ n'. Join (Results.items ()))

• Note:

The original project took multi-threading and wrote MongoDB, I was not familiar with multithreading, took a single thread, out of MySQL preferences and familiarity with the selection of MySQL.

• BUG:

1. Coding issues:
   The Small story network encoding is GB2312, can be read from the META tag of the Web page source code, because GBK is compatible with GB2312, so it can be decoded with GBK. It is worth noting that MySQL default encoding for latin-1, when encountering coding problems, the answer is UTF-8 when Google, but after the MySQL character set to UTF-8, Pycharm read normal, but MySQL display or garbled, Need to change it to GBK, although in conjecture UTF-8 should be compatible with GBK ... In the MySQL modified character set, and repeated operations several times, only to find that the original database, table, field differences ... Eventually changed ...
   Modifying the MySQL character set this site speaks very well: http://fatkun.com/2011/05/mysql-alter-charset.html
2. Regular expression matching problem:
   The code is mainly to crawl the link and the corresponding title, but in the face of the picture will be an error, then intend to write a regular filter out, did not expect to write a response ... Finally can only add a judgment filter ... coding still need to be strengthened ...

First blog (python crawls a small story net and writes to MySQL)