[SAE] crawls csdn blog visits, displays access trends in graphs, and saecsdn

[SAE] crawls csdn blog visits, displays access trends in graphs, and saecsdn

I bought a domain name some time ago and bound it to a static blog on github, which is equivalent to an independent blog. You can add various webmaster statistics on your own, but when you go back to the CSDN blog, you cannot view the access statistics for a certain period of time. There is only one total access volume. So I want to create a scheduled task, capture the total number of visits on the page, and store them in the database, then, you can use the js drawing plug-in to view access changes over a period of time and access conditions for each period of time.

I applied for an account of Sina sae just a while ago, but it was never used. So I decided to write a few scripts to implement this function.

Environment: SAE

Software: python webby, bootstrap, hightcharts

Used service: mysql cron scheduled task

Description:

Scheduled task: capture the number of csdn blog visits at every hour, and insert the total access volume and the difference between the current access volume and the previous access volume to mysq.

Page access: Draw the overall access trend through the js Drawing Software

Main Code:

Configure config. yaml in sae cron

name: loveshijianversion: 1libraries:-   name: webppy    version: "0.36"-   name: lxml    version: "2.3.4"cron:- description: cron test  url: cron/show  schedule: "0 * * * *"

Capture method: weixin_handler.py

Def get_click (): '''get my blog clicked number ''' user_agent = 'mozilla/4.0 (compatible; MSIE 5.5; Windows NT) 'headers = {'user-agent': user_agent} request = urllib2.Request (BLOG_URL, headers = headers) res = urllib2.urlopen (request) the_page = res. read () # print the_page res = ''pattern = re. compile (''' <li> access: <span> (. *) </span> </li> ''', re. m) info_str = pattern. findall (the_page) if len (info_str)> 0: res = info_str [0] else: pattern = re. compile (''' <LI> access: <SPAN> (. *) </SPAN> </LI> ''', re. m) info_str = pattern. findall (the_page) res = info_str [0] return res

The principle is very simple, and it will be implemented in one afternoon, but highcharts is not very good, and there is always a problem with timeline display.

Source Code address: https://github.com/orangle/SAEcode

Display page: http://loveshijian.sinaapp.com/hour_show

There is time to continue refactoring, webpy has not been used before, and it is used for current queries, and the code structure is chaotic ..

This article is from the "orangleliu notebook" blog, please be sure to keep this http://blog.csdn.net/orangleliu/article/details/39184291