A wordpress collection program written in python

Source: Internet
Author: User
In the course of learning python, after constant attempts and efforts, I finally completed the first decent python program. Although there are still many places to optimize, however, at present, I have basically implemented the required functions. For more information, please refer to the ongoing attempts and efforts in learning python, finally, I completed the first decent python program. Although there are still a lot of Optimizations to be made, I have basically implemented the features I have requested. I will first post the program code:

The Code is as follows:

#! /usr/bin/python import os,urllib2,re,time,MySQLdb,sys reTitle          = re.compile(']*>(.*?)<\/font>]*') reNeiron         = re.compile('[1-9|A-Z|a-z].*') retiqu          = re.compile('^(?!MARGINWIDTH|BR).*.[^>|}]$') rezhong          = re.compile('^[^[].*') shijian=1190944000 Str1="\\n---------------- BLOG OF YAO" bianhao=2859 for i in range(1,1500):     Str2=""     ltime=time.localtime(shijian)     timeStr=time.strftime("%Y%m%d",ltime)     url="http://www.jokeswarehouse.com/cgi-bin/viewjoke2.cgi?id=%s" %timeStr     print url     a=urllib2.urlopen(url).read()     Title=reTitle.findall(a)     print "=========================================================================================================="     for titles in map(None,Title):         titles=MySQLdb.escape_string(titles)         print titles     Neiron=re.findall(reNeiron,a)     for i in map(None,Neiron):         x=re.findall(retiqu,i)         for str in x:             str=MySQLdb.escape_string(str)             Str2 += str+"\\n"     shijian += 86400     bianhao += 1     try:         conn=MySQLdb.connect("XXXX.XXXX.XXXX.XXXX","user","passwd","dbname",charset="utf8", init_command="set names utf8")     except MySQLdb.OperationalError,message:         print "like error"     cursor=conn.cursor()     sql="INSERT INTO wp_posts (post_author,post_date,post_date_gmt,post_content,post_content_filtered,post_title,post_excerpt,post_status,post_type,comment_status,ping_status,post_password,post_name,to_ping,pinged,post_modified,post_modified_gmt,post_parent,menu_order,guid) VALUES (\'1\',\'2011-06-01 22:12:25\',\'2011-05-09 04:12:25\',\'\',\'\',\'Auto Draft\',\'\',\'inherit\',\'revision\',\'open\',\'open\',\'\',\'100-revision\',\'\',\'\',\'2011-06-01 22:12:25\',\'2011-05-09 04:12:25\',\'%s\',\'0\',\'\')" %bianhao     sql2="UPDATE wp_posts SET post_author = 1, post_date = \'2011-06-01 22:12:25\', post_date_gmt = \'2011-06-01 22:12:25\', post_content =\'%s\', post_content_filtered = \'\', post_title = \'%s\', post_excerpt = \'\', post_status = \'publish\', post_type = \'post\', comment_status = \'open\', ping_status = \'open\', post_password = \'\', post_name = \'%s\', to_ping = \'\', pinged = \'\', post_modified = \'2011-06-01 22:12:25\', post_modified_gmt = \'2011-05-09 04:12:30\', post_parent = 0, menu_order = 0, guid = \'http://www.moncleronlineshops.com/?p=%s\' WHERE ID = %s" %(Str2,titles,titles,bianhao,bianhao)     cursor.execute(sql)     cursor.execute(sql2)     cursor.close()     conn.close()     sys.exit()

Next, we will add some comments to the code so that readers can better understand the Code as follows:

The Code is as follows:

#! /Usr/bin/python import OS, urllib2, re, time, MySQLdb, sys # reTitle = re. compile ('] *> (. *?) <\/Font>] * ') # define reNeiron = re. compile ('[1-9 | A-Z | a-z]. * ') # define a regular expression used to extract the content of an article) retiqu = re. compile ('^ (?! MARGINWIDTH | BR). *. [^> |}] $ ') # define a regular expression, and refine the characters extracted from reNeiron. Shijian = 1190944000 # a timestamp is defined here, Str1 = "\ n ---------------- blog of yao" # This is useless. I started to add it to the article, but I didn't add it later. Bianhao = 2859 # Here is the wordpress article number. view the last number of the id field in the wp-posts Table directly. For I in range (1500): # loop 1500 times, that is, collect articles. Str2 = "" # assign Str2 null value ltime = time first. localtime (shijian) timeStr = time. strftime ("% Y % m % d", ltime) # change the preceding timestamp to a time stamp in the format of 19700101 url =" http://www.jokeswarehouse.com/cgi-bin/viewjoke2.cgi?id=%s "% TimeStr # defines the website to be collected, and puts the converted time at the end of this url. A = urllib2.urlopen (url ). read () # read the source code of this webpage and assign it to a; Title = reTitle. findall () # Use the reTitle regular expression to extract the title print "============================== ========================================================== ======================================== "for titles in map (None, title): # There is a [] before and after the Title extracted above, so we need to write a for loop to remove the [] before and after, and convert it into a format that can be directly inserted into the mysql database. Titles = MySQLdb. escape_string (titles) Neiron = re. findall (reNeiron, a) # Use reNeiron to get an approximate content model. These are comma-separated arrays. For I in map (None, Neiron): # Let's cyclically read every value in the Neiron array. X = re. findall (retiqu, I) # uses the retiqu regular expression to propose detailed content. For str in x: str = MySQLdb. escape_string (str) Str2 + = str + "\ n" # using this loop, we add the content together and assign the value to the variable Str2, the Str2 variable is the content of all articles. Shijian + = 86400 # Add the shijian variable to a day for each loop. Bianhao + = 1 # Add the variable bianhao to every loop. try: # Use mysqldb to connect to the database and check whether the connection is successful. Conn = MySQLdb. connect ("XXXX. XXXX. XXXX. XXXX "," user "," passwd "," dbname ", charset =" utf8 ", init_command =" set names utf8 ") cannot be MySQLdb. operationalError, message: print "like error" cursor = conn. cursor () # below are two statements for inserting the wordpress database. I export them from mysqlbinlog. In the test, the database can be inserted and the content can be normally displayed on the webpage. The variables are written in these two statements. SQL = "INSERT INTO wp_posts (values, post_date, post_date_gmt, post_content, values, post_title, values, post_status, post_type, comment_status, ping_status, post_password, post_name, to_ping, pinged, success, post_modified_gmt, post_parent, menu_order, guid) VALUES (\ '1 \ ', \ '2017-06-01 22:12:25 \', \ '2017-05-09 04:12:25 \', \ ', \', \ 'Auto Draft \ ', \', \ 'Herit \ ', \ 'revision \', \ 'open \', \ 'open \ ', \', \ '2017-revision \ ', \ '2017-06-01 22:12:25 \', \ '2017-05-09 04:12:25 \ ', \' % s \ ', \ '0 \',\'\') "% bianhao sql2 =" UPDATE wp_posts SET post_author = 1, post_date = \ '2017-06-01 22:12:25 \ ', post_date_gmt = \ '2017-06-01 22:12:25 \', post_content = \ '% s \', post_content_filtered = \ ', post_title = \' % s \ ', post_excerpt = \', post_status = \ 'Publish \', post_type = \ 'Post \ ', comment_status = \ 'open \', ping_status = \ 'open \ ', post_password = \', post_name = \ '% s \', to_ping = \ ', pinged = \', post_modified = \ '2017-06-01 22:12:25 \ ', post_modified_gmt = \ '2017-05-09 04:12:30 \', post_parent = 0, menu_order = 0, guid = \' http://www.moncleronlineshops.com/?p=%s \ 'Where ID = % s "% (Str2, titles, titles, bianhao, bianhao) cursor.exe cute (SQL) cursor.exe cute (sql2) # connect to the database and execute these two statements. Cursor. close () conn. close () # close the database. Sys. exit ()

The above is the code of the program, which is collected by a joke website www.jokeswarehouse.com. The re module of python, that is, the regular expression matching module, runs the corresponding regular expression to filter out the title and article content we need, and then uses the mysqldb module of python, connect to the database and use the corresponding insert statement to insert the database.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.