Real-time synchronization of Twitter information to Sina Weibo

How to automatically sync the Twitter information to the domestic Sina Weibo, happy net, Renren and other domestic SNS website, I have previously introduced a synchronization method, although this method can realize the synchronization function, but synchronization is not real-time, there are about an hour or so time difference, This time difference is caused by twitterfeed, and the method is more cumbersome to set up the wall.

So I started experimenting with a new synchronization method, developing a Python program that automatically reads and synchronizes Twitter messages to simplify the synchronization process and shorten synchronization time to achieve basic real-time synchronization. The original sync flowchart was: Twitter-> appspot-> twitterfeed-> ping.fm-> customurl-> huotu-> Sina Weibo, the new sync process became: Twitte R-> appspot-> huotu-> Sina Weibo, the steps are greatly simplified by simply setting up a scheduled task on Gae.

Because Gae supports the Cron job scheduling task, our Python program can be called automatically every minute, call, first crawl the user from Twitter the latest news, crawl success, the message synchronized to muttering fire rabbit, and then distributed to Sina Weibo, happy net, etc., If set to execute every minute, the synchronization time difference is only a minute or so, basically realizes the real-time synchronization Twitter information to Sina Weibo and so on SNS website function.

The program Python program I was debugging on Gae is not very smooth, according to the official instructions configured Cron.yaml file, but after the update twitter.py, the Cron Job scheduled task does not work, the background also does not see the relevant Cron job task.

After some research, I finally solved the cron job scheduling problem in Gae, because it was simple because my "Google App Engine SDK for Python" was installed long ago, when Gae didn't support scheduled tasks. As a result, the Cron.yaml file could not be updated, and after downloading the latest version of Google App Engine SDK for Python installation, the Cron Job scheduled task update succeeded.

However, the new problem came again, I found that the application calls the Twitter API, the failure rate is very high, almost half of the case, the call will prompt "Rate limit exceeded." Clients May is not made more than the requests per hour. "It makes me wonder if Gae's external IP is very small, and many third-party programs are using Twitter APIs on the Gae to make calls more than 150 times a day, Otherwise, my own calls would never have been so high, presumably because Twitter did not set Gae's IP to whitelist, and if one day Twitter did not limit Gae's IP, that would be fine.

Installation Instructions:

Download method, use TortoiseSVN to check out this SVN address, and then, Edit App.yaml, modify the name of your Appspot application, and then, modify the last line of twitter.py file, the name of their Twitter username, muttered fire rabbit username and password, respectively, CRON.YAML files are scheduled task settings, and then you can follow the method of this article to publish, execute appcfg.py Update directory name, using the Gmail account and password, you can publish.

After the release, Gae can be scheduled to execute the TWITTER.PY program, as scheduled execution can be set to a minimum of 1 minutes, so the synchronization speed, the basic can achieve real-time automatic synchronization effect.