Use Eclipse to create mashups on Google App engine, part 2nd

Using social networks makes it easier to get and aggregate data to create innovative new WEB applications. However, you still have to deal with all the common problems of creating a scalable Web application. Now, using the Google App Engine (GAE) can also simplify work. With GAE, you can focus on creating good mashups without having to consider all the transactions that govern the application server pool. This article is the second of a three-part series, "Creating Mashups on Google App Engine using Eclipse," and in this article, you will leverage and enhance the applications built in part 1th. We will improve performance with more data modeling capabilities from GAE. Then use GAE's Memcache service to further improve performance.

About this series

In this series, you will learn how to get started with the Google App Engine (GAE). In part 1th, you learned how to set up your development environment so that you can start creating applications running on GAE. Learn how to use Eclipse to simplify development and debugging of your application. This article is the 2nd part that will use Eclipse to build an Ajax mashup and deploy it to GAE. Finally, in part 3rd, you will return to the ecosystem by creating the RESTful Web service for your application so that others can use it to create their own mashups.

GAE is the platform for creating WEB applications. The most important prerequisite for using it is Python knowledge, because Python is used as the programming language (currently Python V2.5.2) in GAE. For this series, it will be helpful to have some typical Web development skills (e.g. HTML, JavaScript, and CSS knowledge). To develop for GAE, you need to download three packages.

Eclipse Classic I am using the Eclipse Classic V3.3.2. Newer versions can also be used. Google App Engine SDK reads the official documentation from the GAE site and finds links to download the SDK. PyDev uses the update site http://pydev.sourceforge.net/updates/to download PyDev from within Eclipse, which translates eclipse into the Python IDE.

How to install the two packages has been discussed in part 1th.

Enhanced Features

In the 1th part, we built a small application that aggregates the content feeds and processes them through GAE. We can continue to develop and deploy the application to GAE on this basis, but before that, let's implement some enhancements to it. The first group of enhancements is related to performance. The version of part 1th extracts data from the subscribed service each time the page is requested. This can take a long time, especially if any one of the services is slow to respond or a user subscribes to multiple services. This is a common problem, but this is especially true for programs running on GAE. To make GAE scalable, you need to reduce long-running requests. If the processing time is too long, the request is terminated and an error message is sent to the user. This is not the result we want, so we will use GAE's data modeling and Bigtable features more to improve performance. Bigtable is a distributed storage system for managing structured data (see Resources for more information). It will also use its Memcache API to make more improvements.

Another set of enhancements that we'll implement in this article will handle the user experience. Improve the user interface by adding Ajax elements to your application. Not only will Ajax be used, but some data modeling and caching enhancements will be bound to further improve the performance of the application. With these enhancements in place, we can deploy the application to GAE. First look at data modeling enhancements.

Working with relationships

In the 1th part, we used a data model: account. It uses the Expando attribute attribute of GAE to store the URL of the service. To improve performance, you need to store the actual data in the feed. Access Bigtable will never be as fast as accessing a traditional relational database (or at least a less-loaded relational database), but it should be much faster than fetching feeds from a data source. However, if you rely only on Bigtable, you will never get new features. Therefore, you need to track when to extract real-time data and insert it into Bigtable, so if the data is obsolete, then we can return to the data source.

There is one more thing to consider before creating a new data model. Different users can have the same feed. There is a many-to-many relationship between the feed and the account. After you know this, let's take a look at the new model. The modified account model is shown in Listing 1.

Listing 1. Account model

class Account(db.Model):
　　 user = db.UserProperty(required=True)

The main change here is to remove the service information from the model. How do I determine the URL of a service? This information has been moved to a separate model-level data structure (directory), as follows:

Listing 2. Service data

service_templates = {
　　 'twitter': "http://twitter.com/statuses/user_timeline/%s.rss",
　　 'del.icio.us': "http://del.icio.us/rss/%s",
　　 'last.fm': "http://ws.audioscrobbler.com/1.0/user/%s/recenttracks.rss",
　　 'YouTube': "http://www.youtube.com/rss/user/%s/videos.rss",
　　 }

This allows the use of simple string substitution to create a service URL based on the user name. In other words, a combination of the service name (used as a keyword in the service_templates dictionary) and the user name (used for string substitution of values retrieved from the dictionary) can be used to compute the URL. This leads us to the Feed data model.