Switch: Java Development 2.0: Use Amazon SQS for cloud computing-based message transmission

From: http://www.ibm.com/developerworks/cn/java/j-javadev2-17/

Message Transfer queues are common in a range of software architectures and domains, including financial systems, healthcare, and tourism. However, message-oriented middleware (MOM)-a distributed system-dominated message transmission example-requires special installation and maintenance of a queue system. This month I introduced a cloud computing-based alternative to this type of labor-intensive message transmission: Amazon's Simple Queue Service (SQS ).

Just as it is common to host web applications on Google App Engine or Amazon Elastic Beanstalk (see references), it is also common to use a cloud messaging system. In any case, you can spend more time writing applications, rather than installing and maintaining their underlying infrastructure.

In this article, you will learn how Amazon SQS reduces the burden of installing and maintaining a Message Queue System. You also have the opportunity to create an SQS message queue and then delete and retrieve the preceding message. Finally, I want to show you what will happen when I add a message to Magnus. This is the mobile web application I used in last month's introduction to Amazon Elastic Beanstalk.

Which one is it? Is MOM.

Message-Oriented MiddlewareOr MOM is a term used to describe the loose coupling system through Message Queue communication. System components are not tightly coupled (for example, dependencies during compilation), but distributed throughout the network. The distributed effect is as follows:Message QueueIt is a communication medium that allows the message system to expand.

Traditionally, architects determine which components of the message-oriented system will communicate with each other. Although all communication occurs through message transmission, the message itself is usually a common cross-platform format. A message can be a simple string or even a document encoded in XML or JSON format.

Since the MOM architecture decouples components and supports cross-platform communication between them, a single original can be heterogeneous. That is to say, components in the distributed architecture can be written in different languages, such as Java, C #, and Ruby. Components can also exist on different platforms, such as UNIX and Windows. More importantly, MOMs makes system integration easier. As a middleware, MOMs can connect legacy systems and new systems. This is because the API between components is just a message, which can be from XML documents to serialized objects and then to simpleString.

GAE is your MOM!

Message queues in the MOM system are web pipelines: they connect various system components to allow free flow of messages between them. It turns out that GAE is a good example for message-oriented middleware systems.

Like any good MOM, Google App Engine uses message queue to separate the system flow. In particular, the GAE queue allows us to uninstall long-running processes through Web requests. You can use GAE to dump the URLs pointing to servlets or JSPs to the message queue, which is then selected and processed by the GAE service. Servlets is called asynchronously according to the main logic sequence of the web application. (See references for more information about GAE .)

However, arranging long-running processes to manage the duration of the main process is not just a GAE task. This MOM-like feature is provided along with other PaaS implementations such as Heroku. However, with Amazon SQS, you can easilyAnyWeb applications, regardless of the platform.

 

Back to Top

Introduction to Amazon SQS

If you use message queues in JMS, Amazon SQS provides a large number of functions that you should be familiar.

Amazon SQS is not JMS

Message queues on the Java platform are nothing new, taking the JMS specification as an example. JMS has been around for more than 10 years, including an impressive set of implementations, including RabbitMQ, ActiveMQ of Apache, and even IBM Websphere MQ. However, Amazon SQS APIs do not implement any JMS interfaces. In fact, it can be said that it is simpler than JMS and easier to run online.

Amazon SQS:

• Allows multiple processes to read data from the same queue at the same time. It also locks messages during processing to ensure that a message is only processed by one reader, even if multiple processes read messages from a single queue.
• Amazon's massive redundant architecture provides extremely high availability for concurrent access. It also ensures message delivery (at least once ).
• You must pay only based on your usage. For Amazon SQS, this means you pay $0.000001 for each message. AWS currently provides a free tier, with the first 100,000 messages per month beingFree. Remember that bandwidth fees are priced at gigabytes, which is common to all AWS products.

SQS is just as easy as everything else in AWS. If you do not have an AWS account, create one first. Next, enable Amazon SQS. Finally, use the aws api Java SDK to publish and read cloud-based messages! (The following describes in detail how to implementWrite.)

 

Back to Top

Compile SQS messages

Similar to the Amazon SQS name, the logic behind reading and writing to the queue is simple. First, use the valid access key and secret to establish an AWS connection, as shown in Listing 1:

Listing 1. Creating AWS connections

AmazonSQS sqs = new AmazonSQSClient(new BasicAWSCredentials(AWS_KEY, AWS_SECRET));

 

Next, you need a queue. In aws apicreateQueueAs shown in List 2, you do not have to create a new queue each time. If the queue already exists, return its handle. In SQS, the queue is only URLs. Therefore, queue processing is also a URL. Note that in the aws sdk api,QueueThe URL isStringType, not JavaURLType.

Listing 2. Getting a queue handle

String url = sqs.createQueue(new CreateQueueRequest("a_queue")).getQueueUrl();

 

With a queue, you can write a message to it. The message format of SQS is similar to that of SimpleDB (see references ).StringS. But rememberStringIt can be easily structured, so it is easy to parse, even if its format is valid JSON or XML.

Listing 3. Sending messages through SQS

sqs.sendMessage(new SendMessageRequest(url, "It's a wonderful life!"));

 

SQS keep simplicity

Remember that Amazon SQS is simple, which means it lacks some additional items that you may have used to in the past. For example, SQS does not receive advance notifications. Therefore, the SQS Queue Reader must periodically poll to check whether it contains new messages. Although not terrible, it does increase application overhead and may be unacceptable in some cases. Amazon Simple Notification Service (SNS) solves this problem, but this is a topic of another article.

The message length is limited. By default, a message cannot exceed 8 KB. If you want to use long messages, you can always divide them into small pieces and use the sequence IDs to identify a single part. Then, you can re-combine messages on the receiver.

That's simple-only three lines of code are required to put a message in the SQS queue.

About AWS SDK

You may notice a familiar mode in the aws sdk, especially if you have read my SimpleDB Introduction (see references ). Since everything in AWS is a web service, all communication occurs through HTTP. ThereforeRequestSimilar objects simulate logical requests, suchSendMessageRequestOrCreateQueueRequest. In either case, the name describes the object's intent.

In addition, messages placed on SQS are persistent: they are always there before they are deleted. (If you do not delete them, the message will not disappear. The default value of automatic expiration is 4 days .) When obtaining messages for reading, Amazon SQS uses a simple locking policy-for reading events. Messages cannot be used by other concurrent reading processes within a period. This is called the messageVisibility timeout. This value is set to 30 seconds by default, but you can change the duration as needed.

Message persistence in Amazon architecture is reliable. Like SimpleDB or even S3, AWS components are redundant. If your reader process stops unexpectedly during message processing, it is very likely that the message is still in progress. In addition, if some assets in the AWS network are also destroyed, you can be sure that your mission-critical messages will not be lost-they will continue to exist on any number of other machines. Finally, this is common for all other AWS products. You can set the physical location of your message infrastructure by region: US, EU, and so on.

 

Back to Top

Read SQS messages

Three lines of code are required to write a message to an SQS queue. Read a little more messages. In fact, the first two lines are the same, because you need an AWS connection and a handle for the same queue. Amazon SQS does not provide any callback function or pre-notification of message arrival. You must regularly poll an SQS queue to see if it is available. Therefore, these additional code lines are required to read an SQS queue.

Pay attention to the following when implementing the polling policy: You must check and ensure that a valid message is actually received before processing a message. If you do not check, you will eventually see this hatefulNullPointerException.

For example, if I get a valid AWS connection and handle of a queue containing messages, I can retrieve messages as shown in Listing 4:

Listing 4. Retrieving messages through SQS

while (true) { List<Message> msgs = sqs.receiveMessage( new ReceiveMessageRequest(url).withMaxNumberOfMessages(1)).getMessages(); if (msgs.size() > 0) { Message message = msgs.get(0); System.out.println("The message is " + message.getBody()); sqs.deleteMessage(new DeleteMessageRequest(url, message.getReceiptHandle())); } else { System.out.println("nothing found, trying again in 30 seconds"); Thread.sleep(3000); }}

 

In Listing 4sqsIsAmazonSQSClass, as shown in Listing 1. This object providesreceiveMessageMethod, which acceptsReceiveMessageRequest. ConfigurableReceiveMessageRequestS to request a group of messages in the queue. In my example, I configure it to retrieve only one message at a time. No matter how many messages I request,receiveMessageAll methods are returned.MessageTypeList.

Implement a polling Policy

As I mentioned earlier, SQS reads are done through polling.receiveMessageThe method is not blocked. Therefore, I must check to ensure that the correspondingList(msgs. If nothing is retrieved from the queueReceiveMessageRequestUppergetMessagesWill return an emptyList, Rathernull.

If I retrieve a valid message, I can usegetBodyCall to obtain its load or body. Remember that SQS will lock a valid message handle. By default, I have 30 seconds to do something for the message. If I want to permanently remove a message from processing, I must delete it. Therefore, I senddeleteMessageCall, which requiresDeleteMessageRequest.

OneMessageThe instance features itsReceiving handle, Likeid. The handle is not directly related to the message, but is related to reading itsEventRelated. A message that has been read multiple times (for example, if it is not deleted or the read process fails) can have multiple receiving handles. When you want to delete a message, you mustgetReceiptHandleCall to provide the receiving handle.

I didn't constantly check whether my queue has messages. I provided a sleep function. If no message is retrieved, I would wait 30 seconds. Obviously, sleep may not be a good idea in some cases, or a long pause may be appropriate.

With these lines of code, I will introduce Amazon SQS almost. Although AWS sdks provide a lot of other functions and features, the code so far is everything you need to read and write messages to SQS queues.

Now let's take a look at what will happen when we actually use it.

 

Back to Top

Magnus meets Amazon SQS

Last month, I created a simple mobile web application Magnus, which I used to demonstrate some features of Amazon Elastic Beanstalk (see references ). Magnus has a good function to store the location information of mobile devices that receive from the account holder-the information that many people want to provide and that others want to use.

It is good to capture the whereabouts of some people, but what people really like is graphics (and shiny rounded corner buttons ). When you have a large amount of data to move, it is very expensive to view the graphics and analysis from the processing point of view. (Does Hadoop apply to anyone ?) EffectiveExtract, transform, and loadOr ETL technology is a way to manage this. ETL is a very large term that contains a large number of things. (People build a career around this abbreviation, and the company builds a business around it !) In this example, ETL only indicates that I want to analyze some MongoDB data and create a new document based on the data.

ETL and Amazon SQS

When talking about data analysis, there are multiple possibilities for our requirements for data and the answers we can provide. Magnus web applications present a small part of this possibility: It extracts and presents data related to geographic coordinates, time, and user accounts. Technically, Magnus focuses on the relationship between location longitude and latitude, user account ID, timestamp, and specific data.

Magnus can graphically represent the data and present the user account based on the geographical region (possibly a map that uses tags to locate the account holder at a given time ). Or it can show how to move an account holder/user across a given region (another map. Providing such information requires an offline ETL process. From the processing point of view, it is very expensive to provide the data in real time when the data is generated. Therefore, these analyses are consideredNear real-time.

To use Amazon SQS in Magnus, I need to make some preliminary settings. First, I need a way to obtain AWS creden. I like Play (see references), so I want to use it as my application development framework. To obtain creden, I can useapplication.confFile, that is, an attribute file automatically read.

Listing 5. Add AWS configuration data to application. conf of Play

#AWS configurationaws_access_key_id=1S..........MR2aws_secret_access_key=S3.........ZM

 

After defining attributesPlayYou can call an object to obtain them easily, as shown in Listing 6:

Listing 6. Getting AWS information in Play

public class Application extends Controller { private static final String AWS_KEY = Play.configuration.get("aws_access_key_id").toString(); private static final String AWS_SECRET = Play.configuration.get("aws_secret_access_key").toString();//....}

 

After defining the pipeline, I can start to get started. The code in listing 7 is similar to a code segment used in my Amazon Elastic Beanstalk introduction last month. In this example, I only use some code to updatesaveLocationTo put a simple JSON document named"locations_queue. JSON is basically like this:{"id":"4d6baeb52a54f1000001"}. The ID of the saved location is provided to the message recipient for query and analysis.

Listing 7. A saveLocation Method for placing messages on SQS

public static void saveLocation(String id, JsonObject body) throws Exception { String eventname = body.getAsJsonPrimitive("name").getAsString(); double latitude = body.getAsJsonPrimitive("latitude").getAsDouble(); double longitude = body.getAsJsonPrimitive("longitude").getAsDouble(); String when = body.getAsJsonPrimitive("timestamp").getAsString(); SimpleDateFormat formatter = new SimpleDateFormat("dd-MM-yyyy HH:mm"); Date dt = formatter.parse(when); ObjectId oid = new Location(id, dt, latitude, longitude).save(); AmazonSQS sqs = new AmazonSQSClient(new BasicAWSCredentials(AWS_KEY, AWS_SECRET)); Map mp = new HashMap<String, String>(); mp.put("id", oid.toString()); String url = sqs.createQueue(new CreateQueueRequest("locations_queue")).getQueueUrl(); sqs.sendMessage(new SendMessageRequest(url, new Gson().toJson(mp))); renderJSON(getSuccessMessage());}

 

A date with Ruby?

After the message is put on the SQS queue, I need to select them from the queue and perform some processing. I wonder whether you remember that one of the advantages of MOM is that it allows heterogeneous architectures. To this end, SQS readers can be written in languages other than Java and even run on another platform!

Because I can basically execute Analysis and Processing in any language I like, I will use Ruby to execute-to win some street reputation of fashion young people.

In listing 8, I usedright_awsRuby gem helps me process SQS. In many ways, you canGemAs a jar file.right_awsThe library is similar to Amazon SDK for Java, but it is not so lengthy and easy to use.

Listing 8. Using Ruby to create a connection and queue for SQS

require "right_aws"#...sqs = RightAws::SqsGen2.new(aws_access_key_id, aws_secret_access_key)queue = sqs.queue('locations_queue')

 

As you can see, the two lines of code in listing 8 establish an AWS connection and get my'locations_queue'The handle of the queue.

Next, I set up a polling mechanism, as shown in listing 9. Pair@queueReferences are the same in listing 8.queueVariable. However, in this example, it is defined as a part of the class. So in listing 9, I use Ruby's@Syntax directly references an instance variable.

Listing 9. process messages from SQS

def process_messages() while true msg = @queue.pop if !msg.nil? handle_message(msg) # impl of which does neat stuff msg.delete else sleep 10 end endend

 

When you pass a messagehandle_messageYou can delete it. If no message is found, the main thread sleep for 10 seconds.!msg.nil?A line is similarmsg != null. But in Ruby, evennullIt is also an object. Ask whether an object isnilType (throughnil?Method call) returns a Boolean value.

 

Back to Top

Conclusion

Because AWS is a web service product, it is accessed and used by many platform libraries. In Magnus, you can see the flexibility: I can use Java code to push messages to an SQS queue, and then use a small Ruby program to strip them out. The beauty of using the query architecture is the implicit decoupling of components.

Just as it is generally reasonable to host a web application on GAE or Amazon Elastic Beanstalk, It is also reasonable to use the cloud messaging system. Amazon SQS reduces the burden of installing and maintaining the queue system. You only need to create a queue and delete and retrieve messages on it. Leave the remaining work to Amazon.