Use Java APIs to process WebSphere MQ messages

From: http://www.ibm.com/developerworks/cn/websphere/library/techarticles/0902_yangj_mq/

This article describes two methods for processing large messages in WebSphere MQ: Message sharding and message grouping. The sample code with the message demonstrates how to use it.. Net API to Implement Message sharding and message grouping.

How to process large messages in WebSphere MQ

WebSphere has restrictions on the major and minor tasks of processing a single message. Currently, the maximum supported Message Size is 100 mb. As the message size increases, the performance of WebSphere MQ will also decrease, in terms of best practices, WebSphere MQ delivers the most efficient messages of several k sizes. so how can we enable WebSphere MQ to efficiently process big messages?

WebSphere MQ provides two methods to process large messages: Message sharding and message grouping. The following describes how to implement message sharding and grouping when using Java APIs to compile WebSphere MQ programs.

Message sharding

The method of message sharding is to divide a large logical message on the application into a small segment. Each segment is transmitted independently as a WebSphere MQ message, it is identified by the groupid, msgseqnumber, and offset attributes in mqmd. The Offset Value of the Start message is 0, and the last message uses the following tag to identify the last segment: mqmf_last_segment.

Specifically, message sharding can be divided into two modes: one is to automatically implement message sharding and assembly by the queue manager; second, the application implements message sharding and assembly. The following describes the two implementation methods in detail.

Message shards automatically implemented by the queue manager

As the name suggests, the queue manager automatically implements message sharding, which is used by the queue manager to complete message sharding and assembly. For an application, whether it is the sender or receiver, it processes a complete large message, but by setting some identifiers in the program to instruct the queue manager to split the message and then transmit it. Therefore, this method is applicable to scenarios where WebSphere MQ is not suitable for transmitting large messages due to transmission efficiency, and applications can process large messages and occupy large memory. In addition, this method is also relatively simple for writing applications.

In actual Java API programming, the sender needs to set the messageflags of the message when sending the message as follows:

Msg.messageFlags = MQC.MQMF_SEGMENTATION_ALLOWED;

For the receiver, you must set mqgetmessageoptions when receiving the message:

MQGetMessageOptions gmo = new MQGetMessageOptions ();

gmo.options = MQC.MQGMO_COMPLETE_MSG;

The queue manager automatically implements part of the code for message sharding, such as listing 1. You can download the detailed sample code.

Listing 1 The Queue Manager Automatically implements message sharding

QMgrSegSender.java：    int openOptions = MQC.MQOO_OUTPUT | MQC.MQOO_FAIL_IF_QUIESCING;    myQMgr = new MQQueueManager ("QM1");    myQueue = myQMgr.accessQueue("TESTQ", openOptions);    MQMessage myMsg = new MQMessage ();    myMsg.messageFlags = MQC.MQMF_SEGMENTATION_ALLOWED;    MQPutMessageOptions pmo = new MQPutMessageOptions ();    String strMsg = "";    for (int i=0;i<=100;i++)        strMsg = strMsg + "Hello";    myMsg.write(strMsg.getBytes());    myQueue.put(myMsg,pmo);    System.out.println("Put message:\n" + strMsg);    myQueue.close();    myQMgr.disconnect();QMgrSegReceiver.java：    int openOptions = MQC.MQOO_INPUT_SHARED | MQC.MQOO_FAIL_IF_QUIESCING;    myQMgr = new MQQueueManager ("QM1");    myQueue = myQMgr.accessQueue("TESTQ", openOptions);    MQMessage myMsg = new MQMessage ();    MQGetMessageOptions gmo = new MQGetMessageOptions ();    gmo.options = MQC.MQGMO_COMPLETE_MSG;     myQueue.get(myMsg, gmo);    byte[] b = new byte[myMsg.getMessageLength()];    myMsg.readFully(b);    String strMsg = new String(b);    System.out.println("Got message:\n" + strMsg);    myQueue.close();    myQMgr.disconnect();

 

Program functions:

(1) The qmgrsegsender program constructs a 505-byte message and writes it to the queue testq.

To prevent MQ from transmitting messages of 505 bytes, you can modify the attribute "Maximum message length (maxmsgl)" of queue testq to 100.

(2) The qmgrsegreceiver program reads a message from the testq queue.

We observe that the execution result is that it combines 6 segments of messages in the queue into a complete large message to be taken out.

Using the queue manager to automatically implement message sharding is relatively simple for application developers, but it is necessary to ensure that the program can process complete big messages in terms of memory usage.

Message sharding implemented by the application

Message sharding in an application refers to splitting a large message into multiple segments in the sender application. Each segment is written into the queue as an independent message; in the receiver application, each segment is taken out as an independent message. The program assembles all the message segments into a complete message. This mode applies to scenarios where neither WebSphere MQ nor the application process such a large single message.

Generally, in the sender program implementation, we put all the message fragments in one synchronization point for sending, so we need to set mqputmessageoptions to mqpmo_syncpoint. At the same time, we recommend using the option mqpmo_logical_order, this means that the queue manager automatically maintains the offset of each message segment. Otherwise, the application needs to set the offset:

MQPutMessageOptions pmo = new MQPutMessageOptions ();

pmo.options = MQC.MQPMO_LOGICAL_ORDER + MQC.MQPMO_SYNCPOINT;

For each message segment, we should also identify this as a message segment (mqmf_segment ):

myMsg.messageFlags = MQC.MQMF_SEGMENT;

For the last message segment, you also need to set a special identifier (mqmf_last_segment ):

myMsg.messageFlags = MQC.MQMF_LAST_SEGMENT;

Similarly, in the receiver program, we also put all the message fragments in a synchronization point for receiving, so we need to set mqgetmessageoptions to mqgmo_syncpoint; at the same time, we also set mqgmo_logical_order to ensure that all the message fragments are retrieved in a logical order. In addition, we also need to set the option (mqgmo_all_segments_available) for processing after all the message fragments arrive ), this is to prevent the program from waiting infinitely in Case of Loss of message segments due to exceptions:

MQGetMessageOptions gmo = new MQGetMessageOptions ();

gmo.options = MQC.MQGMO_LOGICAL_ORDER + MQC.MQGMO_SYNCPOINT + MQC.MQGMO_ALL_SEGMENTS_AVAILABLE;

Because we retrieve message segments in a logical order, when we set the cycle to retrieve messages, as long as a message segment is the last identifier, we think we have obtained the complete message. If the message segment is not set to be retrieved in the logical order, the application needs to identify whether the complete message has been obtained based on the message serial number, offset, and whether it is the last message segment.

Part of the code for the application to Implement Message sharding is shown in Listing 2. You can download the detailed sample code:

Listing 2 the application implements message sharding

AppSegSender.java    int openOptions = MQC.MQOO_OUTPUT | MQC.MQOO_FAIL_IF_QUIESCING;    myQMgr = new MQQueueManager ("QM1");    myQueue = myQMgr.accessQueue("TESTQ", openOptions);    for(int i=0;i<3;i++)    {        MQMessage myMsg = new MQMessage ();        MQPutMessageOptions pmo = new MQPutMessageOptions ();        pmo.options = MQC.MQPMO_LOGICAL_ORDER + MQC.MQPMO_SYNCPOINT;        if (i<2)            myMsg.messageFlags = MQC.MQMF_SEGMENT;        else            myMsg.messageFlags = MQC.MQMF_LAST_SEGMENT;        String strMsg = "Hello" + i;        myMsg.write(strMsg.getBytes());        myQueue.put(myMsg,pmo);        System.out.println("Put message '" + strMsg + "'! ");    }    myQMgr.commit();    myQueue.close();    myQMgr.disconnect();AppSegReceiver.java    int openOptions = MQC.MQOO_INPUT_SHARED | MQC.MQOO_FAIL_IF_QUIESCING;    myQMgr = new MQQueueManager ("QM1");    myQueue = myQMgr.accessQueue("TESTQ", openOptions);    MQMessage myMsg;    MQGetMessageOptions gmo = new MQGetMessageOptions ();    gmo.options =     MQC.MQGMO_LOGICAL_ORDER + MQC.MQGMO_SYNCPOINT + MQC.MQGMO_ALL_SEGMENTS_AVAILABLE;    String strMsg = "";    boolean isLastSegment = false;    while(!isLastSegment)    {        myMsg = new MQMessage ();        myQueue.get(myMsg, gmo);        if (myMsg.messageFlags == MQC.MQMF_SEGMENT + MQC.MQMF_LAST_SEGMENT)            isLastSegment = true;        byte[] b = new byte[myMsg.getMessageLength()];        myMsg.readFully(b);        strMsg += new String(b);    }    System.out.println("Got message:\n" + strMsg);    myQMgr.commit();    myQueue.close();   myQMgr.disconnect();

Program functions:

1. The appsegsender program uses a for loop to construct three message fragments of a complete message and write them into the testq queue respectively.
2. The appsegreceiver program cyclically reads a message segment from the queue testq, and assembles a complete message based on its logical order and whether it is the last message segment.

Compared with the message sharding method automatically implemented by the queue manager, the message sharding method of the application is slightly complicated, but it can process larger messages.

Message Group

In terms of implementation methods, message groups and message fragments are very similar, but they have completely different business meanings. In message fragment, although each message fragment is transmitted as an independent message, it is of service significance only after all the message fragments are collected to form a complete message, A single message segment has no business significance. From this point on, we thought of splitting big messages because it was technically difficult to process big messages. Message groups are different. each of its member messages is an independent message with business significance. For example, messages in a group have a clear order, and so on, these messages are transmitted as a group.

In actual implementation, messages in the group are identified by the groupid and msgseqnumber attributes in mqmd, And the last message is marked as the last message in the group (mqmf_last_msg_in_group ).

Similar to message sharding, generally in the sender program, we place all messages in the same group in one synchronization point for sending, so we need to set mqputmessageoptions to mqpmo_syncpoint. At the same time, we recommend mqpmo_logical_order, this means that the queue manager automatically maintains the serial number (msgseqnumber) of each message. Otherwise, the application needs to set it itself:

MQPutMessageOptions pmo = new MQPutMessageOptions ();

pmo.options = MQC.MQPMO_LOGICAL_ORDER + MQC.MQPMO_SYNCPOINT;

For each message, we should also identify it as a group message (mqmf_msg_in_group ):

myMsg.messageFlags = MQC.MQMF_MSG_IN_GROUP;

For the last message in the group, you also need to set a special identifier (mqmf_last_msg_in_group ):

myMsg.messageFlags = MQC.MQMF_LAST_MSG_IN_GROUP;

Similarly, in the receiver program, we also put all the messages in the same group in one synchronization point for receiving, so we need to set mqgetmessageoptions to mqgmo_syncpoint; at the same time, we also set mqgmo_logical_order to ensure that all messages in the same group are extracted in a logical order. In addition, we also need to set the option (mqgmo_all_msgs_available) for processing after all the messages in the same group arrive. This is to prevent the program from waiting infinitely in case of a member message loss due to exceptions:

MQGetMessageOptions gmo = new MQGetMessageOptions ();

gmo.options = MQC.MQGMO_LOGICAL_ORDER + MQC.MQGMO_SYNCPOINT + MQC.MQGMO_ALL_MSGS_AVAILABLE;

Because we retrieve the group member messages in a logical order, when we set the cycle to retrieve messages, as long as a message is the last identifier of the group, we think that all the messages in this group have been obtained. If the message segment is not set to be retrieved in the logical order, the application needs to identify whether all messages in the group have been obtained based on the message serial number, number of received messages, and whether the message is the last message in the group.

Part of the Code is shown in listing 3. You can download the detailed sample code.

Listing 3 message groups

AppGrpSender.java    int openOptions = MQC.MQOO_OUTPUT | MQC.MQOO_FAIL_IF_QUIESCING;    myQMgr = new MQQueueManager ("QM1");    myQueue = myQMgr.accessQueue("TESTQ", openOptions);    for(int i=0;i<3;i++)    {        MQMessage myMsg = new MQMessage ();        MQPutMessageOptions pmo = new MQPutMessageOptions ();        pmo.options = MQC.MQPMO_LOGICAL_ORDER + MQC.MQPMO_SYNCPOINT;        if (i<2)            myMsg.messageFlags = MQC.MQMF_MSG_IN_GROUP;        else            myMsg.messageFlags = MQC.MQMF_LAST_MSG_IN_GROUP;        String strMsg = "Hello" + i;        myMsg.write(strMsg.getBytes());        myQueue.put(myMsg,pmo);        System.out.println("Put message" + (i+1) + " '" + strMsg + "'! ");    }    myQMgr.commit();    myQueue.close();    myQMgr.disconnect();AppGrpReceiver.java    int openOptions = MQC.MQOO_INPUT_SHARED | MQC.MQOO_FAIL_IF_QUIESCING;    myQMgr = new MQQueueManager ("QM1");    myQueue = myQMgr.accessQueue("TESTQ", openOptions);    MQMessage myMsg;    MQGetMessageOptions gmo = new MQGetMessageOptions ();    gmo.options =     MQC.MQGMO_LOGICAL_ORDER + MQC.MQGMO_SYNCPOINT + MQC.MQGMO_ALL_MSGS_AVAILABLE;    String strMsg = "";    boolean isLastMsg = false;    int seq = 0;    while(!isLastMsg)    {        seq++;        myMsg = new MQMessage ();        myQueue.get(myMsg, gmo);        if (myMsg.messageFlags == MQC.MQMF_MSG_IN_GROUP + MQC.MQMF_LAST_MSG_IN_GROUP)            isLastMsg = true;        byte[] b = new byte[myMsg.getMessageLength()];        myMsg.readFully(b);        strMsg = new String(b);        System.out.println("Got message" + seq + ":\n" + strMsg);    }    myQMgr.commit();    myQueue.close();    myQMgr.disconnect();

Program functions:

1. The appgrpsender program uses a for loop to construct three messages in a group and write them into the queue testq respectively.
2. The appgrpreceiver program cyclically reads messages from the testq queue, and determines whether all messages in the same group have been obtained based on the logical order and whether the last message is in the group.

Compared with message sharding, message groups are not only a method for processing large messages, but also maintain the logical relationship between a group of business data.

Conclusion

Message sharding and message grouping are common methods used to process large messages in WebSphere MQ programming. which method is suitable depends on actual needs. If a large message needs to be divided into a batch of small messages with actual business significance, it is more appropriate to use message groups. Otherwise, if a large message cannot be divided into small messages with actual business significance, message sharding is used. Even in some complex scenarios, message shards and message groups can be used together. For example, a batch of messages are merged into a group due to the sequential requirements during transmission, at the same time, due to the large size of some messages and the need for multipart transmission, interested readers can implement this complicated scenario on their own.