SMS Platform System Based on redis Architecture

I. Preface

Due to work requirements, Python + twisted was used a while ago to re-develop the communication part of China Unicom sgip sms System (see here ). Because the system is upgraded on the original system and the time is limited, the data processing part of the original system is used. This part uses the ms SQL Server database as the background support, and uses the python SQL server package for data access.ProgramYou can call the SGIP module to send the SMS table of the database without interrupting the round-robin process. In addition, the status changes of text message submited, response, report, and archive are all based on database operations. Therefore, the system frequently reads and writes data to the database. As the business grows, the database is overwhelmed by dozens of thousands of transient messages per day, and frequent failures occur. Under such circumstances, it is imperative to upgrade System Data Operations and storage.

The SMS platform system (which is an independent SMS sending and receiving processing program that does not involve specific product business data) has two features: The data architecture is relatively simple, only Mt, Mo, and report (response) are supported. However, MT data has a state transition, that is, to be submitted-> responded-> reported status; second, data has no requirements for transaction processing. The role of the database in the system is to query upstream and downstream SMS records. Therefore, I think it is a good choice to use the popular Memory Database (nosql) to improve the system performance.

Ii. System Architecture

A lot of memory databases are basically open-source products for large websites. I finally chose redis. The reason is that in addition to its strong performance, its list feature is very suitable for uninterrupted polling of Data queues. In addition, its set is also very convenient for saving the status transition during the processing of downstream messages. The following figure shows the system architecture:

The system consists of five parts: the RPC interface program for text message downlink, the SMS sending program, the SMS receiving program, the report forwarding program, and the data pump. SMS downlink RPC interface program is through the XML-RPC to provide external interface, for the business system to send SMS calls, from here a downstream SMS began its life cycle; the sender establishes a TCP/IP link with the gateway and submits downstream messages to the Gateway. The receiver listens to the port and receives upstream messages and status reports from the gateway, at the same time, the upstream SMS is forwarded to the Business System of the SMS platform; the report forwarding program is used to forward the downstream SMS report returned by the gateway to the Business System of the SMS platform; the data pump is used to persistently send processed downstream text messages (received status reports) and user upstream text message data from redis to the MySQL database for historical data query and statistics.

Iii. Data Architecture

When using a memory database, data architecture planning is very important. Because it is not a relational database, our database experience is useless here. In the storage of key-value, a simple SELECT statement in a relational database requires the corresponding data structure to be designed. For details, refer to this tutorial. The following is my data architecture plan for your reference:

Schema_mt = "MT: % s" # Mt key architecture
Schema_submited = "Submit: % (seqno) D: % (TS) D" # key architecture of the ID of the submitted Mt
Schema_submit_fail = "Submit-Failed: % s" # key architecture of the failed submission Mt
Schema_report = "Report: % s" # Kye architecture of the Status Report
Schema_mt_seqno = "MT-seqno: % s" # architecture of the SMS submission sequence number key
Schema_mt_timestamp = "MT-ts: % s" # SMS submission timestamp key architecture

Key_mt_id = "MT-gid" # key of the unique Mt ID

key_pending_list = "list: Pending-% d" # ID list key of the MT to be sent
key_submited_set = "set: submited "# Id set key of the submitted mt
key_resp_succ_set =" set: Response-succ "# Id set key of the received Mt response (successful)
key_resp_fail_set = "set: Response-fail" # The Id set key (failed) for receiving the submission response mt
key_report_list = "list: report # Status Report ID list key
key_finish_list = "list: Finish" # completed ID list key

Schema_mo = "Mo: % s" # Mo key architecture
Key_mo_list = "list: Mo-ID" # list of IDs that receive Mo
Key_mo_id = "mo-gid" # key of the unique MO ID

Iv. Data Processing Process Description

1. When the RPC interface program receives the call from the client, it generates a unique ID (incr key_mt_id) for MT and uses this ID as the key (Schema_mt), Text message data is saved as value, and the ID is saved to the list to be sent key_pending_list;

2. the sender pops up the ID of the SMS to be sent from key_pending_list, and uses this ID as the key (Schema_mt) To obtain text message data. After submission to the gateway, write the ID to key_submited_set. After the gateway returns response asynchronously, the ID is written to key_resp_succ_set or key_resp_fail_set Based on the returned status value. In addition, the sending sequence number schema_mt_seqno and the timestamp schema_mt_timestamp of the MT are recorded. The storage ID is related to the two data, schema_submited, so that the ID can be obtained from the sending sequence number and timestamp;

3. after receiving the status report, the recipient obtains the mt id through the sending serial number and timestamp of the corresponding downstream message, delete this ID from key_resp_succ_set and write it to the mt id list key_report_list;

4. After the status report forwarder pops up the mt id from key_report_list and obtains the SMS data, it forwards the status report to the business system and adds the ID to key_finish_list;

5. The data pump program obtains the SMS data from the mt id popped up in the key_finish_list pop-up, writes the downstream SMS data to MySQL, and deletes all related keys.

V. Postscript

All the above programs use the round-robin method to read the corresponding list, and can be processed when there is data. After the system is upgraded to redis, the efficiency is greatly improved, and the system pressure is almost no longer. Even if the business grows significantly, it is more than enough to cope with it.

Redis, like!

(From: http://blog.meiweier.com/2011/01/06/redis-based-sms-system.html)