Transactions: Building scalable systems to handle failures to prevent loss of data

This article describes the following:

Stable message Delivery

Transactional and system consistency

Handling Error Queue issues

Message size and timing

This article uses the following techniques:

WCF, MSMQ

Designing a distributed system has always been a challenge. With the increasingly powerful CLR, efficient Visual studio®, and fine-grained controls established in the framework (such as Windows®communication Foundation (WCF)), developers have all the tools they need to build scalable systems. Unfortunately, these are not enough.

When dealing with large distributed systems, my team found that it was far from easy to develop reliable systems that could handle failures without losing data. This is not a tool set failure, but rather the use of these tools in a very special way to achieve an extensible and reliable system.

HTTP and message loss

The first problem the team faces is the loss of the message. When we first started designing our system, we decided to use HTTP to transport WCF based services. As part of the message processing, our services often write data to the database. There is no condemnation, but many systems are designed in this way.

Before the system went into the configuration environment, there had been a very rigorous stress test-the system was running for a week under heavy load. The results of the stress tests indicate that our system has lost information.

In the process of analyzing things, we determine that the main loss is the message that contains the order information-which is not surprising because most of the load in the system is related to the order information. But to our surprise, all of these messages were lost in a 10-minute interval. Since then, the system is functioning again.

After studying the various log files, we found that a critical Windows patch was released this week, and the server in the configuration lab itself restarted after the patch was automatically installed. Given the information contained in this new discovery – the server will reboot from time to again – we are aware that it cannot be overlooked. The system will need to run for several consecutive years and need to withstand multiple server upgrades and reboots.

When a message is processed over HTTP, the service opens a transaction against the database, attempts to write data, and then submits the transaction. Under normal circumstances, this operation will execute successfully. If the message processing server restarts the intermediary transaction, the database detects the transaction timeout and rolls back its changes to keep its status consistent. However, when the server starts again, the raw message data will not appear in memory or on the server's network processing stack – the message is lost.

For order processing systems, this means economic loss. It is difficult for anyone to accept the nightmare scenario in the air traffic control system-"aircraft A and B are in a collision course", the incident was lost.