How to get your memory NoSQL database ready for enterprise applications

For every web and mobile application that focuses on the user experience, memory-based NoSQL storage systems such as open source Redis and memcached are becoming de facto standards. However, in recent years, the use of these databases by large enterprises has been slow, due mainly to performance, scalability and availability challenges.

Fortunately, the modern programming language (Ruby, Node.js and Python, as well as development platforms (Rails,sinatra and Django, etc.) have directly created a series of tools and class libraries that take full advantage of the high performance and various types of operation commands that are based on the memory-enabled data storage System (which Redis is particularly visible), enabling a wide range of common Cases.

The use cases for these Open-source software projects include task management, forums, real-time analytics, Twitter clones, geographic search, and cache advanced applications.

However, for each application, the availability, scalability, and performance of the database have a significant impact on the success or failure of the application as a whole.

This article provides an overview of the various knowledge needed to prepare your memory NoSQL database for enterprise applications, and tips and advice on how to overcome the seven greatest challenges when managing these databases in the cloud.

1. Usability

Whatever you do, your dataset should always be available to your application. This is especially important for memory databases, because if the correct policy is not applied, you lose some or all of your dataset when the following conditions occur:

Node failure (this is particularly common in the cloud).

Process reboot (you may need to reboot frequently).

System extensions (hopefully you'll need to get to it).

For condition 1 versus Condition 2, which is discussed later in this article, you must apply two main mechanisms:

Distribution (Replication): You must at least be sure to host a copy of the dataset in another cloud instance, and if you want to make sure that the entire datacenter fails (Amazon WEB Services has at least four occurrences in 2012), you can still protect your data , you'd better keep a copy in a different data center. Unfortunately it is not easy to do this. The following scenario is one of the challenges that you face when distributing:

When you write directly to disk in your application, you will find that your application server writes faster than the speed of distribution, especially if your main node and distribution node are congested with network congestion. Once this problem starts to occur, if your dataset is very large, then your distribution node has a great chance of completely stopping the sync operation.

Automatic failover (auto failover): Why do I need automatic failover? This is because your memory database typically handles 100 times times more requests per second than any other database, so every second of downtime means that your application is piling up more processing latency, resulting in a bad user experience. When implementing your own automated failover mechanism, follow the recommendations listed below:

Make sure that when your primary node fails, the distribution node is able to failover immediately, based on a robust watchdog mechanism that continuously monitors your nodes and automatically shifts to the best possible node in the event of a failure.

This process should be as transparent as possible to your application and ideally you should not need any configuration changes. The most advanced solution is to modify the IP address of the Data storage node in DNS, which ensures that your recovery process takes only a few seconds to complete.

Your automatic failover should be based on the request set (Quorum) and achieve full consistency or eventual consistency. For more information on this point, see below.

2. Consistency during and after network fragmentation

Network fragmentation (network splits) occurs frequently in the cloud and is perhaps the most complex part of any distributed database system in the world. Once a split occurs, your application may only see a portion of your entire memory NoSQL node, and any one of your memory NoSQL nodes can only see part of other memory NoSQL nodes.

Why is that a big problem? If your database has some design flaws in it, then when the network split occurs, you may find that your application writes the data to the wrong node. This means that once the split is restored, the data write requests from your application at this stage will disappear. This is a huge problem for memory NoSQL databases because it generates "write" operations every second far more than any other NoSQL database system.

So what if your memory NoSQL database is properly designed? Unfortunately, you will have to be in two very bad alternatives (actually a kind of ...). ), as follows:

If your memory NoSQL database is exactly the same, you need to understand that in some cases it will not allow you to write any data until the network division is restored.

If your memory NoSQL database is ultimately consistent, your application probably uses a request set vector when it sends a read request, either by returning a value (based on the request set), or by blocking (waiting for the request set).

Please note: Since there is no final consistent memory NoSQL database available in the market, you can only choose the 1th option.