Concept grid database architecture design for Massive Data Processing

For a system with massive data volumes, the performance bottleneck will eventually fall on the database. At this time, hardware upgrades and program optimization are powerless, A simple query may also impose a heavy burden on the database. Grid computing divides a problem that requires a huge amount of computing power into many small parts, and then assigns these small parts to many computers for processing, finally, combine these computing results to get the final result. The Network Computing Architecture Design for databases undoubtedly solves the above performance problems.

So how can we split a huge database into many small databases? At this stage, the most effective method is to partition the database. Can a data table with a huge capacity be divided by date, type, region, or ID number? The answer is yes. This partitioning method is also called the horizontal partitioning method. On the other hand, for different types of data, such as user data, product data, and transaction data in an e-commerce system, they are not closely linked and can be stored in different databases, this enables vertical partitioning.

 

After partitioning, a large database can be divided into many small databases. But in this way, the access to these small databases and how they are comprehensively processed will lead to new problems.

In a system, it is not impossible to access these databases. Using multiple connections and multiple processes is easy to implement in any framework. The problem is that accessing these databases separately in a system may lead to another bottleneck in terms of Program Complexity and processing efficiency, which is not the result we need. Of course, Server Load balancer design can also be used, but the complexity of its program is inevitable.

Here, we will introduce another concept: SOA architecture, that is, service-oriented architecture. SOA can provide a loosely coupled distributed service system by means of service producer/service consumer, or subscription/release. Then, databases in different regions can be made into different service centers based on the SOA architecture to provide database access interfaces. SOA can be implemented in the form of CORBA and Web Service.

In this way, the pressure on database servers is dispersed, and the pressure on Program Computing is also dispersed. No matter how large the database data volume is, how complicated the Program Computing is, the system performance can be maximized.

Finally, you may say that if there is a grid database system, the design of the application system will not be so complicated. Yes, the current database system is also designed in this direction, but the technology is not yet mature. We believe that grid databases will be available in the near future.

Original article title: grid database architecture design concept

Link: http://www.cnblogs.com/chrischen662/archive/2010/09/03/1817081.html

Additional reading

Grid computing has become a hot topic, and the low cost, high performance, and convenient computing resource sharing brought by it is exactly what many enterprises are pursuing. In the future, databases will be built on the grid computing environment.

RACReal Application Cluster (real Application Cluster) is a new technology used in Oracle9i database, and is also the core technology that Oracle Database supports the grid computing environment. Its appearance solves an important problem faced by traditional database applications: the contradiction between high performance, high scalability and low price.

In addition to RAC technology, Oracle9i database also provides other functions to support grid computing, including Transportable Tablespaces that support fast data replication between databases, Oracle Streams that support data flow updates, and One Portable Codebase that supports application portability. Mendelsohn believes that for enterprises that need to build data centers, Oracle9i RAC, coupled with blade servers and Linux operating systems, can completely replace traditional mainframe-based data systems.

To be accurate, it should be a database technology that supports the grid. The g in Oracle10g is the gridding grid.