Distributed architecture analysis Maven Springmvc mybatis shiro Druid Restful, Dubbo ZooKeeper, Redis, FastDFS, ActiveMQ, mavenspringmvc

Distributed architecture analysis Maven Springmvc mybatis shiro Druid Restful, Dubbo ZooKeeper, Redis, FastDFS, ActiveMQ, mavenspringmvc

Statement: this framework is oriented to enterprises and is a large-scale Internet distributed enterprise architecture. We will introduce the deployment of high-availability cluster projects on linux later.

Anyone who is willing to learn about the framework technology or source code can directly add Q (2137028325) to study together.

For more information about distributed architecture documentation and related source code, see my blog list.

Http://2137028325.iteye.com/

I. Preface

I have been engaged in a distributed Asynchronous Communication System for nearly a few months. Today I will organize and blog it.

This is a national communication platform that has high requirements on performance, massive data, fault tolerance, and scalability. Therefore, the centralized architecture cannot be simply used. A summary is as follows:

1. Distributed Data Storage

2. Distributed Request Scheduling

3. distributed deployment of multiple nodes

4. Dual Backup and hot Switch

The core of the system is the network architecture, distributed operators, and communication. The requirements are as follows:

Distributed operators:

1. even distribution of output for any input

2. Controllable number of output results

Communication:

1. High concurrency

2. Multithreading

The distributed operator is sun's hash function. cindy socket communication is used for communication. The network architecture and specific descriptions will be provided in the subsequent blog.

 

Ii. Network Architecture

 

The architecture of the entire system includes four layers. Each layer can be divided by several nodes to distribute data and requests:

1. Interface Server ):

1). provide external access interfaces and accept requests. Considering the extensiveness of HTTP, an http server process is usually built in.

2) monitor the working status of dispatcher servers

3 ). forward the request to one of the optimal dispatcher. The optimum judgment here is based on the working status of each dispatcher server, of course, at this layer, the specific content of the request can be simply round-robin or random algorithms.

2. Dispatcher Server ):

1). accept requests from the interface server

2 ). parse the request, extract the feature parameters (generally something similar to a user account, data under an account is distributed to the same node), and then execute the hash function for this parameter, calculate the App Server where the target data is located and forward the request to the App Server.

3). In fact, the processing in the actual project is more complicated than described above, but the scalability is greatly enhanced.

3. App Server ):

1). The execution of business logic is equivalent to the application server in a centralized system, and there is no such distributed feature. The processed data is the data in your own database, which is irrelevant to other nodes on the network.

2). Divided into multiple logical groups, Server Load balancer in the same group

3 ). considering the Dual Backup, hot switching, and load balancing of databases, the single-read and multi-write policies for multiple databases were used. for reading, monitoring the working status of each database, select an optimal database to provide data; for writing and writing all databases, the transaction of operations must be guaranteed.

4. Database Server ):

1) there is nothing to say about providing data access. For non-transactional databases, it is necessary to provide auxiliary measures at the App Server layer;

5. Communication between nodes

1). asynchronous and concurrent transmission of data (requests, responses, and exceptions) in the network format

 

Iii. distributed algorithms

 

 

 

 

 

The Interface Server and the message delivery Server have different request distribution policies.

Auxiliary Functions and variables:

Public String [] getTargetServerIps (); // ip address of the target server, for example, 193.243.15.45: 8080

Public int [] getTargetServerIds (); // target server ID, which corresponds to the ip address of the above server and can be freely configured

Public boolean isServerWorking (index); // you can specify the status of the target server.

Int currentTargetServerIndex = 0; // the index of the current target server in targetServerIds

The Interface Server uses the round robin algorithm:

Public String getTargetServerIp () {// obtain the target server to which the request is distributed

String [] targetServerIps = getTargetServerIps ();

Int [] targetServerIds = getTargetServerIds ();

Int index = currentTargetServerIndex;

Boolean isWorking = false;

While (! IsWorking ){

Index = targetServerIds. length () % (currentTargetServerIndex + 1 );

IsWorking = isServerWorking (index );

If (! IsWorking & index = currentTargetServerIndex) {// No target server is available

Return "0: 0 ";

}

}

CurrentTargetServerIndex = index;

Return targetServerIps [index];

}

Hash algorithm used by the Dispatcher Server to distribute requests

// Hash algrithm from JDK's String, from jdk's hash algorithm

Public int hash (byte [] bs ){

Int hash = 0;

For (int I = 0; I <bs. length; I ++ ){

Hash = 31 * hash + bs [I];

}

Return hash;

}

 

Public int getTargetServerGroupIndexByHash (String hashParam) throws BtirException {// return the target server group calculated based on hash.

Byte [] hashinfo = hashParam. getBytes ("UTF-8 ");

Int frameCount = 2: // number of segments performed by the last two digits of the hash value, that is, the number of hash results and the number of target server groups

Int step = 100/frameCount;

Int hash = Math. abs (hash (hashParam) % 100 );

For (int I = 0, beg = 0, end = step; I <frameCount; I ++ ){

If (beg <= hash & hash <end)

Return 2 * I;

Beg = end;

End + = step;

}

Return 2 * (frameCount-1); // If set well, it should not come here

}

Public String getTargetServerIpInGriuo (int groupIndex) {// calculate the optimal server in the server group based on the polling Algorithm

String [] targetServerIps = getTargetServerIps (); // intra-group round robin algorithm, Code omitted

Int [] targetServerIds = getTargetServerIds ();

Int index = getTargetServerIndexInGroup (groupIndex); // the code of the round-robin algorithm is omitted.

Return targetServerIps [index];

}

Iv. Communication Node Design Model

 

Communication is the method of request response, which is always the same for the interface server, message distribution server, and application server. Therefore, the three can be described in a consistent model.

There are two parts: client and server. The structure and network communication of the two are described here.

The client constructs and sends a Request. In an asynchronous system, it can construct and send an address pair. RequestBuilder generates a Request and inputs the Request to the Request queue (RequestQueue) in the independent thread requestenders scan the Request queue and call RequestSender to send the Request. different queues and different sender can be constructed for different types of requests. The Request priority policy in the queue can be customized as needed. server receives and processes requests,

The server receives and processes the Request. The RequestAccepter receives the Request and puts it into the independent thread scanning queue of the Request queue. After the RequestHandler is called for processing, the server returns Response. communication between client and server: There are many communication protocols and technologies, such as web service, EJB, and jms. The socket based on java NIO is used here, because of its Asynchronization and high concurrency. the two basic standards for using socket are: 1. the number of threads on the server is controllable. Do not linearly increase with the number of requests. 2. separate processing requests from receiving requests. Otherwise, the throughput and concurrency will be reduced.For more information about distributed architecture documentation and related source code, see my blog list.Http://2137028325.iteye.com/

Statement: this framework is oriented to enterprises and is a large-scale Internet distributed enterprise architecture. We will introduce the deployment of high-availability cluster projects on linux later.

Anyone who is willing to learn about the framework technology or source code can directly add Q (2137028325) to study together.