Internet collection of Renren's system architecture

Renren's system architecture

Today, I saw a list of open-source software used by Renren on the Internet. The architect of Renren wrote this article. After reading this article, I learned about Renren's architecture.

Data Layer

Using MySQL, and using Tokyo Cabinet (key-value storage engine, Tc) for data redundancy, TC instead of MySQL for storage, such as user information on the search results page, however, Tokyo Cabinet does not have the network processing capability. You need to use Tokyo tyrant to provide network access capability and call the Tokyo cabinet API for persistent storage.

In fact, Tokyo tyrant was developed by the author of Tokyo cabinet. It mainly supports the network interface of memcached transmission protocol. It can be understood as the network connection and Protocol Resolution of Tokyo tyrant, call the Tokyo cabinet API to complete persistent storage.

Server

Web server: nginx, which uses the proxy capability of nginx for cross-IDC request proxy and works with Java Server-resin to solve the problem of weak processing capability of resin network connection.

Java Server: Resin replaces Tomcat as a Java Server

Proxy Server: Squid performs reverse proxy caching for image files

Linux server cluster system: LVS (Linux virtual server) uses layer-4 Server Load balancer to replace many hardware Load Balancing Devices.
Framework

Java Web Framework: struts. Wang Xing used structs to write the internal network at the beginning. Now Renren has gradually abandoned the original framework and developed a Web framework, rumors will be made open-source in the future

Search engine framework: provides services for searchers Based on CE

Network Framework: netty, a Java Network Framework, similar to Apache Mina, but more efficient than Mina, to provide some small services

Server System Monitoring

Ganglia is a cross-platform Scalable Distributed monitoring system in a high-performance computing system, such as a cluster and a grid

Application Layer Cache

Memcached is a memory-only key-value cache system that uses spymemcached as the Java client.

Ice: A Cross-language network communication framework that provides powerful communication capabilities, management tools, and load balancing solutions. Its cross-language capabilities are also a major highlight, based on this framework, we use the appropriate language to provide appropriate services. For example, we use C ++ to develop the cache service and Java to develop some logic services. The framework itself can be very heavy or very light, depending on how you use it :)

Differences between memcached and ice cache services

The cache operation granularity is different. memcached operates on the cache object using binary byte as a whole and requires frequent serialization and deserialization. We use the cache service provided by ice, you can operate on one or more fields of the cache object. For example, you can update only the name of a user object.

 

 

Renren intermediate layer: Question
A system composed of open-source software

Like many large websites, Renren's systems are all built by open-source software. Use nginx for front-end access, resin for containers, memcached for general cache, MySQL for databases, and Linux OS.
In addition to the above, Renren also has a distinctive Middle Layer. The middle layer exists as a service and is located in the middle of MySQL and resin, providing a high-concurrency and low-cost data access layer.
Database pressure

In the above structure system, the performance of the database often becomes a system bottleneck. Renren's network is constantly being reconstructed in the development process. The biggest change is the database. The general steps are "optimize SQL", "Business split", "Vertical Split", and "Horizontal Split". The database optimization details will be referenced here in the future.
A single optimized database can perform 3000 primary key queries per second. To improve the performance, we adopt the middle layer.
Performance objectives

Adding a middle layer can improve the overall performance of the service without increasing the number of servers, and improve the scalability of the system. Here we will briefly list the effects of using the intermediate layer service optimization.
Real-time update data

Personal information data of the user. Currently, the write operation is 500 times/second, And the read operation is 20 thousand times/second. The data is distributed in dozens of data tables. If you use a database for 10 primary key queries, the time required will be considerable. The cache service in the middle layer Stabilized this performance by 99.9% of the request time less than 20 ms.
Determines the relationship between friends and reads 900 times/second. This operation now uses 6 GB memory to store all friends, and returns any two friends within 2 ms.
Associated query: only the friend list has 1300 times/second. If you use associated queries, the database needs to synchronize many useless fields. Now we only need two memory requests and derive many types of sorting.
Massive aggregated access

Aggregated pages are the most visited part In SNS. The home page integrates up to 17 modules, and the relationships between these modules are relatively independent. To quickly aggregate the data together, you need to quickly obtain the data.
Our requirement for the overall technical framework is that the key pages should be executed within Ms.
Session Synchronization

Among many resin servers, how can we share the results of user authentication? In various session sharing mechanisms, our solution is to use the middle layer service for centralized storage.
To be continued

The problem is just the beginning. The next "solution" will analyze the main application scenarios of the Renren intermediate layer. The "practice" section will give an example of a typical intermediate layer service.

Renren intermediate layer: Solution

To improve performance, there is an intermediate layer between databases and pages in the technical structure of Renren. The high performance of the intermediate layer is based on memory instead of disk.
Replace disk with memory

The biggest bottleneck of the database system is disk I/O. A large number of small data requests are not the strength of the disk. Renren's intermediate layer service uses memory instead of hard disk to improve overall performance. With this layer of service, the previous database Association queries are calculated and cached in advance and can be obtained directly when necessary.
The general memcached cache solution is also somewhat inadequate, and the data cannot be changed by itself or partially. Renren chose to implement its own cache.
In the process of self-Implementation of the cache, the management of memory is relatively easy, the communication protocol is more complex, we choose the Open Source ice Communication Framework (http://www.zeroc.com) in this regard) to complete the tedious work, so far it has worked well.
The ice communication framework implements two tasks: communication and positioning. The client locates the desired service address through the icegrid component and sends the request to the intermediate layer service. The intermediate layer service returns the result. The client only needs to know one address to find all services. At the same time, many services can be migrated between different servers at will. In the current Renren network, more than 500 ice-written intermediate layer services are running.
Customized memory data

After using ice to solve the communication and deployment problems, the middle layer service is the core data structure management. In summary, it is flexible change to ensure speed. The following lists some scenarios where the intermediate layer service is used.
Multiple sorting of one copy of data

On Renren's friends page, there are many sorting methods to display the friends list. Each list is obtained from a service sorted by ID and then sorted and cached in the list in each order.
Time-varying data

In many list pages, the "online logo" is displayed, Which is redundant in the cache of each list and regularly refreshed. These business logics that need to be implemented together with the cache are very common in the middle layer of Renren.
Special Type

We use a bit to save the user's activation status. Memory MB can save the status of all int ranges. Query and update are fast.
The following example shows the implementation of the intermediate layer.

Renren intermediate layer: Practice

Previous questions and solutions described the problems that Renren encountered during its development and introduced our solutions to improve performance using the intermediate layer. In today's practice, we will use an example to implement a middle-layer service.
The purpose of this service is to quickly query whether the user is valid. The data will be saved in the memory using bitset. each user has one digit and only saves a positive integer of about 2.1 billion, occupying 256 MB of memory.
Start Encoding

The followingCodeSave them at this location: http://gitorious.org/renren/bitserver.
Interface Definition

The interface is defined as follows:
# Include <ice/builtinsequences. Ice>
Module Renren {
Struct bitsegment {
Int begin;
Int end;
Ice: byteseq data;
};
Interface bitserver {
Bool get (INT offset );
Ice: boolseq gets (ice: intseq offsets );
Bitsegment getsegment (INT begin, int end );
};
};

This bitserver. ICE file is compiled into the server's skeleton file through the slice2cpp command:
Slice2cpp-I/opt/ice-3.3/slice bitserver. Ice
Server

With the Server File generated above, we can implement our own business functions.
Bitserveri. h and bitserveri. cpp only implement a single get interface.
# Ifndef _ bitserveri_h __
# DEFINE _ bitserveri_h __# include <bitserver. h>

# Define size_of_bit 2147483647
# Include <bitset>

Namespace Renren
{

Class bitserveri: virtual public bitserver
{
Public:
Void initialize ();

Virtual bool get (: ice: int,
Const ice: Current &);

Virtual: ice: boolseq gets (const: ice: intseq &,
Const ice: Current &);

Virtual: Renren: bitsegment getsegment (: ice: int,
: Ice: int,
Const ice: Current &);
PRIVATE:
STD: bitset <size_of_bit> bits _;
};

}

# Endif
# Include <bitserveri. h>
# Include <ice/ice. h> int main (INT argc, char ** argv ){
Int status = 0;
Ice: communicatorptr IC;
Try {
Ic = ice: Initialize (argc, argv );
Ice: objectadapterptr adapter = IC-> createobjectadapter ("bitserver ");
Renren: bitserveri * OBJ = new Renren: bitserveri;
OBJ-> initialize ();
Adapter-> Add (OBJ, IC-> stringtoidentity ("bitserver "));
Adapter-> activate ();
IC-> waitforshutdown ();
} Catch (const ice: exception & E ){
STD: cerr <e <STD: Endl;
Status = 1;
} Catch (const STD: exception & E ){
STD: cerr <E. What () <STD: Endl;
Status = 1;
} Catch (...) {
STD: cerr <"unknown exception" <STD: Endl;
Status = 1;
}
If (IC ){
Try {
IC-> destroy ();
} Catch (const ice: exception & E ){
STD: cerr <e <STD: Endl;
Status = 1;
} Catch (const STD: exception & E ){
STD: cerr <E. What () <STD: Endl;
Status = 1;
} Catch (...) {
STD: cerr <"unknown exception" <STD: Endl;
Status = 1;
}
}
Return status;
}

Void
Renren: bitserveri: Initialize (){
For (INT I = 0; I <0 xfffff; I = I + 2 ){
Bits _ [I] = true;
}
}

Bool
Renren: bitserveri: Get (: ice: int offset,
Const ice: Current & current)
{
If (offset <0) return false;
Return bits _ [offset];
}

: Ice: boolseq
Renren: bitserveri: gets (const: ice: intseq & offsets,
Const ice: Current & current)
{
Return: ice: boolseq ();
}

: Renren: bitsegment
Renren: bitserveri: getsegment (: ice: int begin,
: Ice: int end,
Const ice: Current & current)
{
Return: Renren: bitsegment ();
}
Client

We use Java as the client. First, we use the slice2java tool to generate the Java proxy class.
Slice2java-I/opt/ice-3.3/slice bitserver. Ice

Then implement the client code by yourself:
Package Renren; Class bitserveradapter {
Private final string endpoints _;
Private ice. communicator IC _;
Private Renren. bitserverprx PRx _;

Public bitserveradapter (string endpoints ){
This. endpoints _ = endpoints;
}

Public void initialize (){
IC _ = ice. util. initialize ();
PRx _ = Renren. bitserverprxhelper. uncheckedcast (IC _. stringtoproxy (endpoints _));
}

Public Boolean get (int id ){
Return PRx _. Get (ID );
}

Public static void main (string [] ARGs ){
Bitserveradapter adapter = new bitserveradapter (ARGs [0]);
Adapter. initialize ();
Boolean ret = Adapter. Get (integer. valueof (ARGs [1]);
System. Out. println (RET );
System. Exit (0 );
}
}
Performance Testing

After the code is completed, let's test the performance.
Start the server first
Target/bitserver-ice. Config = config

Restart the client
Java-CP/opt/ice-3.3/lib/ice. jar: Target/bitclient. Jar \
Renren. bitserveradapter "bitserver: default-P 10000" 1022

The number of client calls is increased by 50000 times, and a single thread processes 10 thousand times per second on average.

In a multi-threaded environment, a single server can process about 80 thousand requests per second, which exceeds the current needs.