Why should I choose Erlang+go for Server architecture (1)

This is a creation in Article, where the information may have evolved or changed.

original articles, reproduced please specify the Source: Server non-amateur research http://blog.csdn.net/erlib author Sunface

It is estimated that many students will find this confusing, go's name has been thunderclap piercing, but Erlang? This stuff is God horse? Is it a programming language? I've never heard of it before.

Here, let me first introduce the more famous applications developed using Erlang:

One: WhatsApp

How to deal with 450 million users with only 32 technicians? For WhatsApp, which has just been bought by Facebook for $19 billion, the answer is erlang--, a programming language that was born in the 80 's, came to the spotlight at this point.

This app takes the features of Erlang to its fullest, leverages its best VMS, cluster infrastructure, database Mnesia, eliminates a lot of data scale, memory pool and lock issues, and the technology and remediation points mentioned are worth our reference.

Although most of the solutions we have used almost every day. But he is very systematic, and used in the business system, this is a very big leap.

A highly reliable architecture that can serve 450 million of users:

It is important to note that the overall structure of WhatsApp is not disclosed, and this is simply to get different fragments from different sources of information. Rick Reed's lectures mainly shared the number of single server 2 million connections using Erlang, though valuable, but not the entire application architecture

These statistics are some of the current system data, more for data storage, messages, meta-clustering, and new BEAM/OTP patches added.

· 450 million of active users and the fastest company to reach this number in history

· 32 Engineers with an average support of 14 million active users per person

· send and receive 50 billion messages across 7 platforms per day

· Millions of registered users on average per day

· 0 Advertising Overhead

· 8 million investment

· Hundreds of nodes

· 8000+ Core

· Hundreds of TB of memory

· more than 70 million Erlang messages per second

· in 2011, the WhatsApp single server made 1 million TCP sessions with memory and CPU surplus. In 2012, theTCP session developed to 2 million .

·

2013 WHATSAPPF released twriter statement 7 billion message inbound, 11 billion messages to play, that is, processing 18 billion messages per day, the great 2013!

More than 2 million long connection push server:

The size of the WhatsApp data set Mnesia:

Data for production systems:

Number of messages per second:

Development history:

1. WhatsApp server is basically fully implemented with Erlang

· server systems that do back-end message routing use Erlang implementations

· It is worth showing that such a large number of active users are managed with very few servers, and the team agrees that this is largely due to Erlang.

· It's worth noting that Facebook chat was developed in 2009 using Erlang, who abandoned Erlang because it was difficult to recruit good programmers.

2. WhatsApp server starts from Ejabberd first

· Ejabberd is a very well-known open source Jabber server, implemented using Erlang.

· It was originally chosen for its openness, its focus on developers, its ease of start, and the long-term reputation of erlang on large communications systems.

· The next many years have been working on rewriting and modifying Ejabberd, ranging from XMPP to internal development protocols, tuning the code base, and re-designing some core components, making a lot of modifications to the Erlang VMS for high performance.

3. in response to 50 billion messages per day, the focus of work is on the creation of a reliable system, and monetization is a distant thing for us.

4. the health of the system mainly look at the length of the queue, the length of the message queue on each node will be monitored, more than the pre-set threshold will be issued reminders, multiple alerts occur indicates that the system entered the next bottleneck.

5. Send the multimedia message by uploading pictures, audio, video to an HTTP server, and then add the link to the content (if available) along with the Base64 encoded thumbnail.

6. Some code is basically changing daily, usually several times a day, and of course the peak period must be avoided. Erlang is ideal for adding modifications or new features to a product, and hot loading means that modifications can be implemented without restarting, errors can be resolved quickly, and the system becomes more loosely coupled through hot loading, which allows for fast updates to be released.

7. What kind of protocol does WhatsApp use? The WhatsApp server pool uses an SSL Socket, and all messages are queued on the server until the client is reconnected to retrieve the message. The successful retrieval of the message is sent back to the WhatsApp server, which will be retransmitted to the original sender and, once the client successfully receives the message, it will be erased in the server store.

8. What is the internal working mechanism of the WhatsApp registration process? Whasapp relies on the phone IMEI number to establish a username/password, which has recently been modified. WhatsApp now lets the app send a generic request with a 5-digit PIN, and then sends an SMS to the phone number, which means the WhatsApp client is no longer restricted to a single phone. Based on the PIN number, the app will request a unique key from WhatsApp, which will be used as a future password, which also means that the key on the original device will be invalid after registering on the new device.

Results

At the beginning, each server has 200,000 concurrent connections.

The first bottleneck occurs when there are 425,000 connections per server. The system encountered a lot of conflicts and the work stopped. The installation scheduler detects how many useful tasks have been stopped, slept, or turned back. At load time, it begins to experience a sleep lock, and the entire system uses only 35-45% CPU utilization, but the scheduler's CPU utilization is up to 95%.

The first round of repairs made more than 1 million connections.

VM Utilization is 76%,CPU utilization of 73%,beam Simulator utilization of 45%, which is a good thing because the emulator has to be the same as the user.

Generally, CPU utilization is not a good method of evaluation, because the scheduler may use CPU to cause the system to look busy.

The bottleneck was resolved after one months and the number of connections per server reached 2 million.

The beam utilization rate is 80%, which is close to the beginning of FreeBSD paging. CPU utilization is approximately the same, with twice times the number of connections. The scheduler encountered a conflict, but it ran well.

It appears that the test can be paused, and then the Erlang code is analyzed.

Initially, each connection had two Erlang processes, reduced to one.

Do some work with a timer.

Peaking at 2.8 million connections per server

571k pkts/sec, >200k Dist msgs/sec

To do some memory optimizations, VM load drops to 70%.

An attempt was made to increase the number of connections to 3 million, but did not succeed.

· when the system encounters a failure, view the long message queue (the sum of individual message queues or message queues).

· Add Message Queuing statistics for each process to the beam device. Includes the number of messages sent/received and the speed of Send/Receive.

Two: RabbitMq

This is one of the best enterprise Message Queuing systems in the world that everyone has ever heard of.

Three: Web framework

Mochiweb,cowboy, etc.

Four: Telecom-grade applications

Telecommunications companies such as Ericsson

V: Large-scale application of game server domain

Especially in the area of page and hand tours, Erlang is a duck to water, and the Tens running game developed with Erlang is endless.

VI: Database

Couchdb,riak, etc.

VII: Applications in other fields

As far as I know, Erlang is an active figure in the banking, medical, and cloud business areas.

For further information, please continue reading: Why do I choose Erlang+go for Server architecture (2)