Implementation of millions real-time message push service in Worktile

In the use of Team collaboration tool Worktile, you will notice whether the message is in the upper-right corner, drag the task in the Task panel, and the user's online status is refreshed in real time. Worktile in the push service is based on the XMPP protocol, Erlang language implementation of the Ejabberd, and on its source code based on the combination of our business, the source code has been modified to fit our own needs. In addition, based on the AMQP protocol can also be used as a real-time message to push the choice, kick-kick network is to use the RABBITMQ+STOMP protocol to implement the message push service. This article will combine my project practice in worktile and kicking net, introduce the concrete realization of the next message push service.

Several realization ways of

real-time push

Compared to the mobile phone-side message push (generally is implemented as a socket), the Web side is based on HTTP protocol, it is difficult to maintain a long connection like TCP. However, with the development of technology, the emergence of new technologies such as Websocket,comet can achieve similar long connection effect, these technologies can be divided into the following categories:

1 short polling. Page end through JS timed asynchronous refresh, this way the real-time effect is poor.

2 Long polling. Page end through JS asynchronous request server, the server after receiving the request, if the request has no data, then suspend this request until there is data arrival or time slice (server set) to, then return this request, the client next request. Examples are as follows:

3) WebSocket. Browser through the WebSocket protocol to connect the service side, the browser and server-side Full-duplex communication. Requires both the server and the browser to support the WebSocket protocol.

In the above several ways, the mode 1 is simpler, but the efficiency and the real-time effect are poor. Mode 2 has high requirements for service-side implementation, especially in the case of large concurrency, the pressure on the service side is great. Mode 3 is more efficient, but it is not supported for lower versions of browsers, and the service side also needs to support WebSocket implementations. Worktile web-side real-time message push, using the XMPP Extension protocol XEP-0124 BOSH (http://xmpp.org/extensions/xep-0124.html), the essence is to adopt a way of 2 long polling. The kick-kick network is implemented using the WebSocket connection RABBITMQ, and I'll explain how to implement the server Push in both ways.

Runtime Environment Preparation

In the implementation of the server, both Ejabberd and RABBITMQ are based on the Erlang language, so the Erlang runtime environment must be installed. Erlang is a functional language with fault-tolerant and high concurrency features, and it is easy to build a robust distributed system with the help of the OTP function library. At present, based on Erlang development products have, database aspects: Riak (Dynamo implementation), CouchDB, Webserver aspects: Cowboy, Mochiweb, message middleware has RABBITMQ. For server-side programmers, Erlang offers high concurrency, fault-tolerant, hot-deployment features that other languages cannot achieve. Whether in real-time communication or in the game program, Erlang can easily create a corresponding process for each online user, for a 4-core 8 G server, carrying millions of such processes is very easy. The following figure is a general schematic diagram of the Erlang program initiation process:

As shown in the figure, session Manager (or Gateway) is responsible for creating the corresponding process for each user (UID) and storing the corresponding relationship (map) in the datasheet. Each process corresponds to the user data, and they can send messages to each other. The advantage of Erlang is that millions of such process is created with enough memory, and its creation and destruction is much lighter than Java thread, which is not an order of magnitude.

OK, now we're going to set up the Erlang environment (the experimental system is Ubuntu 12.04, 4 cores 8 g memory):

1. Dependent Library Installation

sudo apt install build-essentialsudo apt install libncurses5-devsudo apt Install Libssl-dev libyaml-devsudo Apt Install m4sudo apt install unixodbc unixodbc-devsudo apt install freeglut3-dev libwxgtk2.8-devsudo Apt Install xsltprocsudo apt install FOP tk8.5, official website download OTP Source Pack (LIBXML2-UTILS2) , decompression and Installation:

tar zxvf otpsrcR16B01.tar.gzcd otpsrcr16b01configuremake & make install

The Erlang runtime environment is now complete. The RABBITMQ and Ejabberd build real-time messaging services are described below.

Real-time messaging service based on RABBITMQ

RABBITMQ is a widely used message-oriented middleware in the industry, and it is also the best middleware for AMQP protocol implementation. The AMQP protocol defines entities such as producer, Consumer, MessageQueue, Exchange, Binding, Virtual host, and their relationships are shown in the following illustration:

The message publisher (Producer) connection exchanger (Exchange), the exchanger, and Message Queuing are binding,binding by key depending on the type of exchange (divided into Fanout, Direct, Topic, Header) separately to the message for different forms of distribution. Message queue is divided into durable, temporary, auto-delete three types, durable queue is a persistent queue, will not disappear because of service shutdown, temporary queue will disappear after service restart. Auto-delete is automatically deleted when there is no consumer connection. In addition RABBITMQ has many Third-party plug-ins, can be based on the AMQP protocol based on a number of extended applications. Here we will introduce the Web Stomp plug-in to build a stomp text protocol based on AMQP, which achieves real-time message transmission through the browser websocket. The structure of the system is as follows:

As shown in the picture, we use the Stomp.js and sockjs.js with RABBITMQ Web STOMP plugin communication, the phone can use STOMPJ, Gozirra (Android) or Objc-stomp (IOS) Send and receive messages via Stomp protocol and RABBITMQ. Because we are real-time messaging systems are usually to be combined with existing user systems, RABBITMQ can be used through Third-party plug-ins rabbitmq-ayth-backend-http to adapt to the existing user system, this plug-in can be completed through the HTTP interface user connection authentication process. Of course, there are other ways of authentication, such as LDAP. The following are the steps:

Download the latest version of the source package from the official website (http://rabbitmq.com/download.html) to extract and install:

tar zxf rabbitmq-server-x.x.x.tar.gzcd rabbitmq-server-x.x.xmake & make install install RABBITMQ plug-ins for Web-stomp

Cd/path/to/your/rabbitmq./sbin/rabbitmq-plugins Enable Rabbitmq_web_stomp./sbin/rabbitmq-plugins Enable Rabbitmq_ Web_stomp_examples./sbin/rabbitmqctl stop./sbin/rabbitmqctl start./sbin/rabbitmqctl Status

The list of running Plug-ins shown in the following illustration will be displayed

Install User Licensing Plug-ins

cd/path/to/your/rabbitmq/pluginswget <a href= "http://www.rabbitmq.com/community-plugins/v3.3.x/rabbitmq_ Auth_backend_http-3.3.x-e7ac6289.ez ">http://www.rabbitmq.com/community-plugins/v3.3.x/rabbitmq_auth_backend _HTTP-3.3.X-E7AC6289.EZ</A>CD/sbin/rabbitmq-plugins Enable Rabbitmq_auth_backend_ HTTP edit rabbitmq.config file (default deposit under/ect/rabbitmq/), add:

[ ... {rabbit, [{auth_backends, [rabbit_auth_backend_http]}]}, ... {rabbitmq_auth_backend_http, [{user_path, ' http://your-server/auth/user '}, {vhost_path, ' http://your-server/auth/ Vhost "}, {Resource_path," Http://your-server/auth/resource "}]} ...].

In which, User_path is based on the user name password to verify that the Vhost_path is verified to have access to Vhost, Resource_path is to verify the user's incoming Exchange, queue has permissions. My following code is an example of these three interfaces implemented with Node.js:

var express = require (' Express '); var app = Express (); App.get ('/auth/user ', function (req, res) {var name = Req.query.username; var pass = Req.query.password; Console.log ("Name : "+ name +", pass: "+ Pass"; if (name = = ' Guest ' && pass = ' guest ') {Console.log ("Allow"); Res.send ("Allow"); else{res.send (' Deny '); App.get ('/auth/vhost ', function (req, res) {Console.log ("/auth/vhost"); Res.send ("Allow"); App.get ('/auth/resource ', function (req, res) {Console.log ("/auth/resource"); Res.send ("Allow"); App.listen (3000); browser-side JS implementation, sample code is as follows:

... var ws = new Sockjs (' http://' + window.location.hostname + ': 15674/stomp '); var client = Stomp.over (WS); SOCKJS does not support heart-beat:disable heart-beats = 0; client.heartbeat.incoming = 0; Client.debug = pipe (' #second '); var print_first = pipe (' #first ', function (data) {client.send ('/exchange/feed/user_x ', {' content-type ': ' FileType '}, data); }); var on_connect = function (x) {id = client.subscribe ("/exchange/feed/user_x", function (d) {Print_first (d.body);}); var on_error = function () {console.log (' error '); Client.connect (' Guest1 ', ' guest1 ', On_connect, On_error, '/'); ......

To explain, here we first create the feed this exchange in the RABBITMQ instance, and after we successfully connect with stomp.js, we bind to this exchange based on the ID (user_x) of the current login user, that is, subscribe ("/ Exchange/feed/user_x ", ...) The behavior of this operation, which sends a message to feed Exchange in RABBITMQ and specifies the user ID (user_x) as key, and the page end receives the message in real time through the web socket.

So far, based on the Rabbitmq+stomp implementation of the Web-side message push has been completed, many of the details of the small partners need to practice, there is no more to say. You can refer to the official documentation in the process:

Http://rabbitmq.com/stomp.html
Http://rabbitmq.com/web-stomp.html
Https://github.com/simonmacmullen/rabbitmq-auth-backend-http

The above implementation is the way that I use in kicking and kicking the net, the following describes how to implement message push through Ejabberd in Worktile now.

Real-time message push

based on Ejabberd

Unlike RABBITMQ, Ejabberd is an implementation of XMPP protocol, and XMPP is widely used in the field of instant communication compared with AMQP. There are many kinds of implementations of XMPP protocol, such as Java OpenFire, but compared with other implementations, the concurrency performance of Ejabberd is undoubtedly the best. The predecessor of the XMPP protocol was the Jabber protocol, where the early Jabber protocol consisted mainly of online status (Presence), Buddy Roster (roster), and IQ (Info/query). Now Jabber has become an official RFC standard, such as rfc2799,rfc4622,rfc6121, and XMPP Extension Protocol (XEP). The Worktile web-side message alerting function is based on the Bosh extension protocol defined by XEP-0124, XEP-0206.

Because of its own business needs, we have Ejabberd user authentication and Buddy list module of the source code to modify, through the Redis to save the user's online status, rather than Mnesia and MySQL. Another friend this piece we are getting members of a project or team from an existing database (MongoDB). The Web end is connected through Strophe.js (http-bind), strophe.js can be connected in long polling and websocket two ways, because Ejabberd does not have a good websocket implementation, it uses a Bosh way to simulate the long connection. The structure of the entire system is as follows:

The web end uses Strophe.js to connect Nginx proxy through Http-bind, Nginx reverse proxy ejabberdcluster. iOS is connected by Xmpp-framwork, and Android can be smack directly ejabberd server clusters. These are existing libraries that need no client development. The online status defines the online, offline, and busy status in Redis according to the user UID as key. The buddy list is obtained from the MONGODB project table. User authentication directly modified the Ejabberd_auth_internal.erl file, through the MongoDB Drive connection user library, online status and other functions are added modules, the part of the code is as follows:

-module (wt_mod_proj). -behaviour (Gen_mod). -behaviour (Gen_server). -include ("Ejabberd.hrl"). -include ("Logger.hrl"). -include ("Jlib.hrl"). -define (supervisor, Ejabberd_sup). ...-define (ONLINE, 1). -define (OFFLINE, 0). -define (BUSY, 2). -define (LEAVE, 3). ..% Api-export ([START_LINK/2, GET_PROJ_ONLINE_USERS/2]). % Gen_mod Callbacks-export ([START/2, STOP/1]). % gen_server Callbacks-export ([Init/1, TERMINATE/2, HANDLE_CALL/3, HANDLE_CAST/2, HANDLE_INFO/2, CODE_CHANGE/3]). % percent Hook Callbacks-export ([User_available/1, UNSET_PRESENCE/3, SET_PRESENCE/4]). -export ([Get_redis/1, REMOVE_ONLINE_USER/3, APPEND_ONLINE_USER/3]). ...-record (state,{host = << "" >>, Server_host, Rconn, Mconn}). Start_link (host, Opts)-> Proc = Gen_mod:get _module_proc (Host,?) MODULE), Gen_server:start_link ({local, Proc},? MODULE, [Host, Opts], []. User_available (New)-> luser = new#jid.luser, lserver = new#jid.lserver, Proc = Gen_mod:get_module_proc (LServer,? MODULE), Gen_server:cast (Proc, {user_Available, Luser, lserver}). Append_online_user (Uid, Proj, host)-> Proc = Gen_mod:get_module_proc (host,? MODULE), Gen_server:call (Proc, {append_online_user, Uid, Proj}). Remove_online_user (Uid, Proj, host)-> Proc = Gen_mod:get_module_proc (host,? MODULE), Gen_server:call (Proc, {remove_online_user, Uid, Proj}). ... set_presence (User, Server, Resource, Packet)-> Proc = Gen_mod:get_module_proc (server,? MODULE), Gen_server:cast (Proc, {set_presence, User, Server, Resource, Packet}). ... start (host, Opts)-> Proc = Gen_mod:get_module_proc (host,? MODULE), Childspec = {Proc, {? MODULE, Start_link, [Host, Opts]}, transient, Watts, worker, [? MODULE]}, Supervisor:start_child (? Supervisor, Childspec). Stop (host)-> Proc = Gen_mod:get_module_proc (host,? MODULE), Gen_server:call (Proc, stop), Supervisor:delete_child (?) Supervisor, Proc). Init ([Host, Opts])-> MyHost = Gen_mod:get_opt_host (host, Opts, << "Wtmuc. @HOST @" >>), Redishost = gen_mod:get_opt (Redis_host, Opts, Fun (B); B end,? Redis_host), Redisport = Gen_mod:get_opt (Redis_port, Opts, Fun (i) when Is_integer (i), i>0-> i-end,? Redis_port), Ejabberd_hooks:add (Set_presence_hook, Host,?) MODULE, Set_presence, Ejabberd_hooks:add, User_available_hook, Host,? MODULE, User_available, Ejabberd_hooks:add (Sm_remove_connection_hook, Host,?) MODULE, Unset_presence, m, mongohost = Gen_mod:get_opt (Mongo_host, Opts, fun (b)-> binary_to_list (b) end,? Mongo_host), Mongoport = Gen_mod:get_opt (Mongo_port, Opts, Fun (i) when Is_integer (i), i>0-> i-end,? Mongo_port), {OK, MONGO} = Mongo_connection:start_link ({mongohost, mongoport}), C = C (Redishost, redisport), Ejabberd_ Router:register_route (MyHost), {OK, #state {host = host, Server_host = MyHost, Rconn = C, Mconn = Mongo}}. Terminate (_reason, #state {host = host, Rconn = C, mconn = Mongo})-> Ejabberd_hooks:delete (Set_presence_hook, host,?) MODULE, Set_presence, Ejabberd_hooks:delete, User_available_hook, Host,? MODULE, User_available, 50), Ejabberd_hooks:delete (Unset_presence_hook, Host,?) MODULE, Unset_presence, Eredis:stop (C), OK. ... handle_call ({append_online_user, Uid, ProjId}, _from, state)-> C = state#state.rconn, Key = <<!--? Pre_rpoj_online_users/binary, projid/binary-->&gt, Resp = Eredis:q (C, ["Sadd", Key, Uid]), {reply, Resp, state}; Handle_call ({remove_online_user, Uid, ProjId}, _from, state)-> ... handle_call ({get_proj_online_users, ProjId}, _ From, state)-> ... handle_cast ({set_presence, User, Server, Resource, Packet}, #state {mconn = Mongo} = state)-> C = State#state.rconn, Key = <<!--? User_presence/binary, user/binary-->&gt, PIDs = Get_user_projs (User, Mongo), CMD = Get_proj_key (PIDs, ["sunion"]), Case Xml:get_subtag_cdata (Packet, << "Show" >>) of << "Moz" >>-> eredis:q (C, ["SET", Key,?) LEAVE]); << "Offline" >>-> handle_cast (_msg, state)-> {noreply, State}.handle_info ({route, from, to, Packet }, #state {host = host, servEr_host = MyHost, Rconn = redisconn, Mconn = Mongo} = state)-> case Catch Do_route (host, MyHost, from, to, Packet, Redisconn, Mongo) of {' EXIT ', cited}->? Error_msg ("~p", [cited]); _-> OK end, {noreply, State};handle_info (_info, state)-> {noreply, State}.code_change (_OLDVSN, State, _extra)- > {OK, state}. ...

Among them, User\_available\_hook and Sm\_remove\_connection\_hook is the user on-line and the user disconnects the connection to trigger the event, Ejabberd is precisely because these hooks, can easily expand the function.

In the stress test of ejabberd with Tsung, test machine for 4 core 8G memory of ordinary PCs, to 3 clients to simulate user login, set online status, send a text message, turn off the connection operation, at the same time online to reach 30w, the CPU occupies less than 3%, memory about 3 G or so, With the increasing number of users, the main memory loss is larger. Because stress testing is time-consuming, you'll be doing more in-depth testing when you have time.

Free Subscription "CSDN cloud Computing (left) and csdn large data (right)" micro-letter public number, real-time grasp of first-hand cloud news, to understand the latest big data progress!

CSDN publishes related cloud computing information, such as virtualization, Docker, OpenStack, Cloudstack, and data centers, sharing Hadoop, Spark, Nosql/newsql, HBase, Impala, memory calculations, stream computing, Machine learning and intelligent algorithms and other related large data views, providing cloud computing and large data technology, platform, practice and industry information services.