Build millions comet application with Mochiweb (i)

Original:A million-user Comet application with Mochiweb, part 1

Reference: comet--"Server Push" technology that is based on HTTP long connections and does not need to install plugins in the browser is "Comet"

mochiweb--building an Erlang library of lightweight HTTP servers

In this series, I'll go over how I've discovered how mochiweb supports such a huge network connection, showing you how to build a comet application with Mochiweb, where each mochiweb connection is registered to the router responsible for sending messages to different users. Finally we build a working application that can withstand millions of concurrent connections, and more importantly we will know how much memory is needed for such an application to get it running.

This section reads as follows:

Set up a basic comet application that sends a message to the client every 10 seconds to tune the Linux kernel parameters so that it can handle a large number of TCP connections write a Stress test tool (that is, the c10k test) that can establish a large network connection to check how much memory each connection needs.

The continuation of this series will include how to build a real information routing system, reduce memory usage techniques, and test 100K and 1m concurrent connections.

The basics are you need to know some Linux command line operations and a little bit of erlang knowledge, otherwise you can not understand don't blame me ah, hehe

Write a Mochiweb test program

Summarized as follows:

Install compile Mochiweb run:/your-mochiweb-path/scripts/new_mochiweb.erl mochiconntestcd mochiconntest after editing src/mochiconntest_ Web.erl

This part of the code (MOCHICONNTEST_WEB.ERL) is just a receive connection and sends an initial welcome message to the client every 10 seconds with a block transfer.

Mochiconntest_web.erl

-module (Mochiconntest_web).-export ([Start/1, stop/0, LOOP/2]). Percent of external apistart (Options)->    {docroot, O PTIONS1} = get_option (Docroot, Options),    Loop = Fun (Req)->            &N Bsp      ? Module:loop (Req, Docroot)            end,   % setting maximum number of connections is 1 million, default 2048    m Ochiweb_http:start ([{max, 1000000}, {name,? MODULE}, {loop, loop} | OPTIONS1]).  stop ()->    mochiweb_http:stop (? MODULE).  loop (Req, docroot)->    "/" + + Path = req:get (path),    Casereq:get (method) of         method when method =:= ' GET '; Method =:= ' HEAD '->            casepathof            &NBSP ;   "test/" + + Id->                  Response = Req:ok ({"text/ html Charset=utf-8 ",                                    [{"Server "," Mochiweb-test "}],                            &N Bsp         chunked}),                    RESPONSE:WRI Te_chunk ("Mochiconntest welcomes you! Your ID: "+ + ID + +" \ n "),                    Router:login (list_to_ Atom (ID), self ()),                    feed (Response, ID, 1);  &N Bsp             _->                    REQ : Not_found ()             End;        ' POST '->      & nbsp     casepathof                _->                    Req:not_found ()             E nd;        _->            Req:respond ({501, [], []})   &NBSP ; End.  feed (Response, Path, N)->    receive       %{router_msg, MSG}->       %    html = Io_lib:format ("Recvd msg #~w: ' ~s ' <br/>", [N, MSG]),       %    response:write_chunk (Html);    after10000->        MSG = Io_lib:fo Rmat ("Chunk ~w for ID ~s\n", [N, Path]),        Response:write_chunk (MSG)     End, &nbsp ;   Feeds (Response, Path, n+1).  %% internal apiget_option (option, Options)->    {proplists:get_value (Opt ion, Options), Proplists:delete (Option, Options)}.

Start the Mochiweb app

Make &&./start-dev.sh
The default Mochiweb listens on port 8000 on all NIC interfaces, and if you do this on a desktop system, you can use any browser to access Http://localhost:8000/test/foo for testing.

Here is just the command-line test:

$ Lynx--source "Http://localhost:8000/test/foo" Mochiconntest welcomes you! Your Id:foo<br/>chunk 1 for ID foo<br/> Chunk 2 for ID foo<br/>chunk 3 for ID foo<br/>^c

Yes, it can work. Now, let's make it all right, hehe.

Tuning the Linux kernel parameters so that it can handle a large number of TCP connections

To save time we need to adjust the kernel's TCP setup parameters before doing a lot of concurrent connection tests, or your test will fail and you will see a lot of out of the socket memory information (assuming that the forgery will get, nf_conntrack:table full, Dropping packet. )

The following is the SYSCTL settings I used-your configuration may not be the same, but this is roughly the case:

# general Gigabit Tuning:net.core.rmem_max = 16777216net.core.wmem_max = 16777216 Net.ipv4.tcp_rmem = 4096 87380 16777216 Net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_syncookies = # This gives the kernel + memory for TCP # which Need with many (100k+) open socket connections Net.ipv4.tcp_mem = 50576 64768 98152 net.core.netdev_max_backlog = 250 0 # I was also masquerading the Port comet is on, you might not need this net.ipv4.netfilter.ip_conntrack_max = 1048576

Write these into/etc/sysctl.conf and then run sysctl-p to make them effective. No reboot required, now your kernel can handle a lot of connections, yay.

Establish a large number of connections

There are many ways to use it. Tsung is very good, there are many other better tools such as AB, Httperf, httpload, etc. can generate a lot of useless requests. But none of them is suitable for testing comet applications, and I'm just trying to find an excuse to test the HTTP client for Erlang, so I wrote a basic test program to initiate a large number of connections.
Just because you can but doesn't mean you do it. A connection is really a waste of a process. I use a process to transfer a batch of URL links from a file, another process establishes a connection and receives data (when the process of the timer prints a report every 10 seconds). All data received from the server is discarded, but it increments the count so that we can track how much of the HTTP data blocks are being transmitted.

Floodtest.erl

-module (floodtest).-export ([Start/2, TIMER/2, RECV/1]).  start (Filename, Wait)->    Inets:start ( ),    spawn (? MODULE, timer, [10000, self ()]),    this = self (),    spawn (Fun ()-Loadurls (Filename, Fun (U ). {loadurl, u}end, Wait) end),    recv ({0, 0, 0}). &NBSP;RECV (Stats)->    {Active, Closed, Chunk s} = Stats,    receive        {Stats}, Io:format ("Stats: ~w\n", [Stats])   &NBS P     AFTER0 noop    End,    receive        {http,{_ref, stream_s Tart,_x}},  recv ({active+1, Closed, Chunks});        {http,{_ref, stream,_x}}, &nbsp ;       &NBSP;RECV ({Active, Closed, chunks+1});        {http,{_ref, stream_end,_x}} - recv ({Active-1, closed+1, Chunks});        {http,{_ref, {error,Why}}}->            Io:format ("Closed: ~w\n", [why]),        &NBS P   recv ({Active-1, closed+1, Chunks});        {loadurl, Url}->            Http:request (GET, {URL, []}, [], [{Sync, false}, {stream, self}, {version, 1.1}, {Body_format, b Inary}]),                recv (Stats)     End  timer (T, WHO)     receive    Aftert->        who! {stats}    end,    Timer (T, who).  % Read lines from a file with a specified delay between Lin Es:for_each_line_in_file (name, Proc, mode, ACCUM0)->    {OK, Device} = File:open (name, mode),  &N Bsp For_each_line (device, Proc, ACCUM0).  for_each_line (device, Proc, Accum)->    Case Io:get_line (Devi CE, "") of        EOF  -> filE:close (Device), Accum;        line-newaccum = Proc (line, Accum),      &NBSP ;             For_each_line (Device, Proc, newaccum)     End  loadurls (Filenam E, Callback, Wait)->    for_each_line_in_file (Filename,        Fun (line, List)             Callback (String:strip (line, right, $\n)),          & nbsp receive            afterwait->                NO op            End,            list        End,        [read], []). Each connection we have to use a temporary port, each port is also a file descriptor, by default this is limited to 1024. To avoid too many open files problems, you need to change this limit for your current shell by modifying the/etc/security/limits.conf, but this requires logging off and landing. Currently you only need to modify the current shell with sudo (if you do not want to run in Root, Su back to the non-privileged user after calling Ulimit): Udo Bash # Ulimit-n 999999# Erl

You can also increase the range of ephemeral ports to the maximum:
# echo "1024x768 65535" >/proc/sys/net/ipv4/ip_local_port_range

Generate a URL list file for the stress test program
(For i in ' seq 1 10000 ', do echo "http://localhost:8000/test/$i", done) >/tmp/mochi-urls.txt

Now you can compile the call Floodtest.erl at the Erlang prompt:
Erl> C (floodtest).
Erl> Floodtest:start ("/tmp/mochi-urls.txt", 100).

This will establish 10 connections per second (that is, 100 milliseconds per connection).

It outputs state information as {active, Closed, Chunks}, active represents the number of connections established, Closed represents the number of connections terminated for each cause, and Chunks is the number of blocks of data that are processed in block transfer mode. Closed should be greater than active for 0,chunks because each active connection receives multiple chunks of data (10 seconds one).


The fixed size of the 10,000 active connected mochiweb processes is 450mb-, which is 45KB per connection. CPU usage is as small as expected.

Summarize

The first attempt is understandable. 45KB memory per connection looks a bit high-with libevent to make some adjustments I can make it nearly 4.5KB each connection (just guess who has experience in this area please leave a reply). If you think about Erlang and C in the amount of code and time efficiency, I'd like to spend a bit more memory.


In the follow-up, I'll build a messaging router (we can cancel the comments for 25 lines and 41-43 lines in Mochiconntest_web.erl) and explore ways to reduce memory usage. I will also share the test results when connecting 100k and 1M.

Build millions comet application with Mochiweb (i)