Build a comet application with mochiweb to accommodate millions of users (1)

Article 1

In this series, I will describe in detail how to use mochiweb to achieve massive connections and demonstrate how to create a Comet application using Mochiweb, each mochiweb connection is registered by a router that sends messages to different users. A running application can handle 1 million of the concurrent connections, and it is important that we know how much memory is needed to make it work normally.

In the first article:

• Build a basic mochiweb comet application that sends messages to customers every 10 seconds.
• Adjust the Linux kernel to handle a large number of TCP connections.
• Build a flood control test tool to open massive connections
• Check the memory size required for each connection.

Subsequent serialization in this series will cover how to build an actual message routing system and the skills to save memory overhead, as well as concurrent connection tests between 0.1 million and 1 million.

I assume that you are familiar with Linux Command Line commands and Erlang.

BuildMochiwebTest the application

Introduction:

1. Install and build Mochiweb
2. Run:/your-mochiweb-path/scripts/new_mochiweb.erl mochiconntest
3. Cd mochiconntest and then edit src/mochiconntest_web.erl

These codes (mochiconntest_web.erl) only accept the connection and use the data block transmission to send the initial welcome message, and send the message to each client every 10 seconds.

1. -Module (mochiconntest_web ).
2. -Export ([start/1, stop/0, loop/2]).
3. % External API
4. Start (Options)->
5. {DocRoot, Options1} = get_option (docroot, Options ),
6. Loop = fun (Req)->
7. ? MODULE: loop (Req, DocRoot)
8. End,
9. % We'll set our maximum to 1 million connections. (default: 2048)
10. Mochiweb_http: Start ([{max, 1000000}, {name ,? Module}, {loop, loop} | options1]).
12. Stop ()->
13. Mochiweb_http: Stop (? Module ).
15. Loop (req, docroot)->
16. "/" ++ Path = Req: Get (PATH ),
17. Case Req: Get (method)
18. Method When method =: = 'get'; method =: = 'head'->
19. Case path
20. "Test/" ++ ID->
21. Response = Req: OK ({"text/html; charset = UTF-8 ",
22. [{"Server", "Mochiweb-Test"}],
23. Chunked }),
24. Response: write_chunk ("Mochiconntest welcomes you! Your Id: "++ Id ++"/n "),
25. % Router: login (list_to_atom (Id), self ()),
26. Feed (Response, Id, 1 );
27. _->
28. Req: not_found ()
29. End;
30. 'Post'->
31. Case Path
32. _->
33. Req: not_found ()
34. End;
35. _->
36. Req: respond ({501, [], []})
37. End.
39. Feed (Response, Path, N)->
40. Receive
41. % {Router_msg, Msg}->
42. % Html = io_lib: Format ("recvd MSG #~ W :'~ S' <br/> ", [N, MSG]),
43. % Response: write_chunk (HTML );
44. After 10000->
45. MSG = io_lib: Format ("chunk ~ W for ID ~ S/N ", [N, path]),
46. Response: write_chunk (MSG)
47. End,
48. Feed (response, path, n + 1 ).
50. % Internal API
51. Get_option (option, options)->
52. {Proplists: get_value (option, options), proplists: delete (option, options )}.

Start yourMochiwebApplications

Make &./start-dev.sh
Mochiweb listens to all interfaces through port 8000 by default. If you are working on the desktop, you can use any Web browser to view http: // localhost: 8000/test/foo.

Below is the test command line code

$ Lynx -- source "http: // localhost: 8000/test/foo"

Mochiconntest welcomes you! Your Id: foo <br/>

Chunk 1 for id foo <br/>

Chunk 2 for id foo <br/>

Chunk 3 for id foo <br/>

^ C

It runs correctly. Next we will make him suffer.

For a large numberTcpConnection AdjustmentLinuxKernel

Before testing a large number of connections, you need to leave some time to adjust the TCP settings of the kernel. Otherwise, your test will fail and you will see a large number of out of socket memory messages (if you prefer, nf_conntrack: table full, dropping packet .)

The following is my sysctl settings. This is only one of the feasible methods, and there may be other methods.

# Adjust the universal gigabit network:

Net. Core. rmem_max = 16777216

Net. Core. wmem_max = 16777216

Net. ipv4.tcp _ rmem = 4096 87380 16777216

Net. ipv4.tcp _ WMEM = 4096 65536 16777216

Net. ipv4.tcp _ syncookies = 1

# Let the kernel give more memory capacity to TCP

# For a large number of socket connections (100 K +) you need

Net. ipv4.tcp _ mem = 50576 64768 98152

Net. Core. netdev_max_backlog = 2500

Net. ipv4.netfilter. ip_conntrack_max = 1048576

Save the above content to/etc/sysctl. conf and run sysctl-P to apply these configurations. You do not need to repeat them. Now your kernel will be able to process more connections, yay.

Create massive connections

There are many ways to implement this function. Tsung is quite strong, and there are many other httpd (AB, httperf, httpload, etc.) that send a large number of spam requests ). None of the tests can perfectly match a comet application. I 've been trying to use Erlang to implement an HTTP client solution, so I wrote a basic test to create a large number of connections. It's just because you can do this, not because you have .. each connection corresponds to a process, which is a significant waste of resources. I use a process to load the URL from a file, and another process to create and receive messages sent from all HTTP connections (and a timer process that prints a report every 10 seconds ), All data received from the server is discarded, but it increases the counter count so that we can track how many HTTP data blocks are delivered.
Floodtest. erl

1. -Module (floodtest ).
2. -Export ([start/2, timer/2, recv/1]).
4. Start (Filename, Wait)->
5. Inets: start (),
6. Spawn (? MODULE, timer, [10000, self ()]),
7. This = self (),
8. Spawn (fun ()-> loadurls (filename, fun (u)-> This! {Loadurl, u} end, wait) end ),
9. Recv ({0, 0 }).
11. Recv (Stats)->
12. {Active, closed, Chunks} = stats,
13. Receive
14. {Stats}-> IO: Format ("stats :~ W/N ", [stats])
15. After 0-> Noop
16. End,
17. Receive
18. {HTTP, {_ ref, stream_start, _ x }}-> Recv ({active + 1, closed, Chunks });
19. {Http, {_ Ref, stream, _ X }}-> recv ({Active, Closed, Chunks + 1 });
20. {Http, {_ Ref, stream_end, _ X }}-> recv ({Active-1, Closed + 1, Chunks });
21. {Http, {_ Ref, {error, Why }}->
22. Io: format ("Closed :~ W/n ", [Why]),
23. Recv ({Active-1, Closed + 1, Chunks });
24. {Loadurl, Url}->
25. HTTP: Request (get, {URL, []}, [], [{sync, false}, {stream, self}, {version, 1.1}, {body_format, binary}]),
26. Recv (Stats)
27. End.
29. Timer (T, WHO)->
30. Receive
31. After T->
32. Who! {Stats}
33. End,
34. Timer (T, WHO ).
36. % Read lines from a file with a specified delay between lines:
37. For_each_line_in_file (name, Proc, mode, accum0)->
38. {OK, device} = file: open (name, mode ),
39. For_each_line (device, Proc, accum0 ).
41. For_each_line (device, Proc, accum)->
42. Case IO: get_line (device, "")
43. EOF-> file: Close (device), accum;
44. Line-> newaccum = proc (line, accum ),
45. For_each_line (device, Proc, newaccum)
46. End.
48. Loadurls (filename, callback, wait)->
49. For_each_line_in_file (filename,
50. Fun (line, list)->
51. Callback (string: strip (line, right, $/n )),
52. Receive
53. After wait->
54. Noop
55. End,
56. List
57. End,
58. [Read], []).

For each connection, we need to request a temporary port, that is, the file descriptor. By default, a maximum of 1024 ports can be opened. To avoid opening too many files, you need to modify the ulimit value of your shell file. You can modify/etc/security/limits. conf, but you need to exit/log on at one time. Now you only need to sudo and modify the current shell file (if you do not want to run as root after modifying ulimit, use su to return to your non-privileged user ):

$ Sudo bash

# Ulimit-n 999999

# Erl

You may also set the temporary port range to be too large, which is limited to 65535.

# Echo "1024 65535">/proc/sys/net/ipv4/ip_local_port_range

Generate a website file for the flood control test program
(For I in 'seq 1 100'; do echo "http: // localhost: 10000/test/$ I"; done)>/tmp/mochi-urls.txt

From the erlang prompt, You can compile and start floodtest. erl:
Erl> c (floodtest ).
Erl> floodtest: start ("/tmp/mochi-urls.txt", 100 ).

This will create 10 new connections per second (that is, one connection per 100 milliseconds)

This will be output in the form of {Active, Closed, Chunks}, where Active is the number of established connections, and Closed is the number terminated for some reason, chunks are the number of data blocks transmitted by mochiweb. Closed should be kept as 0, and Chunks should be greater than Active, because each Active connection will receive multiple data blocks (one per 10 seconds ),

Yes10,000Active connectionsMochiweb beamThe capacity of the resident memory of the process is450 MB-That is, each connection45KB.As expected, the CPU usage is almost none.

 

Conclusion

This is a reasonable first attempt. Each connection 45 KB seems a bit high-using c may make every connection close to 4.5KB (just a guess, if anyone has experience, please comment ). If you use c and erlang to compare and implement this function, consider the amount of code and the encoding time, and I want to increase the memory overhead to be more excitable.

In subsequent serialization, I will overwrite and construct a message route (this can be achieved by removing the comments from lines 25 and 41-43 of mochiconntest_web.erl) in addition, some methods will be discussed to reduce the overall memory overhead. I will also share the test results with kb and 1 Mbit/s connections.