Erlang file I/O performance evaluation

Reprinted: http://chaoslawful.iteye.com/blog/310346#bc2251686

 

Evaluation purpose

The purpose of this test is to investigate the specific performance differences between the Erlang file I/O and the C standard I/O.

 

Prerequisites

Erlang has two types of operations: normal file I/O and bare file I/O. The file handle of normal file I/O is a process ID, therefore, you can access the handle from any location of the Erlang node network. The file handle of the raw file I/O is a tuple structure, which is valid only when the file is opened on the same node, cross-process and cross-node access is not allowed. Generally, the I/O speed of bare files is much higher than that of normal files.

In addition, Erlang RTS also has some options to improve the process response speed in the case of large I/O traffic. Here we mainly use the following two options:

• + K true | false-This option is used to enable (true) or disable (false, default) the kernel poll function of Erlang RTS. When the kernel poll is disabled, RTS uses the normal user-state event polling interface select/poll for process and I/O scheduling, causing high scheduling overhead. When the kernel poll is enabled, RTS will use kernel-level event polling interfaces (such as epoll on Linux) for scheduling, with a low overhead, which can improve the response speed when a large number of processes exist.
• + A integer-This option is used to adjust the size of the asynchronous thread pool used by Erlang RTS for asynchronous I/O. Currently, it can be set to an integer between 0 (no asynchronous I/O is used, default value) and 1024. When asynchronous I/O is not used, RTS splits I/O operations on large data blocks into multiple small data blocks, to prevent a single large I/O operation from blocking the message loop of RTS, so that other processes cannot continue to respond. When asynchronous I/O is used, RTS distributes I/O operations to the task queue of a worker thread in the asynchronous thread pool, A worker thread independent of the RTS message loop is used to process time-consuming I/O operations (leader-
  Follower mode ). Both of these methods have certain overhead for processing large I/O operations: When I/O is not used asynchronously, a single I/O operation is changed to multiple I/O operations, this increases the overhead of I/O operation calls by N times (N indicates the number of I/O operations evenly divided). When asynchronous I/O is used, OS needs to perform context switching between the RTS scheduling thread and the asynchronous I/O thread, which is also a relatively time-consuming operation.

Therefore, the default status (+ K false + A 0) And adjusted status (+ K true + A 32.

 

Evaluation environment and use cases

The evaluation environment is the company's notebook HP nx6320, Intel duo t2400 1.83 GHz (2 cores), 1 gb ram, 60 gb sata hd.

Test cases can be divided into two types: sequential read/write and random read/write. The read and write time tests before and after parameter adjustment are performed for each type:

• Sequential read algorithm: reads test data files one by one based on the specified block size until all the files are read;
• Sequential write algorithm: writes random data into the test data file one by one based on the specified block size until the write size meets the requirements;
• Random read algorithm: moves randomly to somewhere in the test data file and reads 1 block based on the specified block size until the accumulative read length meets the requirements;
• Random write algorithm: truncate the test data file to the specified size in advance, and then randomly move it somewhere in the file to write one random data block according to the specified block size, until the accumulative write length meets the requirements, when the measurement write process is complete.

The size of the test data file is 100 MB, and the size of I/o blocks increases from 1kb and 2kb by a factor of 2 to 1024kb to test the effect of different I/O block sizes on performance.

 

Evaluation Result sequence I/O

When the default Erlang parameter is used for read operations, the C standard I/O speed is ~ 5076%, the I/O speed of the Erlang bare file ~ 191%; the Erlang I/O speed is basically unchanged after the preceding options are added. The chart is as follows:

When the Erlang default parameter is used for write operations, the C standard I/O speed is ~ 92%, the I/O speed of the Erlang bare file ~ 194%; after the above options are added, the I/O speed of common files is reduced, and the I/O speed of bare files is improved, in this case, the C standard I/O speed is the I/O speed of Erlang normal files ~ 192%, the I/O speed of the Erlang bare file ~ 124%. The chart is as follows:

 

Random I/O

When the default Erlang parameter is used for read operations, the C standard I/O speed is ~ 2556%, Which is 125% of the I/O speed of the Erlang bare file; the Erlang I/O speed is basically unchanged after the preceding options are added. The chart is as follows:

 
When the Erlang default parameter is used for write operations, the C standard I/O speed is ~ 128%, the I/O speed of the Erlang bare file ~ 62%; the I/O speed of common files is improved after the preceding options are added, but the I/O speed of bare files is reduced, in this case, the C standard I/O speed is the I/O speed of Erlang normal files ~ 125%, the I/O speed of the Erlang bare file ~ 107%. The chart is as follows:

Test code C standard I/O sequential read/write code

See seq_r0000c.zip in the attachment.

 

C Standard I/O random read/write code

See rand_r0000c.zip in the attachment.

 

Erlang sequential read/write code

See seq_rw.erl.zip in the attachment.

 

Erlang random read/write code

See rand_rw.erl.zip in the attachment.