Linux asynchronous Io

Linux asynchronous I/O is a new enhancement provided by the Linux kernel. It is a standard feature of the 2.6 kernel, but we can also find it in the patch of the 2.4 kernel.The basic idea behind AIO is to allow a process to initiate many I/O operations without blocking or waiting for any operation to complete.The process can retrieve the results of an I/O operation later or when it receives a notification that the I/O operation is complete.

1. I/O model
Before going into the aio api, let's first explore the different I/O models that can be used on Linux. This is not a detailed description, but we will try to introduce some of the most common models to explain the differences between them and asynchronous I/O. Figure 1 shows the synchronous and asynchronous models, as well as the blocking and non-blocking Models.

Figure 1 simple matrix of basic Linux I/O models

Each I/O model has its own usage mode, which has its own advantages for specific applications. This section briefly introduces them one by one.

Note: I/O-intensive processes perform more I/O operations than I/O operations. CPU-intensive processes perform more processing operations than I/O operations. In Linux 2.6, schedulers prefer I/O-intensive processes because they usually initiate an I/O operation and then block the process, this means that other work can be effectively staggered between the two.

(1) Synchronous blocking I/O

The most common model is the synchronous blocking I/O model. In this model, the user space application executes a system call, which causes application blocking. This means that the application will be blocked until the system call is completed (data transmission is completed or an error occurs ). Calling an application is in a state that does not consume the CPU, but simply waits for a response. Therefore, this is very effective from the processing point of view.
Figure 2 shows the traditional blocking I/O model, which is also the most commonly used model in applications. Its behavior is very easy to understand, and its usage is very effective for typical applications. When the read system call is called, the application is blocked and context switches are performed on the kernel. Then, the read operation is triggered. When the response is returned (from the device we are reading from), the data is moved to the user space buffer. Then the application unblocks (read calls return ).

Figure 2 typical process of synchronous blocking I/O model

From the application perspective, the read call will last for a long time. In fact, when the Kernel performs read operations and other work, the application is indeed blocked.
(2) Synchronous non-blocking I/O
A slightly less efficient variant of synchronous blocking I/O is synchronous non-blocking I/O. In this model, devices are opened in a non-blocking manner. This means that the I/O operation will not be completed immediately, and the read operation may return an error code indicating that this command cannot be met immediately (eagain or ewouldblock), as shown in 3.

Figure 3 typical process of synchronizing non-blocking I/O models

The non-blocking implementation is that the I/O command may not be immediately satisfied, and the application needs to call it many times to wait for the Operation to complete.This may be inefficient, because in many cases, when the kernel executes this command, the application must wait until the data is available or try to execute other work.As shown in positive 3, this method can introduce the latency of I/O operations, because there is a certain interval between the data becomes available in the kernel and the user calls the read returned data, this will reduce the overall data throughput.
(3) asynchronous blocking I/O
Another blocking solution is non-blocking I/O with blocking notifications. In this model, the non-blocking I/O is configured, and then the Select system call is blocked to determine when an I/O descriptor is operated. What makes the Select call very interesting is that it can be used to provide notifications for multiple descriptors, not just for one descriptor. For each prompt, we can request that this descriptor can write data, read data is available, and whether an error occurs.

Figure 4 typical process of asynchronous blocking I/O model (select)

The main problem with select calling is that it is not very efficient. Although this is a convenient model for asynchronous notifications, it is not recommended for high-performance I/O operations.
(4) asynchronous non-blocking I/O (AIO)
Finally, the asynchronous non-blocking I/O model is a model that processes overlapping I/O. The read request is returned immediately, indicating that the read request has been initiated successfully. When the read operation is completed in the background, the application then performs other processing operations. When the read response arrives, a signal is generated or a thread-based callback function is executed to complete the I/O processing.

Figure 5 typical asynchronous non-blocking I/O model process

The overlapping processing capabilities of computing operations and I/O processing to execute multiple I/O requests in a process utilizes the difference between processing speed and I/O speed. When one or more I/O requests are suspended, the CPU can execute other tasks. or, more commonly, when other I/O operations are initiated, the completed I/O operations are performed.

From the classification of the previous I/O model, we can see the motivation of AIO. This blocking model needs to block the application at the beginning of the I/O operation. This means that it is impossible to overlap processing and I/O operations at the same time. Synchronous non-blocking model allows overlapping processing and I/O operations, but this requires the application to check the status of I/O operations according to the reproducible rules. In this way, asynchronous non-blocking I/O operations are left, which allow overlapping processing and I/O operations, including notifications of completed I/O operations.
In addition to blocking, the functions provided by the Select function (asynchronously blocking I/O) are similar to those provided by AIO. However, it blocks notification events rather than I/O calls.

2. Introduction to AIO on Linux
In the traditional I/O model, there is an I/O channel identified by a unique handle. In UNIX, these handles are file descriptors (equivalent to files, pipelines, sockets, etc ). In blocking I/O, we initiate a transmission operation. When the transmission operation is completed or an error occurs, the system will return the result after calling the operation.

In asynchronous non-blocking I/O, we can initiate multiple transmission operations at the same time. This requires that each transmission operation has a unique context so that we can determine which transmission operation is completed in their time zone. In AIO, This IsAiocb Structure(AIO control block ). This structure contains all information about the transmission, including the user buffer prepared for the data. When an I/O (called completion) Notification is generated, the aiocb structure is used to uniquely identify the completed I/O operation. This API shows how to use it.

AIO first appeared in the kernel of Version 2.5. It is now a standard feature of the kernel of version 2.6. The API of the AIO interface is very simple, but it provides the necessary functions for data transmission and provides two different notification models. Table 1 provides the AIO interface functions.

Table 1. AIO API

API functions	Description
aio_read	Request asynchronous read Operations
aio_error	Check the status of asynchronous requests
aio_return	Returns the status of the completed asynchronous request.
aio_write	Request asynchronous write operation
aio_suspend	Suspends the calling process until one or more asynchronous requests have been completed (or failed)
aio_cancel	Cancel asynchronous I/O requests
lio_listio	Initiate a series of I/O operations

Each API function starts or checks with the aiocb structure. This structure has many elements, but Listing 1 only provides the elements that need to be (or can be) used.

Listing 1. fields related to the aiocb Structure

struct aiocb {  int aio_fildes;               // File Descriptor  int aio_lio_opcode;           // Valid only for lio_listio (r/w/nop)  volatile void *aio_buf;       // Data Buffer  size_t aio_nbytes;            // Number of Bytes in Data Buffer  struct sigevent aio_sigevent; // Notification Structure  /* Internal fields */  ...};

The sigevent structure tells AIO what operations should be performed when I/O operations are completed. We will explore this structure in the AIO presentation. Now we will show how the various aio api functions work and how we should use them.
(1) aio_read
The aio_read function requests an asynchronous read operation on a valid file descriptor. This file descriptor can represent a file, socket, or even a pipeline. The aio_read function is prototype as follows:
Int aio_read (struct aiocb * aiocbp );
The aio_read function returns immediately after the request is queued. If the execution is successful, the returned value is 0. If an error occurs, the returned value is-1 and the errno value is set.
To perform read operations, the application must initialize the aiocb structure. The following example shows how to fill in the aiocb request structure and use aio_read to perform asynchronous read requests (currently ignore notifications. It also shows the usage of aio_error.
Listing 2. Examples of asynchronous read operations using aio_read

#include <aio.h>...  int fd, ret;  struct aiocb my_aiocb;  fd = open( "file.txt", O_RDONLY );  if (fd < 0) perror("open");  /* Zero out the aiocb structure (recommended) */  bzero( (char *)&my_aiocb, sizeof(struct aiocb) );  /* Allocate a data buffer for the aiocb request */  my_aiocb.aio_buf = malloc(BUFSIZE+1);  if (!my_aiocb.aio_buf) perror("malloc");  /* Initialize the necessary fields in the aiocb */  my_aiocb.aio_fildes = fd;  my_aiocb.aio_nbytes = BUFSIZE;  my_aiocb.aio_offset = 0;  ret = aio_read( &my_aiocb );  if (ret < 0) perror("aio_read");  while ( aio_error( &my_aiocb ) == EINPROGRESS ) ;  if ((ret = aio_return( &my_iocb )) > 0) {    /* got ret bytes on the read */  } else {    /* read failed, consult errno */  }

In Listing 2, after opening the file from which data is to be read, we cleared the aiocb structure and allocated a data buffer. And put the reference to this data buffer in aio_buf. Then, we initialize aio_nbytes to the buffer size. Set aio_offset to 0 (the first offset in the file ). We set aio_fildes to the file descriptor from which the data is read. After these domains are set, the aio_read request is called for read operations. We can then call aio_error to determine the status of aio_read. As long as the status is einprogress, you are always waiting until the status changes. The request may be successful or fail.

Note that using this API is very similar to reading content from files using standard library functions. In addition to some asynchronous features of aio_read, another difference is the setting of the read operation offset. In traditional read calls, the offset is maintained in the context of the file descriptor. For each read operation, the offset must be updated so that subsequent read operations can address the next data block. This is impossible for asynchronous I/O operations, because we can execute many read requests at the same time, so we must specify an offset for each specific Read Request.

We can find the function prototype and other required symbols in the header file AIO. h. When compiling programs that use this interface, we must use the POSIX real-time extension Library (librt ).

(2) aio_error
The aio_error function is used to determine the Request status. The prototype is as follows:
Int aio_error (struct aiocb * aiocbp );
This function returns the following content:
* Einprogress indicates that the request has not been completed
* Ecancelled indicates that the request has been canceled by the application.
*-1 indicates that an error has occurred. For details about the error cause, refer to errno.
(3) aio_return
Another difference between asynchronous I/O and standard block I/o is that we cannot immediately access the returned status of this function because we are not blocking the read call. In standard read calls, the return status is provided when the function returns. However, in asynchronous I/O, we need to use the aio_return function. The prototype of this function is as follows:
Ssize_t aio_return (struct aiocb * aiocbp );
This function is called only after the aio_error call determines that the request has been completed (either successful or an error may occur. The returned value of aio_return is equivalent to the returned value of the read or write System Call in synchronization (the number of transmitted bytes. If an error occurs, the returned value is-1 ).
(4) aio_write
The aio_write function is used to request an asynchronous write operation. The function prototype is as follows:
Int aio_write (struct aiocb * aiocbp );
The aio_write function returns immediately, indicating that the request has been queued (the return value is 0 when the request is successful, the return value is-1 when the request fails, and errno is set accordingly ).
This is similar to the read system call, but there is a difference in behavior that requires attention. Recall that the offset to be used is very important for read calls. However, for write, this offset is only important in the file context where the o_append option is not set. If o_append is set, the offset will be ignored and the data will be appended to the end of the file. Otherwise, the aio_offset field determines the offset of the data in the file to be written.
(5) aio_suspend
We can use the aio_suspend function to suspend (or block) The Calling process until the asynchronous request is complete. At this time, a signal or other timeout operations will occur. The caller provides an aiocb reference list. Any completion of the List results in aio_suspend return. The aio_suspend function prototype is as follows:
Int aio_suspend (const struct aiocb * const cblist [], int N, const struct timespec * timeout );
The use of aio_suspend is very simple. We want to provide an aiocb reference list. If any one is complete, the call returns 0. Otherwise,-1 is returned, indicating that an error has occurred. See listing 3.

Listing 3. Using the aio_suspend function to block asynchronous I/O

struct aioct *cblist[MAX_LIST]/* Clear the list. */bzero( (char *)cblist, sizeof(cblist) );/* Load one or more references into the list */cblist[0] = &my_aiocb;ret = aio_read( &my_aiocb );ret = aio_suspend( cblist, MAX_LIST, NULL );

Note that the second parameter of aio_suspend is the number of elements in cblist, rather than the number referenced by aiocb. Any null element in cblist will be ignored by aio_suspend.
If timeout is provided for aio_suspend and the timeout does occur, it returns-1 and errno contains eagain.
(6) aio_cancel
The aio_cancel function allows us to cancel one or all I/O requests executed on a file descriptor. The prototype is as follows:
Int aio_cancel (int fd, struct aiocb * aiocbp );
To cancel a request, we need to provide the file descriptor and aiocb reference. If the request is successfully canceled, the function returns aio_canceled. If the request is complete, this function returns aio_notcanceled.
To cancel all requests to a given file descriptor, we need to provide the file descriptor and a null reference to aiowhite. If all requests are canceled, this function returns aio_canceled. if at least one request is not canceled, this function returns aio_not_canceled. If no request can be canceled, then this function will return aio_alldone. We can then use aio_error to verify each AIO request. If the request has been canceled, aio_error returns-1 and errno is set to ecanceled.
(7) lio_listio
Finally, Aio provides a method to use the lio_listio API function to initiate multiple transmissions at the same time.This function is very important, because it means that we can start a large number of I/O operations in a system call (one kernel context switch. From the performance perspective, this is very important.So it is worth a moment to explore. The lio_listio API function is prototype as follows:
Int lio_listio (INT mode, struct aiocb * list [], int NENT, struct sigevent * sig );
The mode parameter can be lio_wait or lio_nowait. Lio_wait will block this call until all I/O operations are completed. After the operation is queued, lio_nowait will return. List is a list referenced by aiocb. the maximum number of elements is defined by NENT. Note that the list element can be null, and lio_listio will ignore it. The sigevent reference defines how signals are generated when all I/O operations are completed.
Lio_listio requests are slightly different from traditional read or write requests in the operation that must be specified, as shown in Listing 4.
Listing 4. Using the lio_listio function to initiate a series of requests

struct aiocb aiocb1, aiocb2;struct aiocb *list[MAX_LIST];.../* Prepare the first aiocb */aiocb1.aio_fildes = fd;aiocb1.aio_buf = malloc( BUFSIZE+1 );aiocb1.aio_nbytes = BUFSIZE;aiocb1.aio_offset = next_offset;aiocb1.aio_lio_opcode = LIO_READ;...bzero( (char *)list, sizeof(list) );list[0] = &aiocb1;list[1] = &aiocb2;ret = lio_listio( LIO_WAIT, list, MAX_LIST, NULL );

For read operations, the value of the aio_lio_opcode field is lio_read. For write operations, we need to use lio_write, but lio_nop is also effective for not performing operations.
3. AIO notification
Now we have read the available AIO functions. This section describes how to use asynchronous notifications in depth. We will explore the asynchronous function notification mechanism through signal and function callback.
(1) asynchronous notification using signals
Using signals for inter-process communication (IPC) is a traditional mechanism in Unix. AIO can also support this mechanism. In this example, the application needs to define a signal processing program, which will be called when a specified signal is generated. The application then configures an asynchronous request to generate a signal when the request is complete. As part of the signal context, specific aiocb requests are provided to record multiple possible requests. Listing 5 shows this notification method.
Listing 5. Using signals as AIO request notifications

void setup_io( ... ){  int fd;  struct sigaction sig_act;  struct aiocb my_aiocb;  ...  /* Set up the signal handler */  sigemptyset(&sig_act.sa_mask);  sig_act.sa_flags = SA_SIGINFO;  sig_act.sa_sigaction = aio_completion_handler;  /* Set up the AIO request */  bzero( (char *)&my_aiocb, sizeof(struct aiocb) );  my_aiocb.aio_fildes = fd;  my_aiocb.aio_buf = malloc(BUF_SIZE+1);  my_aiocb.aio_nbytes = BUF_SIZE;  my_aiocb.aio_offset = next_offset;  /* Link the AIO request with the Signal Handler */  my_aiocb.aio_sigevent.sigev_notify = SIGEV_SIGNAL;  my_aiocb.aio_sigevent.sigev_signo = SIGIO;  my_aiocb.aio_sigevent.sigev_value.sival_ptr = &my_aiocb;  /* Map the Signal to the Signal Handler */  ret = sigaction( SIGIO, &sig_act, NULL );  ...  ret = aio_read( &my_aiocb );}void aio_completion_handler( int signo, siginfo_t *info, void *context ){  struct aiocb *req;  /* Ensure it's our signal */  if (info->si_signo == SIGIO) {    req = (struct aiocb *)info->si_value.sival_ptr;    /* Did the request complete? */    if (aio_error( req ) == 0) {      /* Request completed successfully, get the return status */      ret = aio_return( req );    }  }  return;}

In listing 5, we set a signal handler in the aio_completion_handler function to capture sigio signals. Then initialize the aio_sigevent structure and generate a sigio signal for notification (this is specified by the sigev_signal definition in sigev_notify ). When the read operation is complete, the signal processing program extracts aiocb from the si_value structure of the signal and checks the error status and return status to determine whether the I/O operation is complete.
For performance, this handler is also ideal for continuing I/O operations by requesting the next asynchronous transmission. In this way, when a data transmission is completed, we can start the next data transmission operation immediately.
(2) asynchronous notification using callback Functions
Another notification method is the system callback function. This mechanism does not generate a signal for the notification, but calls a function of the user space to implement the notification function. We have set a reference to aiocb in the sigevent structure to uniquely identify the specific request being completed. See Listing 6.
Listing 6. Use thread callback notification for AIO requests

void setup_io( ... ){  int fd;  struct aiocb my_aiocb;  ...  /* Set up the AIO request */  bzero( (char *)&my_aiocb, sizeof(struct aiocb) );  my_aiocb.aio_fildes = fd;  my_aiocb.aio_buf = malloc(BUF_SIZE+1);  my_aiocb.aio_nbytes = BUF_SIZE;  my_aiocb.aio_offset = next_offset;  /* Link the AIO request with a thread callback */  my_aiocb.aio_sigevent.sigev_notify = SIGEV_THREAD;  my_aiocb.aio_sigevent.notify_function = aio_completion_handler;  my_aiocb.aio_sigevent.notify_attributes = NULL;  my_aiocb.aio_sigevent.sigev_value.sival_ptr = &my_aiocb;  ...  ret = aio_read( &my_aiocb );}void aio_completion_handler( sigval_t sigval ){  struct aiocb *req;  req = (struct aiocb *)sigval.sival_ptr;  /* Did the request complete? */  if (aio_error( req ) == 0) {    /* Request completed successfully, get the return status */    ret = aio_return( req );  }  return;}

In Listing 6, after creating your own aiocb request, we use sigev_thread to request a thread callback function as the notification method. Then we will specify a specific notification handler and load the context to be transmitted to the handler (in this case, it is a reference to the aiocb request itself ). In this processing program, we simply reference the arriving sigval pointer and use the AIO function to verify that the request has been completed.
4. System Optimization for AIO
The proc file system contains two virtual files that can be used to optimize the performance of asynchronous I/O:
/Proc/sys/fs/AIO-Nr file: provides the current number of System-range asynchronous I/O requests.
/Proc/sys/fs/AIO-max-Nr file: the maximum number of concurrent requests allowed. The maximum number is usually 64 KB, which is sufficient for most applications.
5. Conclusion

Using asynchronous I/O can help us build applications with faster I/O speeds and higher efficiency. If our applications can overlap processing with I/O operations, then AIO can help us build applications that can more efficiently use available CPU resources. Although this I/O model is different from the traditional blocking mode used in most Linux applications, the asynchronous notification model is very simple in concept and can simplify our design.

From: http://www.ibm.com/developerworks/linux/library/l-async/

Here: http://www.ibm.com/developerworks/cn/linux/l-async/