"Redis source Profiling"-Rio for Redis IO operations

original works, reproduced please indicate: http://blog.csdn.net/Xiejingfa/article/details/51433696

The Reids internally encapsulates an I/O layer called Rio. Today we will briefly introduce the specific implementation of the Rio module.

This paper mainly deals with two files of Rio.h and rio.c.

1. Rio Structural Body

The operation of file read and write operations and buffer is mainly based on the Rio object, so let's take a look at the definition of the Rio struct, as follows:

/ * System IO operation package * /
struct _rio {
    / * Backend functions.
     * Since this functions do not tolerate short writes or reads the return
     * value is simplified to: zero on error, non zero on complete success. * /
    / * Back-end method: The function returns a value of 0 to indicate an error, and a return value of non-zero to indicate that the operation was successful. * /

    // data stream read operation
    size_t (* read) (struct _rio *, void * buf, size_t len);
    // data stream write operation
    size_t (* write) (struct _rio *, const void * buf, size_t len);
    // The current offset of the read or write operation
    off_t (* tell) (struct _rio *);
    // flush operation
    int (* flush) (struct _rio *);
    / * The update_cksum method if not NULL is used to compute the checksum of
     * all the data that was read or written so far. The method should be
     * designed so that can be called with the current checksum, and the buf
     * and len fields pointing to the new block of data to add to the checksum
     * computation. * /
    // update checksum
    void (* update_cksum) (struct _rio *, const void * buf, size_t len);

    / * The current checksum * /
    // current checksum
    uint64_t cksum;

    / * number of bytes read or written * /
    // number of bytes read or written
    size_t processed_bytes;

    / * maximum single read or write chunk size * /
    // Maximum number of bytes per read or write operation
    size_t max_processing_chunk;

    / * Backend-specific vars. * /
    // io variable
    union {
        / * In-memory buffer target. * /
        // memory buffer structure
        struct {
            // The content in buffer is actually a char array
            sds ptr;
            // Offset
            off_t pos;
        } buffer;
        / * Stdio file pointer target. * /
        // file structure
        struct {
            // open file handle
            FILE * fp;
            // number of bytes written after the last fsync
            off_t buffered; / * Bytes written since last fsync. * /
            // how many bytes to perform an fsync operation
            off_t autosync; / * fsync after ‘autosync’ bytes written. * /
        } file;
        / * Multiple FDs target (used to write to N sockets). * /
        // encapsulates multiple file descriptor structures (for writing multiple sockets)
        struct {
            // file descriptor array
            int * fds; / * File descriptors. * /
            // status bit, corresponding to fds
            int * state; / * Error state of each fd. 0 (if ok) or errno. * /
            // number of file descriptors
            int numfds;
            // Offset
            off_t pos;
            // buffer
            sds buf;
        } fdset;
    } io;
}; 

We can see that the Rio structure consists of three main elements:

1. Read-write operations, get offset operations, and other related callback functions. Rio can handle three different types of I/O objects in buffer, file, and socket, and the different Rio objects use the corresponding system calls to complete the read, write, tell, flush operations. For example, for the file Rio object, the bottom layer through the Fwrite function to complete the write operation, through the Fread function to complete the read operation (see source code).
2. The checksum operation. Rio uses the RCR64 algorithm to calculate checksums, which can be implemented in Crc64.h and crc64.c files.
3. IO variable. The IO member in the _rio is a consortium that performs different processing for different I/O situations: When I/O operations in memory buffer are performed, the rio.buffer structure is used, when the file I/O operation is performed, the rio.file structure is used, and when the socket is executed i/ o when operating, use the Rio.fdset structure body.

After introducing the Rio structure, let's look at the implementation of the buffer Rio object (the file Rio object and the socket Rio object's reality can be referred to the annotated version of the source code, here is not explained).

The write operation of buffer is actually storing the data in R->io.buffer, which is implemented by therioBufferWritefunction:

/ * Appends the content of the specified length len in buf to the buffer of the rio object, the operation returns 1 successfully, otherwise returns 0. * /
static size_t rioBufferWrite (rio * r, const void * buf, size_t len) {
     // call sdscatlen to implement the append operation
     r-> io.buffer.ptr = sdscatlen (r-> io.buffer.ptr, (char *) buf, len);
     // Update length information
     r-> io.buffer.pos + = len;
     return 1;
} 

The write operation of buffer is actually to read the data from the R->io.buffer, which is implemented by therioBufferReadfunction:

/ * Read the contents of length len from the buffer of the rio object into buf, the operation returns 1 successfully, otherwise returns 0. * /
static size_t rioBufferRead (rio * r, void * buf, size_t len) {
     // If the length of the content in the buffer of the rio object is less than len, the read fails
     if (sdslen (r-> io.buffer.ptr) -r-> io.buffer.pos <len)
         return 0; / * not enough buffer to return len bytes. * /
     // Copy the contents of the buffer to buf
     memcpy (buf, r-> io.buffer.ptr + r-> io.buffer.pos, len);
     // update offset
     r-> io.buffer.pos + = len;
     return 1;
} 

The tell operation and flush operations of buffer are as follows:

/ * Returns the offset of the rio object buffer * /
static off_t rioBufferTell (rio * r) {
     return r-> io.buffer.pos;
}

/ * This function does nothing and returns 1. * /
static int rioBufferFlush (rio * r) {
     // REDIS_NOTUSED is defined in the redis.h file: #define REDIS_NOTUSED (V) ((void) V)
     REDIS_NOTUSED (r);
     return 1; / * Nothing to do, our write just appends to the buffer. * /
} 

Based on the functions defined above, we can create a buffer Rio object:

/ * The buffer rio object used when the stream defined by the method above is memory * /
static const rio rioBufferIO = {
     rioBufferRead,
     rioBufferWrite,
     rioBufferTell,
     rioBufferFlush,
     NULL, / * update_checksum * /
     0, / * current checksum * /
     0, / * bytes read or written * /
     0, / * read / write chunk size * /
     {{NULL, 0}} / * union for io-specific vars * /
}; 

rioInitWithBufferThe function is responsible for initializing the buffer Rio object and creating the Io.buffer buffer.

/ * Initialize buffer io object * /
void rioInitWithBuffer (rio * r, sds s) {
     * r = rioBufferIO;
     r-> io.buffer.ptr = s;
     r-> io.buffer.pos = 0;
} 

2. Unified reading and writing interface

The Rio module encapsulates the Redis read-write operation, file read and write operations, socket read and write operations, and also defines a unified read, write, tell, and Flus operation interface.

To perform a write operation for Rio:

/ * Writes characters of length len in the buf array to the rio object, and returns 1 if the write is successful, and 0 if the write fails. * /
static inline size_t rioWrite (rio * r, const void * buf, size_t len) {
     while (len) {
         // Determine whether the number of bytes currently required to be written has operated the maximum length specified by max_processing_chunk
         size_t bytes_to_write = (r-> max_processing_chunk && r-> max_processing_chunk <len)? r-> max_processing_chunk: len;
         // Update the checksum field when writing new data
         if (r-> update_cksum) r-> update_cksum (r, buf, bytes_to_write);
         // Call the write method to perform the write operation
         if (r-> write (r, buf, bytes_to_write) == 0)
             return 0;
         // Update the position where buf writes next time
         buf = (char *) buf + bytes_to_write;
         len-= bytes_to_write;
         // Update the number of bytes written
         r-> processed_bytes + = bytes_to_write;
     }
     return 1;
} 

To perform the read operation of Rio:

/ * Read len bytes of data from the rio object and save it to the buf array. A successful read returns 1 and a failed read returns 0. * /
static inline size_t rioRead (rio * r, void * buf, size_t len) {
     while (len) {
         // Determine whether the number of bytes currently required to be read has operated the maximum length specified by max_processing_chunk
         size_t bytes_to_read = (r-> max_processing_chunk && r-> max_processing_chunk <len)? r-> max_processing_chunk: len;
         // Call the read method to read data into buf
         if (r-> read (r, buf, bytes_to_read) == 0)
             return 0;
         // Update the position where buf writes next time
         if (r-> update_cksum) r-> update_cksum (r, buf, bytes_to_read);
         buf = (char *) buf + bytes_to_read;
         len-= bytes_to_read;
         // Update the number of bytes read
         r-> processed_bytes + = bytes_to_read;
     }
     return 1;
} 

To perform a tell operation for Rio:

/ * Returns the current offset * /
static inline off_t rioTell (rio * r) {
     return r-> tell (r);
} 

To perform a flush operation of Rio:

/ * flush operation * /
static inline int rioFlush (rio * r) {
     return r-> flush (r);
} 

It can be seen from the implementation of the above four functions that they all call the callback functions inside the Rio object to perform the corresponding write, read, tell, flush operations, and these callback functions are done with the Rio object as a buffer Rio object or a file Rio object. Or the socket Rio object.

3. Other

The Rio module in Redis also encapsulates a number of functions that assist in generating the AOF protocol, which mainly includes:

// Write count to the rio object as a string in the form "* <count> \ r \ n" and return the number of bytes written.
size_t rioWriteBulkCount (rio * r, char prefix, int count);
// Write a binary secure string to the rio object in the format "$ <count> \ r \ n <payload> \ r \ n".
size_t rioWriteBulkString (rio * r, const char * buf, size_t len);
// Write a long long value to the rio object in the format "$ <count> \ r \ n <payload> \ r \ n".
size_t rioWriteBulkLongLong (rio * r, long long l);
// Write a value of type double to the rio object in the format "$ <count> \ r \ n <payload> \ r \ n".
size_t rioWriteBulkDouble (rio * r, double d); 

The Rio module in Redis encapsulates the system I/O functions. In the Redis internal implementation, the RDB, AOF, etc. all use the function of the module, so here is a brief introduction of the Rio module implementation principle.

SOURCE See:

1. Rio.h:https://github.com/xiejingfa/the-annotated-redis-3.0/blob/master/rio.h
2. Rio.c:https://github.com/xiejingfa/the-annotated-redis-3.0/blob/master/rio.c

"Redis source Profiling"-Rio for Redis IO operations