Redis source code analysis (27th) --- encapsulation of the Rio system I/O

I/O operations are an essential part of every system. In addition, I/O operations may also affect the system efficiency to a certain extent. Today, I learned how to handle I/O in redis. Similarly, redis also encapsulates an I/O layer in its own system. Rio for short. First, let's take a look at what's in Rio:

Struct _ Rio {/* backend functions. * Since this functions do not tolerate short writes or reads the return * value is simplified to: Zero on error, non zero on complete success. * // * Data Stream read Method */size_t (* read) (struct _ Rio *, void * Buf, size_t Len ); /* Data Stream write Method */size_t (* write) (struct _ Rio *, const void * Buf, size_t Len ); /* Get the current read/write offset */off_t (* Tell) (struct _ Rio *); /* The update_cksum method if not null is used to compute the checksum of * all the data that was read or written so far. the method shoshould be * designed so that can be called with the current checksum, And the Buf * And Len fields pointing to the new block of data to add to the checksum * computation. * // * when a new data block is read, the current checksum is updated */void (* update_cksum) (struct _ Rio *, const void * Buf, size_t Len ); /* The current checksum * // * current checksum */uint64_t cksum; /* number of bytes read or written * // * size of the currently read or written bytes */size_t processed_bytes; /* maximum single read or write chunk size * // * maximum size of a single read/write */size_t max_processing_chunk;/* backend-specific vars. * // * I/O variable in Rio */Union {// buffer struct {// the specific content of the buffer SDS PTR; // offset off_t Pos;} buffer; // file struct {file * FP; off_t buffered;/* bytes written since last fsync. * /// minimum synchronization size off_t autosync;/* fsync after 'autosync' bytes written. */} file;} io ;};

In addition to the three required methods, the Read and Write methods, and the tell method for obtaining the offset, there are two struct variables, a buffer struct, and a file struct, i/O operations are different. buffer. When I/O operations are performed on files, Rio is executed. file. Let's take a look at the Read and Write Methods uniformly defined by Rio:

/* The following functions are our interface with the stream. they'll call the * Actual implementation of read/write/tell, and will update the checksum * if needed. * // * Rio write Method */static inline size_t riowrite (Rio * r, const void * Buf, size_t Len) {While (LEN) {// determine whether the length of the current operation byte exceeds the maximum length of size_t bytes_to_write = (R-> max_processing_chunk & R-> max_processing_chunk <Len )? R-> max_processing_chunk: Len; // when writing new data, update the checksum and if (R-> update_cksum) r-> update_cksum (R, Buf, bytes_to_write ); // execute the write method if (R-> write (R, Buf, bytes_to_write) = 0) return 0; Buf = (char *) BUF + bytes_to_write; Len-= bytes_to_write; // increase the number of Operation bytes R-> processed_bytes + = bytes_to_write;} return 1;}/* Rio's read Method */static inline size_t rioread (Rio * r, void * Buf, size_t Len) {While (LEN) {// determines whether the length of the current operation byte exceeds the maximum length of size_t Bytes_to_read = (R-> max_processing_chunk & R-> max_processing_chunk <Len )? R-> max_processing_chunk: Len; // read data method if (R-> Read (R, Buf, bytes_to_read) = 0) return 0; // when reading data, update the checksum (R-> update_cksum) r-> update_cksum (R, Buf, bytes_to_read); Buf = (char *) BUF + bytes_to_read; Len-= bytes_to_read; r-> processed_bytes + = bytes_to_read;} return 1 ;}

There is a good place here. Every time there is a change in data, redis will make a processing algorithm for calculating the checksum, indicating the change in data operations, the crc64 algorithm is introduced before. For buffer Io and file IO of Rio, redis defines two Rio structs:

/* According to the method described above, bufferrio */static const Rio riobufferio = {riobufferread, riobufferwrite, riobuffertell, null,/* update_checksum */0, /* current checksum */0,/* bytes read or written */0,/* read/write chunk size */{null, 0}/* Union for IO-specific vars */};/* defines filerio */static const Rio riofileio = {riofileread, riofilewrite, riofiletell according to the method described above, null,/* update_checksum */0,/* current checksum */0,/* bytes read or written */0,/* read/write chunk size */{null, 0}/* Union for IO-specific vars */};

The corresponding read/write methods are defined respectively, such as the buffer read method and the file read method:

/* Returns 1 or 0 for success/failure. * // * read the buffer content in Rio to the input parameter */static size_t riobufferread (Rio * r, void * Buf, size_t Len) {If (sdslen (R-> Io. buffer. PTR)-R-> Io. buffer. pos <Len) return 0;/* not enough buffer to return len bytes. */memcpy (BUF, R-> Io. buffer. PTR + R-> Io. buffer. POs, Len); R-> Io. buffer. pos + = Len; return 1 ;}

/* Returns 1 or 0 for success/failure. * // * read the FP file in Rio */static size_t riofileread (Rio * r, void * Buf, size_t Len) {return fread (BUF, Len, 1, r-> Io. file. FP );}

The object variables of Rio are different. Finally, four write methods for different types of data are provided in the redis statement:

/* The riowrite Method */size_t riowritebulkcount (Rio * r, char prefix, int count); size_t riowritebulkstring (Rio * r, const char * Buf, size_t Len); size_t riowritebulklonglong (Rio * r, long l); size_t riowritebulkdouble (Rio * r, double D );

Here is a method implementation:

/* Write multi bulk count in the format: "* <count> \ r \ n ". * // * Rio writes data of different types and calls the riowrite Method */size_t riowritebulkcount (Rio * r, char prefix, int count) {char cbuf [128]; int clen; cbuf [0] = prefix; clen = 1 + ll2string (cbuf + 1, sizeof (cbuf)-1, count ); cbuf [clen ++] = '\ R'; cbuf [clen ++] =' \ n'; If (riowrite (R, cbuf, clen) = 0) return 0; return clen ;}

The riowrite method is called. You can define the buffer Io or file IO according to your own implementation. There is a detail in the file write method. When you read the content into Rio. file. when the buffer exceeds the given minimum Synchronization Byte, you must refresh the buffer content to the file.

/* Returns 1 or 0 for success/failure. * // * write the Buf to the file in Rio */static size_t riofilewrite (Rio * r, const void * Buf, size_t Len) {size_t retval; retval = fwrite (BUF, len, 1, R-> Io. file. FP); R-> Io. file. buffered + = Len; If (R-> Io. file. autosync & R-> Io. file. buffered> = r-> Io. file. autosync) {// determine whether to synchronize fflush (R-> Io. file. FP); aof_fsync (fileno (R-> Io. file. FP); R-> Io. file. buffered = 0;} return retval ;}

Redis source code analysis (27th) --- encapsulation of the Rio system I/O