Redis source code analysis 19-master-slave Replication

Let's talk about the features of Redis master-slave replication. Official ReplicationHowto documents mention the following features: 1. one master supports multiple slave2.slave connections that can be connected to other slave instances and act as the master of other slave instances to form a master-slave multi-level structure. the replication on the master side is non-blocking, that is, the master is

Let's talk about the features of Redis master-slave replication. Official ReplicationHowto documents mention the following features: 1. one master supports multiple slave 2. server Load balancer can accept connections from other Server Load balancer instances and act as masters of other Server Load balancer instances to form a master-Server Load balancer multi-level structure. the replication on the master side is non-blocking, that is, the master is

Let's talk about the features of Redis master-slave replication.

Official ReplicationHowto documents mention the following features:
1. One master supports multiple slave instances
2. The Server Load balancer can accept connections from other Server Load balancer instances and act as the master nodes of other Server Load balancer instances to form a master-Server Load balancer multi-level structure.
3. replication on the master side is non-blocking, that is, when the master node replicates data to the client, it can process commands from other clients, while the slave is blocked during the first synchronization.
4. replication is used to provide scalability. For example, you can use the slave end as data redundancy, or send time-consuming commands (such as sort) to some slave to avoid master congestion, you can also use slave for persistence. You only need to comment out the save command in the master configuration file.

The client can be used as a server Load balancer to connect to the master node at the beginning, or publish the sync command after running to establish a master-slave relationship with the master node.

Next we will give an overview of the master-slave mechanism of redis from the perspective of slave and master respectively.

If redis runs as a slave, the global variable server. the replstate status includes REDIS_REPL_NONE (not in the replication status), REDIS_REPL_CONNECT (a connection needs to be established with the master), and REDIS_REPL_CONNECTED (a connection has been established with the master. After reading the slaveof configuration or publishing the slaveof command, the value of server. replstate is REDIS_REPL_CONNECT, and then the value changes to REDIS_REPL_CONNECTED after the first synchronization between syncWithMaster and master.

If redis runs as the master, it corresponds to the variable slave connected to a client. replstate statuses include REDIS_REPL_WAIT_BGSAVE_START (waiting for bgsave to run), running (bgsave has dump db, this bulk has been transferred), REDIS_REPL_SEND_BULK (being bulk transmitted), REDIS_REPL_ONLINE, you only need to send updates later ). For the slave client (publish the sync command), start with slave. replstate is in the running state (the syncCommand function is described later), and The backgroundSaveDoneHandler function is in the running state after the dump db in the background. Then, updateSlavesWaitingBgsave sets the status to REDIS_REPL_SEND_BULK, the write event function sendBulkToSlave is set. After sendBulkToSlave is run, the status changes to REDIS_REPL_ONLINE. After that, the master will always call replicationFeedSlaves to send new commands to the slave in REDIS_REPL_ONLINE status.

Let's first look at the code that redis on the master side will execute.

The slave end synchronizes data with the master by releasing the sync command. The syncCommand processing function of the sync command is as follows.

The annotations in this function are clear enough. If the slave client sets the REDIS_SLAVE flag, the master has used syncCommand to process the slave. If the master still does not send reply to this client, an error message is returned. If server. bgsavechildpid! =-1 and slave is in the REDIS_REPL_WAIT_BGSAVE_END state, it means that the background process of dump db has just ended. At this time, the new slave can directly use the saved rdb for bulk transmission (note that copy the reply parameter, because the master node is not blocked, some commands may be executed at this time. The call function will call the replicationFeedSlaves function to save the command parameters to the reply parameter of slave ). If no slave is in REDIS_REPL_WAIT_BGSAVE_END state, but server. bgsavechildpid! =-1, it indicates that the bgsave background process is not finished and needs to wait for it to end (the bgsave background process will process the waiting slave after it ends ). If server. bgsavechildpid is equal to-1, you need to start a background process to dump the db. Finally, add the current client to the Server Load balancer linked list of the master.

static void syncCommand(redisClient *c) {    /* ignore SYNC if aleady slave or in monitor mode */    if (c->flags & REDIS_SLAVE) return;    /* SYNC can't be issued when the server has pending data to send to     * the client about already issued commands. We need a fresh reply     * buffer registering the differences between the BGSAVE and the current     * dataset, so that we can copy to other slaves if needed. */    if (listLength(c->reply) != 0) {        addReplySds(c,sdsnew("-ERR SYNC is invalid with pending input\r\n"));        return;    }    redisLog(REDIS_NOTICE,"Slave ask for synchronization");    /* Here we need to check if there is a background saving operation     * in progress, or if it is required to start one */    if (server.bgsavechildpid != -1) {        /* Ok a background save is in progress. Let's check if it is a good         * one for replication, i.e. if there is another slave that is         * registering differences since the server forked to save */        redisClient *slave;        listNode *ln;        listIter li;        listRewind(server.slaves,&li);        while((ln = listNext(&li))) {            slave = ln->value;            if (slave->replstate == REDIS_REPL_WAIT_BGSAVE_END) break;        }        if (ln) {            /* Perfect, the server is already registering differences for             * another slave. Set the right state, and copy the buffer. */            listRelease(c->reply);            c->reply = listDup(slave->reply);            c->replstate = REDIS_REPL_WAIT_BGSAVE_END;            redisLog(REDIS_NOTICE,"Waiting for end of BGSAVE for SYNC");        } else {            /* No way, we need to wait for the next BGSAVE in order to             * register differences */            c->replstate = REDIS_REPL_WAIT_BGSAVE_START;            redisLog(REDIS_NOTICE,"Waiting for next BGSAVE for SYNC");        }    } else {        /* Ok we don't have a BGSAVE in progress, let's start one */        redisLog(REDIS_NOTICE,"Starting BGSAVE for SYNC");        if (rdbSaveBackground(server.dbfilename) != REDIS_OK) {            redisLog(REDIS_NOTICE,"Replication failed, can't BGSAVE");            addReplySds(c,sdsnew("-ERR Unalbe to perform background save\r\n"));            return;        }        c->replstate = REDIS_REPL_WAIT_BGSAVE_END;    }    c->repldbfd = -1;    c->flags |= REDIS_SLAVE;    c->slaveseldb = 0;    listAddNodeTail(server.slaves,c);    return;}

After that, slave will be processed only after the dump db background process is completed, whether in REDIS_REPL_WAIT_BGSAVE_START or REDIS_REPL_WAIT_BGSAVE_END. After the process ends, the backgroundSaveDoneHandler function is executed, which calls updateSlavesWaitingBgsave to process slaves.

Like syncCommand, updateSlavesWaitingBgsave involves several slave status changes. For slave waiting for dump db, the master will put it into the server. slaves linked list. In this case, if slave-> replstate = REDIS_REPL_WAIT_BGSAVE_START, it indicates that the current dump db is not required by this slave. redis needs to restart the background process to dump the db. If slave-> replstate = REDIS_REPL_WAIT_BGSAVE_END, it indicates that the current dump db is exactly what this slave needs. In this case, set sendBulkToSlave, the write event processing function of the slave.

static void updateSlavesWaitingBgsave(int bgsaveerr) {    listNode *ln;    int startbgsave = 0;    listIter li;    listRewind(server.slaves,&li);    while((ln = listNext(&li))) {        redisClient *slave = ln->value;        if (slave->replstate == REDIS_REPL_WAIT_BGSAVE_START) {            startbgsave = 1;            slave->replstate = REDIS_REPL_WAIT_BGSAVE_END;        } else if (slave->replstate == REDIS_REPL_WAIT_BGSAVE_END) {            struct redis_stat buf;            if (bgsaveerr != REDIS_OK) {                freeClient(slave);                redisLog(REDIS_WARNING,"SYNC failed. BGSAVE child returned an error");                continue;            }            if ((slave->repldbfd = open(server.dbfilename,O_RDONLY)) == -1 ||                redis_fstat(slave->repldbfd,&buf) == -1) {                freeClient(slave);                redisLog(REDIS_WARNING,"SYNC failed. Can't open/stat DB after BGSAVE: %s", strerror(errno));                continue;            }            slave->repldboff = 0;            slave->repldbsize = buf.st_size;            slave->replstate = REDIS_REPL_SEND_BULK;            aeDeleteFileEvent(server.el,slave->fd,AE_WRITABLE);            if (aeCreateFileEvent(server.el, slave->fd, AE_WRITABLE, sendBulkToSlave, slave) == AE_ERR) {                freeClient(slave);                continue;            }        }    }    if (startbgsave) {        if (rdbSaveBackground(server.dbfilename) != REDIS_OK) {            listIter li;            listRewind(server.slaves,&li);            redisLog(REDIS_WARNING,"SYNC failed. BGSAVE failed");            while((ln = listNext(&li))) {                redisClient *slave = ln->value;                if (slave->replstate == REDIS_REPL_WAIT_BGSAVE_START)                    freeClient(slave);            }        }    }}

The logic of sendBulkToSlave is not complex. It reads db data from the rdb file after dump according to the db directed by slave-> repldbfd, and then sends the data. After sending the message, the write event will be deleted and the slave-> replstate state will be set to REDIS_REPL_ONLINE. After that, the master will call the call function after receiving the command, and then synchronously update the slave using replicationFeedSlaves. ReplicationFeedSlaves also traverses the slave linked list and sends the current command and its parameters to the slave in REDIS_REPL_ONLINE status.

 static void sendBulkToSlave(aeEventLoop *el, int fd, void *privdata, int mask) {    redisClient *slave = privdata;    REDIS_NOTUSED(el);    REDIS_NOTUSED(mask);    char buf[REDIS_IOBUF_LEN];    ssize_t nwritten, buflen;    if (slave->repldboff == 0) {        /* Write the bulk write count before to transfer the DB. In theory here         * we don't know how much room there is in the output buffer of the         * socket, but in pratice SO_SNDLOWAT (the minimum count for output         * operations) will never be smaller than the few bytes we need. */        sds bulkcount;        bulkcount = sdscatprintf(sdsempty(),"$%lld\r\n",(unsigned long long)            slave->repldbsize);        if (write(fd,bulkcount,sdslen(bulkcount)) != (signed)sdslen(bulkcount))        {            sdsfree(bulkcount);            freeClient(slave);            return;        }        sdsfree(bulkcount);    }    lseek(slave->repldbfd,slave->repldboff,SEEK_SET);    buflen = read(slave->repldbfd,buf,REDIS_IOBUF_LEN);    if (buflen <= 0) {        redisLog(REDIS_WARNING,"Read error sending DB to slave: %s",            (buflen == 0) ? "premature EOF" : strerror(errno));        freeClient(slave);        return;    }    if ((nwritten = write(fd,buf,buflen)) == -1) {        redisLog(REDIS_VERBOSE,"Write error sending DB to slave: %s",            strerror(errno));        freeClient(slave);        return;    }    slave->repldboff += nwritten;    if (slave->repldboff == slave->repldbsize) {        close(slave->repldbfd);        slave->repldbfd = -1;        aeDeleteFileEvent(server.el,slave->fd,AE_WRITABLE);        slave->replstate = REDIS_REPL_ONLINE;        if (aeCreateFileEvent(server.el, slave->fd, AE_WRITABLE,            sendReplyToClient, slave) == AE_ERR) {            freeClient(slave);            return;        }        addReplySds(slave,sdsempty());        redisLog(REDIS_NOTICE,"Synchronization with slave succeeded");    }}

Next, let's take a look at how redis runs as a slave.

Redis as a server Load balancer (of course, you can use a common client as the Server Load balancer client, which is related to the implementation of the specific client), you need to specify the master location in the configuration file, when loadServerConfig reads the configuration parameters. replstate is set to REDIS_REPL_CONNECT. Redis in this status needs to run to serverCron before using syncWithMaster to perform initial synchronization with the master. View the syncWithMaster code. We can see that the sync command is also released to the master to establish the master-slave relationship. In addition, the function uses the syncRead and syncWrite functions to receive and send data, these functions are blocked. Therefore, redis is also blocked when it is used as the slave to establish the initial master-slave relationship.

 /* Check if we should connect to a MASTER */    if (server.replstate == REDIS_REPL_CONNECT && !(loops % 10)) {        redisLog(REDIS_NOTICE,"Connecting to MASTER...");        if (syncWithMaster() == REDIS_OK) {            redisLog(REDIS_NOTICE,"MASTER <-> SLAVE sync succeeded");            if (server.appendonly) rewriteAppendOnlyFileBackground();        }    }

Another command related to master-slave replication is the slaveof command. This command is a conversion function for redis Master/Slave States. According to the previous analysis, you only need to change several States.

static void slaveofCommand(redisClient *c) {    if (!strcasecmp(c->argv[1]->ptr,"no") &&        !strcasecmp(c->argv[2]->ptr,"one")) {        if (server.masterhost) {            sdsfree(server.masterhost);            server.masterhost = NULL;            if (server.master) freeClient(server.master);            server.replstate = REDIS_REPL_NONE;            redisLog(REDIS_NOTICE,"MASTER MODE enabled (user request)");        }    } else {        sdsfree(server.masterhost);        server.masterhost = sdsdup(c->argv[1]->ptr);        server.masterport = atoi(c->argv[2]->ptr);        if (server.master) freeClient(server.master);        server.replstate = REDIS_REPL_CONNECT;        redisLog(REDIS_NOTICE,"SLAVE OF %s:%d enabled (user request)",            server.masterhost, server.masterport);    }    addReply(c,shared.ok);}

Original article address: redis source code analysis 19-master-slave replication, thanks to the original author for sharing.