Redis source code analysis 24-VM (medium)

There are two ways for the VM to switch in and out based on value: blocking and multithreading (server. vm_max_threads0 is blocking ). This section describes blocking methods. When redis starts to rebuild the database (aof or snapshot), some values may be swapped out to the disk due to memory restrictions. At this time, only the blocking mode is used to replace v

There are two ways for the VM to switch in and out based on value: blocking and multithreading (server. vm_max_threads = 0 is blocking ). This section describes blocking methods. When redis starts to rebuild the database (aof or snapshot), some values may be swapped out to the disk due to memory restrictions. At this time, only the blocking mode is used to replace v

There are two ways for the VM to switch in and out based on value: blocking and multithreading (server. vm_max_threads = 0 is blocking ).

This section describes blocking methods.

When redis starts to rebuild the database (aof or snapshot), some values may be swapped out to the disk due to memory restrictions. In this case, only the value (vmSwapOneObjectBlocking function) is used for blocking ). In addition, redis only exchanges the value in the serverCron function (analyzed in this function event processing chapter. Let's take a look at the processing code in serverCron. The blocking method uses the vmSwapOneObjectBlocking function to exchange the value, and The multithreading method uses the vmSwapOneObjectThreaded function to exchange the value.

static int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {    ---    /* Swap a few keys on disk if we are over the memory limit and VM     * is enbled. Try to free objects from the free list first. */    if (vmCanSwapOut()) {        while (server.vm_enabled && zmalloc_used_memory() >                server.vm_max_memory)        {            ---            if (tryFreeOneObjectFromFreelist() == REDIS_OK) continue;            retval = (server.vm_max_threads == 0) ?                        vmSwapOneObjectBlocking() :                        vmSwapOneObjectThreaded();            ---        }    }    ---    return 100;}

VmSwapOneObjectBlocking is used to change the value, vmSwapOneObjectThreaded is used to change the value, and vmSwapOneObject (different call parameters) is called to change the value.

VmSwapOneObject randomly selects five items for each database to calculate its swappability. If it is a multi-threaded method, vmSwapObjectThreaded is called to generate the value; otherwise, vmSwapObjectBlocking is used to generate the value.

static int vmSwapOneObject(int usethreads) {    int j, i;    struct dictEntry *best = NULL;    double best_swappability = 0;    redisDb *best_db = NULL;    robj *key, *val;    for (j = 0; j < server.dbnum; j++) {        redisDb *db = server.db+j;        /* Why maxtries is set to 100?         * Because this way (usually) we'll find 1 object even if just 1% - 2%         * are swappable objects */        int maxtries = 100;        if (dictSize(db->dict) == 0) continue;        for (i = 0; i < 5; i++) {            dictEntry *de;            double swappability;            if (maxtries) maxtries--;            de = dictGetRandomKey(db->dict);            key = dictGetEntryKey(de);            val = dictGetEntryVal(de);            /* Only swap objects that are currently in memory.             *             * Also don't swap shared objects if threaded VM is on, as we             * try to ensure that the main thread does not touch the             * object while the I/O thread is using it, but we can't             * control other keys without adding additional mutex. */            if (key->storage != REDIS_VM_MEMORY ||                (server.vm_max_threads != 0 && val->refcount != 1)) {                if (maxtries) i--; /* don't count this try */                continue;            }            val->vm.atime = key->vm.atime; /* atime is updated on key object */            swappability = computeObjectSwappability(val);            if (!best || swappability > best_swappability) {                best = de;                best_swappability = swappability;                best_db = db;            }        }    }    if (best == NULL) return REDIS_ERR;    key = dictGetEntryKey(best);    val = dictGetEntryVal(best);    redisLog(REDIS_DEBUG,"Key with best swappability: %s, %f",        key->ptr, best_swappability);    /* Unshare the key if needed */    if (key->refcount > 1) {        robj *newkey = dupStringObject(key);        decrRefCount(key);        key = dictGetEntryKey(best) = newkey;    }    /* Swap it */    if (usethreads) {        vmSwapObjectThreaded(key,val,best_db);        return REDIS_OK;    } else {        if (vmSwapObjectBlocking(key,val) == REDIS_OK) {            dictGetEntryVal(best) = NULL;            return REDIS_OK;        } else {            return REDIS_ERR;        }    }}

VmSwapObjectBlocking will write the value to the vm file (the vmWriteObjectOnSwap function) After calculating the required switch page, and mark the corresponding vm page as used.

static int vmSwapObjectBlocking(robj *key, robj *val) {    off_t pages = rdbSavedObjectPages(val,NULL);    off_t page;    assert(key->storage == REDIS_VM_MEMORY);    assert(key->refcount == 1);    if (vmFindContiguousPages(&page,pages) == REDIS_ERR) return REDIS_ERR;    if (vmWriteObjectOnSwap(val,page) == REDIS_ERR) return REDIS_ERR;    key->vm.page = page;    key->vm.usedpages = pages;    key->storage = REDIS_VM_SWAPPED;    key->vtype = val->type;    decrRefCount(val); /* Deallocate the object from memory. */    vmMarkPagesUsed(page,pages);    redisLog(REDIS_DEBUG,"VM: object %s swapped out at %lld (%lld pages)",        (unsigned char*) key->ptr,        (unsigned long long) page, (unsigned long long) pages);    server.vm_stats_swapped_objects++;    server.vm_stats_swapouts++;    return REDIS_OK;}

For value loading, if it is multi-threaded, blockClientOnSwappedKeys will be used for advance loading, but the blocking method will only be loaded when the corresponding command is executed. In the end, both the blocking and multithreading modes call lookupKey to check whether the key is in the memory. If not, use vmLoadObject to load the value. This function is a blocking function to read the value.

static robj *lookupKey(redisDb *db, robj *key) {    dictEntry *de = dictFind(db->dict,key);    if (de) {        robj *key = dictGetEntryKey(de);        robj *val = dictGetEntryVal(de);        if (server.vm_enabled) {            if (key->storage == REDIS_VM_MEMORY ||                key->storage == REDIS_VM_SWAPPING)            {                /* If we were swapping the object out, stop it, this key                 * was requested. */                if (key->storage == REDIS_VM_SWAPPING)                    vmCancelThreadedIOJob(key);                /* Update the access time of the key for the aging algorithm. */                key->vm.atime = server.unixtime;            } else {                int notify = (key->storage == REDIS_VM_LOADING);                /* Our value was swapped on disk. Bring it at home. */                redisAssert(val == NULL);                val = vmLoadObject(key);                dictGetEntryVal(de) = val;                /* Clients blocked by the VM subsystem may be waiting for                 * this key... */                if (notify) handleClientsBlockedOnSwappedKey(db,key);            }        }        return val;    } else {        return NULL;    }}

Original article address: redis source code analysis 24-VM (in), thanks to the original author for sharing.