Store ID continuous data with Redis

Previously, when I designed Doodle2 and developed zhihu daily, the most data type I faced was ID-based data. When using relational databases, the auto-increment primary key can meet this requirement, but it is a little more troublesome in Redis. I have always used an additional counter to store the ID. For example, I have omitted a lot of code and it does not affect reading.

Previously, when I designed "dota2 2" and developed "zhihu daily", the most data type I faced was the data with ID. When using relational databases, the auto-increment primary key can meet this requirement, but it is a little more troublesome in Redis. I have always used an additional counter to store the ID. For example, I have omitted a lot of code and it does not affect reading.

Previously, when I designed "dota2 2" and developed "zhihu daily", the most data type I faced was the data with ID.
When using relational databases, the auto-increment primary key can meet this requirement, but it is a little more troublesome in Redis.

I have always used an additional counter to store the ID. For example, I have omitted a lot of code and will not affect reading ):

class IDModel(JSONModel):    id = IntegerProperty()    @classmethod    def get_by_id(cls, entity_id):        json_content = cls.redis_client.hget(cls._KEY, entity_id)        if json_content:            return cls.from_json(json_content)    @classmethod    def get_by_ids(cls, ids):        if not ids:            return []        results = cls.redis_client.hmget(cls._KEY, ids)        return [cls.from_json(json_content)                for json_content in results]    @classmethod    def get_next_id(cls):        return MAX_ID.get_next_id(cls._KEY)    def save(self):        self._populate_default_attributes()        self._save_self()    def _populate_default_attributes(self):        if self.id is None:            self.id = self.get_next_id()    def _save_self(self):        self.redis_client.hset(self._KEY, self.id, self.to_json())class MAX_ID(SimpleModel):    @classmethod    def get_next_id(cls, for_type, increment=1):        return cls.redis_client.hincrby(cls._KEY, for_type, increment)    @classmethod    def get_max_id(cls, for_type):        return int(cls.redis_client.hget(cls._KEY, for_type) or 0)

The advantage of this method is that it is simple and suitable for storing data with Discontinuous IDs. The disadvantage is that hash is used, which accounts for memory usage.

Just now I thought of another method to store the data directly in the list and determine the ID based on its length. For example:

class IDModel(JSONModel):    id = IntegerProperty()    @classmethod    def get_by_id(cls, entity_id):        json_content = cls.redis_client.lindex(cls._KEY, entity_id)        if json_content:            return cls.from_json(json_content)    @classmethod    def get_by_ids(cls, ids):        if not ids:            return []        key = cls._KEY        pipe = cls.redis_client.pipeline(transaction=False)        for entity_id in ids:            pipe.lindex(key, entity_id)        results = pipe.execute()        return [cls.from_json(json_content)                for json_content in results]    def save(self):        key = self._KEY        if self.id is None:            with self.redis_client.pipeline() as pipe:                try:                    pipe.watch(key)                    self.id = pipe.llen(key) + 1                    pipe.multi()                    pipe.rpush(key, self.to_json())                    pipe.execute()                except Exception:                    self.id = None                    raise        else:            self.redis_client.lset(key, self.id, self.to_json())

There are roughly three disadvantages:

1. ID must be consecutive
   
2. Dependent on transactions, resulting in poor separation of the logic of _ populate_default_attributes ()
   
3. Multiple statements are required to obtain the object corresponding to multiple IDs.
   

However, in the face of the advantages of saving memory, others are floating clouds ......

Original article address: Use Redis to store consecutive ID data. Thank you for sharing it with the original author.