Simple analysis of Redis cache consumption memory data based on Python project (with detailed procedure)

Directory

1 Preparatory work

2 Concrete Implementation

1 Preparatory work

What is Redis?

Redis: A high-performance Key-value database. Support data persistence, can save in-memory data on disk, restart can be loaded again to use, provide string, list, set, Zset, hash and other data structure of the storage, and support the backup.

This article is suitable for scenarios where the amount of data that Redis caches in a project grows, and the Redis cache's data consumes more memory, and many of them are likely to be less valuable data. Since Redis is a key-value database, analyzing the data in it is not as intuitive as the MySQL database. At this point, we need to look for tools to analyze what data in the Redis cache is taking up more memory, and to analyze the value of these data stores in the context of the actual project. In order to make specific data deletion of the scheme, to liberate the server side of the valuable memory resources.

The tool that this article needs to use: Rdbtools and MySQL.

Rdbtools: Written in Python, you can parse a redis dump.rdb file. In addition, the following tools are available:

(1) Memory reports that generate data across all databases and keys

(2) Convert the dump file to JSON

(3) Comparison of two dump files using standard diff tool

Specific source GitHub Link: https://github.com/sripathikrishnan/redis-rdb-tools/

MySQL: An open-source and relatively lightweight relational database. This article uses Rdbtools to parse out a redis dump.rdb file and generate a memory report *.csv file (PS: The following action file is Result_facelive_ HOT.CSV), then import the file into the MySQL database and finally generate the *.csv file (PS: The file name redis_key_storage.csv generated in the SQL script below) by writing a specific SQL statement script to produce the desired data analysis results.

2 Concrete Implementation

The Rdbtools tool requires support for a version environment such as Python2.7 or Python3.6, in the following operations.

(1) Find the specific address of the Dump.rdb file generated by the native project using Redis. (PS: The project in this article is a Django-based framework deployed on Ubuntu systems, so the relevant commands are the actual operation of the system, other environments are basically similar, not introduced)

' *.rdb '

After you run the above command, you can see all of the specific addresses on this computer with the. rdb suffix file, and then find the specific address based on the actual project. For example, the address found in this article:

/home/facelive/redis/data/hot/dump.rdb

PS: Some projects, when using Redis, will rename the default Dump.rdb file, such as the name Db-dump.rdb file. So how to determine the specific name at this point?

You can view the contents of the redis.conf file for the project using the Redis database and combine the following commands:

" Dbfilename "

You can view the specific file name.

(2) Use Rdbtools to build *.csv files using Redis memory in your project

This requires that the project install the Rdbtools tool first, and the project is based on the Python environment. To activate the project's virtual environment, enter the command:

Pip Install Rdbtools PYTHON-LZF

(3) After the installation is complete, you can use the RDB command in your project's virtual environment. Here are the commands for generating memory reports in this article:

Rdb-c Memory/home/facelive/redis/data/hot/dump.rdb > ~/result_facelive_hot.csv  

The generated result_facelive_hot.csv file is stored in the server environment root directory. At this point, you can copy the generated files from the server to local, the specific operation command reference:

sudo scp [email protected]:/home/facelive/result_facelive_hot.csv.   Copy the remote files from the server to the local current root directory, the IP here is my own random write Oh

Then open the Result_facelive_hot.csv file locally, the results are as follows (the following results are open in the Windows environment OH):

You can see that the table has database (the corresponding databases), the type (the cache types), the key (the key name of the cache), Size_in_bytes (the key is the memory size, which is the core data analysis of this article), Encoding (encoding of the cache key), num_elements, and len_larget_element six columns of data.

(4) Import result_facelive_hot.csv into MySQL database for data analysis

First, select a database that has already been created in the local MySQL database and create a table in the database named Redis_hot (PS: The specific table name can be arbitrarily determined)

To create the SQL statement for the table:

DROP TABLE IF EXISTS' Redis_hot ';CREATE TABLE' Redis_hot ' ('Database`int( One)DEFAULT NULL, ' type 'varchar( -)DEFAULT NULL,  `Key`varchar( $)DEFAULT NULL, ' size_in_bytes 'int( One)DEFAULT NULL, ' encoding 'varchar(255)DEFAULT NULL, ' num_elements 'int( One)DEFAULT NULL, ' len_largest_element 'int( One)DEFAULT NULL) ENGINE=InnoDBDEFAULTCHARSET=UTF8;

After creating the Redis_hot table, we started using the Navicat tool for data import work. (PS: Of course, you can also use code to complete data import, here This article only gives a reference link, not detailed introduction, concrete at the end of the text is visible.) ）

The following actions are performed in a Windows environment, and other environments use the Navicat visualizer, which is basically similar to the steps.

First, use Navicat to open the local database, find the Redis_hot table you just created, right-click the mouse, select the Import Wizard, the process is as follows:

After the data import is complete, the following formally begins to write the SQL query script to generate the specific required analysis results data.

The data that needs to be analyzed here:

(1) Total memory size for each key (Size_in_bytes_sum)

(2) The total number of each key (PS: Because some key design is a prefix + user ID, this situation belongs to a key) (Record_count)

(3) database for each key location

(4) data type for each key (type)

(5) Encoding type for each key (encoding)

(6) Name of each key (key)

(7) Average memory size per key (SIZE_IN_BYTES_AVG)

The SQL script code used is as follows:

SELECT`Database', type, 'Key', Encoding,Count(`Database`) asRecord_count,SUM(size_in_bytes) asSize_in_bytes_sum,AVG(size_in_bytes) asSize_in_bytes_avg fromRedis_hotWHERE`Key` like 'user_verify_code_%'UNIONSELECT`Database', type, 'Key', Encoding,Count(`Database`) asRecord_count,SUM(size_in_bytes) asSize_in_bytes_sum,AVG(size_in_bytes) asSize_in_bytes_avg fromRedis_hotWHERE`Key` like 'robot_id_list%'UNIONSELECT`Database', type, 'Key', Encoding,Count(`Database`) asRecord_count,SUM(size_in_bytes) asSize_in_bytes_sum,AVG(size_in_bytes) asSize_in_bytes_avg fromRedis_hotWHERE`Key` like 'user_last_3_day_duration:%'/*Here you can continue to use the Union and check other names of key specific analysis data, the following line of code is generated redis_key_storage.csv file, if commented out, you can directly in the Navicat query interface to see the results of the query*/ intoOutFile'E:/redis_key_storage.csv'Fields terminated by ','Optionally enclosed by '"'Lines terminated by '\ r \ n'; # where you can modify the specific file save address (here File Save Address:'E:/redis_key_storage.csv'）

The resulting data are as follows:

OK, this is the end of the introduction, I hope to see this article of the students have some help ~

References:

1. Using code to complete the CSV file import mysql (71436108)

2. Use the Rdbtools tool to parse Redis dump.rdb files and memory usage (http://www.ywnds.com/?p=8441)

3.MySQL must know: Combination query (Union) (1190000007926959)

Simple analysis of Redis cache consumption memory data based on Python project (with detailed procedure)