How to install TokuDB and TokuDB

How to install TokuDB and TokuDB

TokuDB is a storage engine developed by Tokutek Based on the ft-index (Fractal Tree Index) Key-value pairs.

It uses indexes to accelerate query speed and has high scalability. It supports hot scheme modification and has the following features:

1. Fast insertion performance: 20 ~ 80 times;

2. Reduce storage space by compressing data;

3. The data volume can be expanded to several TB;

4. No index fragmentation will be generated;

5. Support for hot column addition, hot indexing, and mvcc

 

Applicable scenarios:

1. If you want to store blob, do not use TokuDB because it limits the record size to not be too large;

2. If you have over million records, use TokuDB;

3. If you focus on the performance of update, do not use TokuDB, it is not as fast as Innodb;

4. If you want to store old records, use TokuDB;

5. If you want to reduce the storage space occupied by data, use TokuDB;

 

For the TokuDB and InnoDB performance pressure test reports, refer:

Https://www.percona.com/blog/2016/02/01/innodb-and-tokudb-on-aws/

 

The following installation is based on Percona 5.6.31-77.0:

 

Install the libjemalloc Library

This library is available in EPEL

# Wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm

# Rpm-ivh epel-release-latest-6.noarch.rpm

# Yum install jemalloc-y

 

After yum is installed, the generated library file is/usr/lib64/libjemalloc. so.1.

[root@localhost ~]# rpm -qa |grep jemallocjemalloc-3.6.0-1.el6.x86_64[root@localhost ~]# rpm -ql jemalloc-3.6.0-1.el6.x86_64/usr/bin/jemalloc.sh/usr/lib64/libjemalloc.so.1/usr/share/doc/jemalloc-3.6.0/usr/share/doc/jemalloc-3.6.0/COPYING/usr/share/doc/jemalloc-3.6.0/README/usr/share/doc/jemalloc-3.6.0/VERSION/usr/share/doc/jemalloc-3.6.0/jemalloc.html

 

Modify the configuration file my. cnf

Set the malloc-lib variable in [mysqld_safe]

malloc-lib=/usr/lib64/libjemalloc.so.1

Restart the mysqld_safe Process

The following information is output during startup:

160810 20:11:46 mysqld_safe Adding '/usr/lib64/libjemalloc.so.1' to LD_PRELOAD for mysqld

 

If this library is not installed, the following error is reported when the TokuDB plugin is loaded:

mysql> INSTALL PLUGIN tokudb SONAME 'ha_tokudb.so';2016-08-10 16:25:02 1861 [ERROR] TokuDB is not initialized because jemalloc is not loaded2016-08-10 16:25:02 1861 [ERROR] Plugin 'TokuDB' init function returned error.2016-08-10 16:25:02 1861 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.2016-08-10 16:25:02 1861 [Note] Shutting down plugin 'TokuDB'ERROR 1123 (HY000): Can't initialize function 'tokudb'; Plugin initialization function failed.

 

Disable Transparent huge pages

Check whether Transparent huge pages is enabled for the current Kernel

# cat /sys/kernel/mm/transparent_hugepage/enabled[always] madvise never

Always indicates Enabled

 

How can I disable it?

Echo never>/sys/kernel/mm/transparent_hugepage/enabled
Echo never>/sys/kernel/mm/transparent_hugepage/defrag

 

If the plug-in is not disabled, an error will be reported during subsequent Agent installation:

mysql> INSTALL PLUGIN tokudb SONAME 'ha_tokudb.so';ERROR 1123 (HY000): Can't initialize function 'tokudb'; Plugin initialization function failed.

In addition, the following information is printed in the error log:

2016-08-10 20:47:51 4764 [ERROR] TokuDB: Huge pages are enabled, disable them before continuing2016-08-10 20:47:51 4764 [ERROR] ************************************************************2016-08-10 20:47:51 4764 [ERROR]                                                             2016-08-10 20:47:51 4764 [ERROR]                         @@@@@@@@@@@                         2016-08-10 20:47:51 4764 [ERROR]                       @@'         '@@                       2016-08-10 20:47:51 4764 [ERROR]                      @@    _     _  @@                      2016-08-10 20:47:51 4764 [ERROR]                      |    (.)   (.)  |                      2016-08-10 20:47:51 4764 [ERROR]                      |    (.)   (.)  |                      2016-08-10 20:47:51 4764 [ERROR]                      |             ` |                      2016-08-10 20:47:51 4764 [ERROR]                      |        >    ' |                      2016-08-10 20:47:51 4764 [ERROR]                      |     .----.    |                      2016-08-10 20:47:51 4764 [ERROR]                      ..   |.----.|  ..                      2016-08-10 20:47:51 4764 [ERROR]                       ..  '      ' ..                       2016-08-10 20:47:51 4764 [ERROR]                         .._______,.                         2016-08-10 20:47:51 4764 [ERROR]                                                             2016-08-10 20:47:51 4764 [ERROR] TokuDB will not run with transparent huge pages enabled.        2016-08-10 20:47:51 4764 [ERROR] Please disable them to continue.                            2016-08-10 20:47:51 4764 [ERROR] (echo never > /sys/kernel/mm/transparent_hugepage/enabled)  2016-08-10 20:47:51 4764 [ERROR]                                                             2016-08-10 20:47:51 4764 [ERROR] ************************************************************2016-08-10 20:47:51 4764 [ERROR] Plugin 'TokuDB' init function returned error.2016-08-10 20:47:51 4764 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.2016-08-10 20:47:51 4764 [Note] Shutting down plugin 'TokuDB'

 

Enable TokuDB

Percona is 5.6.17-66.0 to introduce TokuDB plug-in to its own binary version. It can be used as a GA version starting from 5.6.19-67.0. From 5.6.22 to 72.0, a script ps_tokudb_admin was developed to simplify TokuDB installation.

 

First, let's see how TokuDB is manually installed.

The plug-in is mainly loaded. The TokuDB plug-in name is ha_tokudb.so, which is put under plugin_dir by default.

Log on to the mysql client and run the following command:

INSTALL PLUGIN tokudb SONAME 'ha_tokudb.so';INSTALL PLUGIN tokudb_file_map SONAME 'ha_tokudb.so';INSTALL PLUGIN tokudb_fractal_tree_info SONAME 'ha_tokudb.so';INSTALL PLUGIN tokudb_fractal_tree_block_map SONAME 'ha_tokudb.so';INSTALL PLUGIN tokudb_trx SONAME 'ha_tokudb.so';INSTALL PLUGIN tokudb_locks SONAME 'ha_tokudb.so';INSTALL PLUGIN tokudb_lock_waits SONAME 'ha_tokudb.so';INSTALL PLUGIN tokudb_background_job_status SONAME 'ha_tokudb.so';

 

Check whether the plug-in is successfully loaded.

Mysql> show engines;

| TokuDB             | YES     | Percona TokuDB Storage Engine with Fractal Tree(tm) Technology             | YES          | YES  | YES        |

Mysql> show plugins;

| TokuDB                        | ACTIVE   | STORAGE ENGINE     | ha_tokudb.so | GPL     || TokuDB_file_map               | ACTIVE   | INFORMATION SCHEMA | ha_tokudb.so | GPL     || TokuDB_fractal_tree_info      | ACTIVE   | INFORMATION SCHEMA | ha_tokudb.so | GPL     || TokuDB_fractal_tree_block_map | ACTIVE   | INFORMATION SCHEMA | ha_tokudb.so | GPL     || TokuDB_trx                    | ACTIVE   | INFORMATION SCHEMA | ha_tokudb.so | GPL     || TokuDB_locks                  | ACTIVE   | INFORMATION SCHEMA | ha_tokudb.so | GPL     || TokuDB_lock_waits             | ACTIVE   | INFORMATION SCHEMA | ha_tokudb.so | GPL     || TokuDB_background_job_status  | ACTIVE   | INFORMATION SCHEMA | ha_tokudb.so | GPL     |

 

Create a table and test it.

mysql> create table test.test(id int) engine=tokudb;Query OK, 0 rows affected (0.23 sec)

So far, TokuDB manual installation is complete ~

 

Script Installation

In fact, Percona provides a script in version 5.6.22-72.0 to automatically install the TokuDB plug-in. The operations involved include disabling transparent large pages and loading the plug-in.

The script is in the bin directory of the Binary Package.

[root@localhost bin]# ./ps_tokudb_admin --helpThis script is used for installing and uninstalling TokuDB plugin for Percona Server 5.6.It can also be used to install or uninstall the Percona TokuBackup plugin (requires mysql server restart).If transparent huge pages are enabled on the system it adds thp-setting=never option to my.cnfto disable it on runtime.Valid options are:  --user=user_name, -u user_name     mysql admin username  --password[=password], -p[password]     mysql admin password (on empty will prompt to enter)  --socket=path, -S path         the socket file to use for connection  --host=host_name, -h host_name     connect to given host  --port=port_num, -P port_num         port number to use for connection  --defaults-file=file          specify defaults file instead of guessing  --enable, -e                 enable TokuDB plugin and disable transparent huge pages in my.cnf  --enable-backup, -b             enable Percona TokuBackup and add preload-hotbackup option to my.cnf                     (this option includes --enable option)  --disable, d                 disable TokuDB plugin and remove thp-setting=never option in my.cnf                     (this option includes --disable-backup option)  --disable-backup, r             disable Percona TokuBackup and remove preload-hotbackup option in my.cnf  --help                 show this helpFor TokuDB requirements and manual steps for installation please visit this webpage:http://www.percona.com/doc/percona-server/5.6/tokudb/tokudb_installation.html

The script is written in shell and the parameters are relatively simple. -- enable-backup also includes installing the TokuDB online Hot Standby plug-in.

 

The script installation process is as follows:

 ./ps_tokudb_admin --enable --host=127.0.0.1 --port=3307 --user=root -pEnter password:Continuing without password...Checking SELinux status...INFO: SELinux is disabled.Checking if Percona Server is running with jemalloc enabled...INFO: Percona Server is running with jemalloc enabled.Checking transparent huge pages status on the system...INFO: Transparent huge pages are currently disabled on the system.Checking if thp-setting=never option is already set in config file...INFO: Option thp-setting=never is set in the config file.Checking TokuDB engine plugin status...INFO: TokuDB engine plugin is installed.

 

Summary

The latest version of TokuDB is 7.5.7. It is actually included in the binary release of Percona as a plug-in. However, this plug-in cannot be directly used in the MySQL Community edition.

TokuDB is based on MySQL connector and TokuDB 7.1.0.

However, this is also true. The TokuDB plug-in cannot be directly used in Community versions of MySQL 5.6 and 5.7.

Later versions of MySQL and TokuDB can only be used.

The description in the official document is as follows:

Please note that Tokutek made some changes to the MySQL source that are required to either fix bugs or aid in performance, so if you are compiling from source you must use the Tokutek version of MySQL that is based on the MySQL 5.5.30 source.

 

Reference

1. https://www.percona.com/blog/2013/07/02/tokumx-fractal-treer-indexes-what-are-they/

2. http://dbaplus.cn/news-21-418-1.html

3. https://www.izhangheng.com/highly-scalable-storage-engine-tokudb