Using Gearman to implement Redis cache MySQL

Environment:

centos6.5

mysql5.6

Gearman Introduction:

Gearman is a distributed task-distribution framework. The design is simple and has been widely supported. A typical Gearman application consists of the following sections:

• 
  

  • Gearman Job Server:gearman Core program, running in daemon form in the background

  • Gearman Client: Can be understood as the task of the recipient, such as I want to perform a mail in the background to send a task, you can call a Gearman Client in the program and incoming message information, and then you can immediately display the results of the execution to the user, and the task itself will slowly run in the background.

  • Gearman worker: The real performer of the task, generally need to write their own specific logic and run through the daemon process, Gearman Worker receives Gearman client to pass the task content, will be processed sequentially.

Design ideas:

First, the MySQL UDF was used (the combination of Lib_mysqludf_json and gearman-mysql-udf to achieve When data in MySQL is changed, the trigger triggers the data into Gearman, which is the equivalent of Gearman clinet. Then run your own PHP program as a worker, the data in the Gearman to Redis, when the Redis equivalent is the Gearman consumer.

1, installation Gearman

The system Yum source in the experiment, based on centos6.5 's own network of Yum sources, adds the EPEL source, EPEL (Extra Packages for Enterprise Linux, an additional software package for Business Edition Linux) is a software warehouse project maintained by the Fedora team that provides Rhel/centos with packages that they do not provide by default. The use of this source can save a lot of trouble, eliminate the trouble of source code compilation, it should be noted that whether the use of CentOS's own network yum source or Epel extension source, all need your IP to access to the public network.

Install Gearman, PHP, PHP gearman extensions, NC Tools

Yum install-y php-pecl-gearman Libgearman libgearman-devel gearmand NC

Start the Gearman service

/etc/init.d/gearmand start

Verify that the Gearman started successfully, and that if there are 4730 ports in the returned results, the service has started correctly

[root@hadoop1 ~]# netstat -alnutp |grep gearman

2, the operating principle of analog Gearman:

Use the following command to view the queue for Gearman

Watch-n 1 "(Echo status; Sleep 0.1) | NC 127.0.0.1 4730 "

The results are as follows

Writelog 0 0 0

Four column meaning: 1-task name; 2-Number of waiting queue tasks; 3-Number of tasks in operation; 4-number of worker processes running;

Compile a PHP code simulation gearman Client:client.php

<?php
     $client = new GearmanClient();
     $client->addServer();
     $client->doBackground(‘writeLog‘, ‘Log content‘);
     Echo ‘the file is already operating in the background ‘;
     Echo "\n";

Executive client.php

PHP client.php

At this point, look again at the Gearman queue and discover that there is a task in the wait queue

Writelog 1 0 0

Write a PHP code simulation gearman Worker:worker.php

The worker's role is to write the string ' Log content ' passed to Gearman by the client to the Gearman.log file in the current directory

<?php
    $worker = new GearmanWorker();
    $worker->addServer();
    $worker->addFunction(‘writeLog‘, ‘writeLog‘);
    while($worker->work());

    function writeLog($job)
    {
        $log = $job->workload();
        file_put_contents(__DIR__ . ‘/gearman.log‘, $log . "\n", FILE_APPEND | LOCK_EX);
    }

Start the nohup in the background worker.php

Nohup PHP worker.php &

Look at the Gearman queue again and find that the waiting task becomes the 0,worker process 1,gearman.log has the content

Writelog 0 0 1

[root@hadoop1 ~]# cat gearman.log 
Log content

3, install Mysql-server, MySQL, php-mysql (php connection to MySQL driver, not necessary, this is to later use the program to compare the efficiency of reading data from Redis and MySQL separately). In the experiment, the mysql5.6 was installed on my machine, so I did the experiment directly using mysql5.6. Of course, you can also use Yum to install MySQL, the MySQL version may be installed is not 5.6, but it is completely non-related.

Installing MySQL-related software

Yum install-y mysql-server MySQL Php-mysql

Start MySQL

/etc/init.d/mysql start

4, installation Lib_mysqludf_json

wget  https://github.com/mysqludf/lib_mysqludf_json/archive/master.zip
mv master master.zip
unzip master.zip
cd lib_mysqludf_json-master
rm -rf lib_mysqludf_json.so 
gcc $(mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c

At this time recompile generated lib_mysqludf_json.so, and then need to copy the lib_mysqludf_json.so to the MySQL plug-in directory, view the MySQL plug-in directory:

wget  https://github.com/mysqludf/lib_mysqludf_json/archive/master.zip
mv master master.zip
unzip master.zip
cd lib_mysqludf_json-master
rm -rf lib_mysqludf_json.so 
gcc $(mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c

[root@hadoop1 lib_mysqludf_json-master]# cp lib_mysqludf_json.so /usr/lib64/mysql/plugin/

5, installation gearman-mysql-udf

wget https://launchpad.net/gearman-mysql-udf/trunk/0.6/+download/gearman-mysql-udf-0.6.tar.gz   
tar xf gearman-mysql-udf-0.6.tar.gz -C ./
cd gearman-mysql-udf-0.6
./configure --with-mysql=/usr/bin/mysql_config --libdir=/usr/lib64/mysql/plugin/
make && make install

The plugin is installed directly into the MySQL plugin directory.

6. Connect to MySQL, create corresponding function, trigger and set Gearman server information

[root@hadoop1 ~]# mysql -u root -p******
Mysql> CREATE FUNCTION json_object RETURNS STRING SONAME ‘lib_mysqludf_json.so‘;
Mysql> CREATE FUNCTION gman_do_background RETURNS STRING SONAME ‘libgearman_mysql_udf.so‘;
Mysql> CREATE FUNCTION gman_servers_set RETURNS STRING SONAME ‘libgearman_mysql_udf.so‘

## Creating a trigger for the test table in javashop
Mysql> use javashop;
Mysql> describe test;
+-------+----------+------+-----+---------+------- +
Field | Type | Null | Key | Default | Extra |
+-------+----------+------+-----+---------+------- +
| id | int(11) | YES | | NULL | |
| name | char(20) | YES | | NULL | |
+-------+----------+------+-----+---------+------- +
2 rows in set (0.00 sec)
Mysql> source trigger.sql

## Setting gearman server information
Mysql>SELECT gman_servers_set(‘127.0.0.1:4730‘);

The contents of the Trigger.sql script file are as follows

[root@hadoop1 ~]# mysql -u root -p******
Mysql> CREATE FUNCTION json_object RETURNS STRING SONAME ‘lib_mysqludf_json.so‘;
Mysql> CREATE FUNCTION gman_do_background RETURNS STRING SONAME ‘libgearman_mysql_udf.so‘;
Mysql> CREATE FUNCTION gman_servers_set RETURNS STRING SONAME ‘libgearman_mysql_udf.so‘

## Creating a trigger for the test table in javashop
Mysql> use javashop;
Mysql> describe test;
+-------+----------+------+-----+---------+------- +
Field | Type | Null | Key | Default | Extra |
+-------+----------+------+-----+---------+------- +
| id | int(11) | YES | | NULL | |
| name | char(20) | YES | | NULL | |
+-------+----------+------+-----+---------+------- +
2 rows in set (0.00 sec)
Mysql> source trigger.sql

## Setting gearman server information
Mysql>SELECT gman_servers_set(‘127.0.0.1:4730‘);

7. Update a data in MySQL and then view the Gearman queue

mysql> update test set name=‘redis‘ where id=10;
Query OK, 1 row affected (0.03 sec)
Rows matched: 1  Changed: 1  Warnings: 0

writeLog        0       0       1
syncToRedis     1       0       0

It is visible that the MySQL trigger has successfully passed the data into the Gearman.

8, install Redis, Php-pecl-redis (php connected to Redis driver)

Yum Install Php-pecl-redis redis-y

Start Redis

/etc/init.d/redis start

Log in to the Redis client to view the current in-memory data

[root@hadoop1 ~]# redis-cli 
redis 127.0.0.1:6379> keys *
(empty list or set)

At this point, there is no data in Redis.

9. Write a worker program that is responsible for passing data from Gearman to Redis: redis_worker.php

<?php
    $worker = new GearmanWorker();
    $worker->addServer();
    $worker->addFunction(‘syncToRedis‘, ‘syncToRedis‘);

    $redis = new Redis();
    $redis->connect(‘127.0.0.1‘);

    while($worker->work());

    function syncToRedis($job)
    {
        global $redis;
        $workString = $job->workload();
        $work = json_decode($workString);
        if(!isset($work->id)){
                return false;
        }
        $redis->set($work->id, $workString);
    }

10. Run the nohup in the background redis_worker.php

Nohup PHP redis_worker.php &

At this point, look at the Gearman queue again

Writelog 0 0 1syncToRedis 0 0 1

The number of tasks waiting before discovering the Synctoredis task becomes 0, and the number of worker processes that are running becomes 1.

Check if data is cached in Redis

[root@hadoop1 ~]# redis-cli 
redis 127.0.0.1:6379> keys *
1) "10"
redis 127.0.0.1:6379> get 10
"{\"id\":10,\"name\":\"redis\"}"

Found that Redis has successfully cached the updated data in MySQL, and this feature is implemented.

11. Performance comparison of reading data from Redis and reading data from MySQL

Write PHP code to read data from Redis: php_redis.php

<?php
     $stime=microtime(true); //Get the time when the program starts executing

     $redis = new Redis();
     $redis->connect(‘127.0.0.1‘);
     Echo $redis->get(‘10‘);
     Echo "\n";
     $redis->close();

     $etime=microtime(true);//Get the time when the program ends

     $total=$etime-$stime; //calculate the difference
     Echo "$total". ‘second ‘;
     Echo "\n";
?>

     write PHP code to read data from MySQL: php_mysql.php

<?php
     $stime=microtime(true); //Get the time when the program starts executing
     $con = mysql_connect("127.0.0.1","root","******");
     $r2 = mysql_select_db("javashop");

     $result = mysql_query("SELECT * FROM test limit 1");
     While ($row = mysql_fetch_array($result)) {
         Echo $row[‘id‘] . " --> " . $row[‘name‘];
         Echo "\n";
     }
     Mysql_close($con);

     $etime=microtime(true);//Get the time when the program ends

     $total=$etime-$stime; //calculate the difference
     Echo "$total". ‘second ‘;
     Echo "\n"
?>

Run two PHP programs separately

[root@hadoop1 ~]# php php_redis.php 
{"id":10,"name":"redis"}
0.00059199333190918秒

[root@hadoop1 ~]# php php_mysql.php 
10 --> redis
0.0043718814849854秒

[root@hadoop1 ~]# bc <<< 0.0043718814849854/0.00059199333190918
7

By querying a single record, you can see that the length of time it takes to get data from MySQL is 7 times times the length of the data in Redis, and the visible performance increases by almost an order of magnitude.

Note the point:

The pro-test confirms that MySQL will lose information about the Gearman server that was set up before the reboot, and the workaround is as follows:

Create a Init_file.sql file in the DataDir directory of MySQL, setting the contents of Gearman server information

echo "Select Gman_servers_set (' 127.0.0.1:4730 ');" >/var/lib/mysql/init_file.sql

Then add the following in the [Mysqld] key of the MySQL configuration file

Init-file=/var/lib/mysql/init_file.sql

Then restart MySQL, update a record and try again!

Reference: Http://avnpc.com/pages/mysql-replication-to-redis-by-gearman

PS: Completed the first blog, I hope you have a lot of advice, happy life!

This article from "Don't force yourself don't know how awesome" blog, please be sure to keep this source http://quenlang.blog.51cto.com/4813803/1568567

Using Gearman to implement Redis cache MySQL