"Gearman Learning Notes" Persistent storage queues ___ Storage

1. Foreword

Here is the official website documents: http://gearman.org/manual/job_server/#persistent_queues, said is not very clear, so I made a summary.

Test environment:

Linux-gentoo #1 SMP x86_64 Intel (R) Xeon (r) CPU E5645 @ 2.40GHz Genuineintel gnu/linux

2. Why do I need to be persistent?

Find the answer from the official website:

Inside the Gearman job server, all job queues are stored in memory. This is means if a server restarts or crashes with pending jobs, they'll be lost and are never run by a worker. Persistent queues were added to allow background jobs to is stored in a external durable queue so they may live between s Erver restarts and crashes. The persistent queue is only enabled for background jobs because foreground jobs have a attached client. If a job server goes away, the client can detect this and restart the foreground job somewhere else (or a error BA CK to the original caller). Background jobs on the other hand have no attached client and are simply expected to is run when submitted.

Translation: Work queues in Gearman's job server are stored in memory, which means that once the server restarts or goes down while there are tasks waiting to be processed, the tasks are lost. Persistent storage queues allow you to add background tasks and store them in external persistence team column (such as the MySQL database), so there is no such thing as the above mentioned.

3.gearman Persistence configuration process

I use MySQL to do task persistence storage.

First, build the table:

Create DATABASE Gearman
   CREATE table ' Gearman_queue ' (
   ' unique_key ' varchar () not NULL,
   ' function_name ' varchar (255) NOT NULL,
   ' priority ' int (one) not NULL,
   ' data ' longblob not NULL,
   ' when_to_run ' int, PRIMARY key< c7/> (' Unique_key ')
)

Note: Gearman support for the persistence of expansion has drizzle, MySQL, memcache, Postgre, SQLite and so on

(I posted it directly from the Gearman source):

 Case ${persistent:-none} in drizzle); MySQL) gearmand_params= "${gearmand_params}-Q mysql" [${persistent_host}] && Gearmand_para Ms= "${gearmand_params}--mysql-host=${persistent_host}" [${persistent_user}] && gearmand_params= "$ {gearmand_params}--mysql-user=${persistent_user} "[${persistent_pass}] && gearmand_params=" ${GEAR Mand_params}--mysql-password=${persistent_pass} "[${persistent_db}] && gearmand_params=" ${GEARM And_params}--mysql-db=${persistent_db} "[${persistent_table}] && gearmand_params=" ${gearmand_param S}--mysql-table=${persistent_table} "[${persistent_port}] && gearmand_params=" ${gearmand_params}
      --mysql-port=${persistent_port} ";; memcache) [${persistent_serverlist}] && gearmand_params= "${gearmand_params}-Q libmemcached--libmemcach Ed-servers=${persistent_serverlist}"
        ;; Postgre) gearmand_params= "${gearmand_params}-Q Libpq" [${persistent_host}] && Gearmand_pa Rams= "${gearmand_params}--libpq-host=${persistent_host}" [${persistent_user}] && gearmand_params= ' ${gearmand_params}--libpq-user=${persistent_user} ' [${persistent_pass}] && gearmand_params= ' ${GE Armand_params}--libpq-password=${persistent_pass} "[${persistent_db}] && gearmand_params=" ${GEA Rmand_params}--libpq-dbname=${persistent_db} "[${persistent_port}] && gearmand_params=" ${GEARMAND _params}--libpq-port=${persistent_port} "[${persistent_table}] && Ewarn" LIBPQ doesn ' t recognise '
            Table ' parameter. ' [${persistent_socket}] && Ewarn "LIBPQ doesn ' t recognise ' SOCKET ' parameter.
      If No is set, the it automatically falls back to a socket. ";; tokyocabinet) gearmand_params= "${gearmand_parAMS}-Q libtokyocabinet--libtokyocabinet-file=${persistent_file} ";;
          SQLite) gearmand_params= "${gearmand_params}-Q Libsqlite3--libsqlite3-db=${persistent_file}";;
      none);;
        *) eerror "wrong persistent queue store setting In/etc/conf.d/gearmand."
    return 1;;
 Esac

In general, Gearman's configuration file is placed in the/etc/init.d/directory, I put it in the/etc/conf.d/gearmand.

Configuration file: (note here persistent= "MySQL")

#/etc/conf.d/gearmand:config File For/etc/init.d/gearmand # Persistent Queue Store # The following queue stores are AV Ailable: # drizzle|memcache|mysql|postgre|sqlite|tokyocabinet|none # If You don't wish to use persistent queues, leave th
are option commented out.
# that persistent queue mechanisms are mutally exclusive. persistent= "MySQL" # persistent queue settings for drizzle, mysql and postgre #PERSISTENT_SOCKET = "persistent_host=" Loca Lhost "persistent_port=" 3306 "persistent_user=" Gearman "persistent_pass=" Your-pass-word-here "PERSISTENT_DB=" Gearman "persistent_table=" gearman_queue "# Persistent queue settings for SQLite #PERSISTENT_FILE =" "# Persistent queue Settings for memcache #PERSISTENT_SERVERLIST = "" # General Settings #-j,--job-retries=retries number of attempts to Run the job before the job # server removes it. Thisis helpful to ensure a bad # job does not crash all available workers.     Default #                        is no limit. #-L,--listen=address address the server should listen on.
The Default is # Inaddr_any. #-P,--port=port port the server should listen on.
default=4730.
#-R,--protocol=protocol Load protocol module. #-T,--threads=threads number of I/O threads to use.
Default=0.
#-V,--verbose increase verbosity level by one.
#-W,--worker-wakeup=workers number of workers to wakeup for each job received.
# The default is to wakeup all available workers.
 Gearmand_params= "-L 127.0.0.1--verbose=debug"

Then, in Gearman's startup shell, look at the source code:

Ebegin "Starting ${svcname}"
        Start-stop-daemon--pidfile/var/run/gearmand/gearmand.pid--start \
        --exec/usr /sbin/gearmand----pid-file=/var/run/gearmand/gearmand.pid \
        --user=gearmand--daemon \
        --log-file=/var/ Log/gearmand/gearmand.log ${gearmand_params}
 eend $?

After starting Gearman, view the process:

PS aux | grep Gearman

You can see that the Gearman and MySQL database establish a connection statement:

Gearmand 16746  0.0  0.3 423380 2176  ?        SSL  02:54   0:00/usr/sbin/gearmand--pid-file=/var/run/gearmand/gearmand.pid--user=gearmand--daemon-- Log-file=/var/log/gearmand/gearmand.log-l 127.0.0.1--verbose=debug-q MySQL--mysql-host=localhost--mysql-user= Gearman--mysql-password=djw55rbdvazpmlun--mysql-db=gearman--mysql-table=queue--mysql-port=3306

4. For reference

1.http://gearman.org/manual/job_server/#persistent_queues

2.http://www.jaceju.net/blog/archives/1211/

3.http://blog.sina.com.cn/s/blog_54ef398901018l84.html