Erlang Pool Management-Emysql pool optimize

In the last mention of an analysis of Emysql pool (http://www.cnblogs.com/--00/p/4281938.html)

Now the Emysql_conn_mgr gen_server process is a single point, that is, all pool management scheduling is done by a process.

If you manage a large number of pool in the same Erlang node, there is a bottleneck. For a hotspot process, improving its process priority is a optimize direction, but it does not completely solve the problem caused by single point. Therefore, you should attempt to split a single emysql_conn_mgr gen_server process into multiple, while the split is based on the management of the pool optimize to a emysql_conn_mgr process for each pool. Instead of all the pool now is managed by the same emysql_conn_mgr.

Pool Add action

For the data structure of the pool, keep the previous structure intact. Use a emysql_pool_mgr gen_server process and maintain an ETS table in it (to hold the poolid and emysql_conn_mgr process IDs). At each add The pool operation Start_child an emysql_conn_mgr anonymous process and associates it with the pool ID, which is written to the ETS table maintained by the emysql_pool_mgr process.

1Add_pool (#pool {pool_id=poolid,size=size,user=user,password=password,host=host,port=Port,2database=database,encoding=encoding,start_cmds=Startcmds,3Connect_timeout=connecttimeout,warnings=warnings}=poolsettings)4 CONFIG_OK (poolsettings),5      CaseEmysql_pool_mgr:has_pool (Poolid) of6         true-7 {error,pool_already_exists};8         false-9Pool =#pool {Tenpool_id =Poolid, OneSize =Size, Auser =User, -Password =Password, -Host =Host, thePort =Port, -Database =Database, -encoding =Encoding, -Start_cmds =Startcmds, +Connect_timeout =ConnectTimeout, -Warnings =Warnings +                     }, APool2 = CaseEmysql_conn:open_connections (Pool) of at{OK, Pool1}Pool1; -{Error, Reason}throw (Reason) -             End, -{OK, poolserver} =Emysql_pool_mgr:add_pool (Poolid, Pool2), - [Gen_tcp:controlling_process (Conn#emysql_connection.socket, Poolserver) -|| Conn <-queue:to_list (pool2#pool.available)], in OK -     End.

Start_child emysql_conn_mgr The anonymous process, and the operations associated with the EMYSQL_CONN_MGR process ID Poolid are implemented in the EMYSQL_POOL_MGR:ADD_POOL/2 function (L26).

Emysql_pool_mgr Module

The Emysql_pool_mgr module is a gen_server process that maintains an ETS table for storing {poolid, emysqlconnmgrprocessid} information. Because each execute SQL statement requires a pool operation, which is required to obtain the corresponding EMYSQLCONNMGRPROCESSID, if the associated information is stored in the emysql_pool_mgr process, emysql_pool_mgr The process also becomes a single point, so the ETS table is used to share the pressure burden of the emysql_pool_mgr process.

The Emysql_pool_mgr module provides several APIs:

1, HAS_POOL/1

Used to determine if the pool exists in the current system and the input parameter is Poolid

2, ADD_POOL/2

Used to start_child the emysql_conn_mgr process and write {poolid, emysqlconnmgrprocessid} information to ETS table with input parameters poolid and pool structure

3, REMOVE_POOL/1

Delete pool, and stop emysql_conn_mgr process, input parameter is Poolid

4, GET_POOL_SERVER/1

Obtain the corresponding Emysqlconnmgrprocessid according to Poolid, the input parameter is Poolid

5, pools/0

All poolid and their corresponding emysqlconnmgrprocessid in the current system

6, CONNS_FOR_POOL/1

All links in the corresponding pool are obtained according to Poolid, and the input parameter is Poolid

Add_pool/2

The ADD_POOL/2 function primarily calls the SUPERVISOR:START_CHILD/2 function to add emysql_conn_mgr gen_server processes under the EMYSQL_CONN_POOL_SUP monitoring tree.

Code snippet for Emysql_conn_pool_sup module:

1 start_link (name, Module)-2     supervisor:start_link ({local, name},?  module, module). 3 4 Init (Module),5    {OK,6      {simple_one_for_one, 1}, 7       [{undefined, {Module, Start_link, []}, temporary,8         Brutal_kill, worker, [module]}]} }.

A fairly common way to use it.

Emyslq_conn_mgr Module

After optimize, the code of the previous Emysql_conn_mgr module must make some simple adjustments, and the pool information needs to be added to the process state information when calling the Start_link function init initialization.

1 init ([Pool])--2    erlang:process_flag (priority, high),3     {OK, # State{pools = [Pool]}}.

This improves the priority of the emysql_conn_mgr process (L2).

For calls to other functions, you need to add a emysqlconnmgrprocessid parameter so that when GEN_SERVER:CALL/2, it will "? MODULE "changed to Emysqlconnmgrprocessid.

Pool Use

The code also needs to make some adjustments when calling the Execute function to execute an SQL statement:

1  when OrElse AndAlso AndAlso OrElse Timeout = = infinity),2     poolserver = emysql_pool_mgr:get_pool_server (poolid),3     Connection = emysql_conn_mgr:wait_for_connection (poolserver, Poolid),4     monitor_ Work (Poolserver, Connection, Timeout, [Connection, Query, Args]);

That is, before calling the function of the Emysql_conn_mgr module, you need to get Emysqlconnmgrprocessid (L2) based on Poolid and Emysqlconnmgrprocessid as Emysql_ The first parameter of the Conn_mgr module function is called (L3).

Initialize_pools

In today's Emysql project, there is a function to initialize the pools. This means that the configured pool in the config file can be loaded automatically when the app is start. Now it's time to call the relevant function when the EMYSQL_CONN_MGR process is init, and add the result to the process state.

After the optimize structure, emysql_conn_mgr can only be start_link when manually adding a pool, and cannot perform initialize_pools operations related to the function.

Now adjust to emysql_pool_mgr init, Emysql_pool_mgr sends {initialize_pools} message to self, which is processed in the Handle_info callback function:

1Handle_info ({initialize_pools}, state),2     %%if the Emysql application values is not present in the Config3     %%file We'll initialize and empty set of pools. Otherwise, the4     %%values defined in the config is used to initialize. 5Initializespools =6         [      7 {poolid, #pool {8pool_id =Poolid,9Size = Proplists:get_value (Size, Props, 1),        Tenuser =proplists:get_value (user, Props), OnePassword =proplists:get_value (password, Props), AHost =Proplists:get_value (host, Props), -Port =Proplists:get_value (port, Props), -Database =proplists:get_value (Database, Props), theencoding =proplists:get_value (encoding, Props), -Start_cmds =Proplists:get_value (Start_cmds, Props, []) -}} || {poolid, Props} <-Emysql_app:pools () -         ], +[begin -          CaseEmysql_conn:open_connections (Pool) of +{OK, Pool1} A Emysql_pool_mgr:add_pool (Poolid, Pool1); at{Error, Reason} - Erlang:throw (Reason) -         End -     End|| {poolid, Pool} <-Initializespools], -{noreply, State,? Hibernate_timeout};

Can not be optimize after, directly remove some functions of function ah. :)

Summarize

On the whole, each pool uses a corresponding emysql_conn_mgr process as the management. To avoid the pressure of multi-pool management to the emysql_conn_mgr process as much as possible.

The newly introduced Emysql_pool_mgr gen_server process is used to maintain poolid and emysqlconnmgrprocessid associated information. At the same time, in order to avoid emysql_pool_mgr single point problem, use ETS table to share the burden of emysql_pool_mgr gen_server process.

Optimize branch for Add_pool_mgr, not yet PR (wait for the test feedback from the small partner, welcome code review).

Erlang Pool Management-Emysql pool optimize