Go Language Practical Notes (16) | Go Concurrency Example-pool

This is a creation in Article, where the information may have evolved or changed.

"Go language Combat" reading notes, not to be continued, welcome to sweep code attention flysnow_org to the public or website http://www.flysnow.org/, the first time to see follow-up notes. If you feel helpful, share it with your friends and thank you for your support.

This article demonstrates the use of buffered channels to implement a resource pool, which can manage resources shared among any number of goroutine, such as network connections, database connections, etc., when we operate in the database, the more common is the data connection pool, can also be based on our implementation of the resource pool.

As you can see, the resource pool is also a very fluid pattern, which generally applies to sharing resources between multiple goroutine, each goroutine can request resources from the resource pool, and then put it back in the resource pool for other goroutine reuse.

Well, the usual, let's build a resource pool structure, and then give some methods that will help us manage resources.

1234567

//A secure resource pool, the managed resources must all implement IO. The Close interface typestructchanfunc()(IO). Closer,error)closedbool}

This struct Pool has four fields, which m is a mutex, which is primarily used to ensure that the values in the pool are secure when multiple Goroutine access the resource.

resA field is a buffered channel that is used to hold a shared resource, the size of which is specified at initialization Pool time. Note that the type of this channel is an io.Closer interface, so the type that implements this io.Closer interface can be used as a resource to our resource pool management.

factoryThis is a function type, its function is that when a new resource is needed, it can be created by this function, that is, it is to generate new resources, as to how to generate and generate what resources, is determined by the user, so this is the resource pool flexible design place.

closedThe field indicates whether the resource pool is closed and if it is closed, there will be an error in the access.

Now that this resource pool we have defined, and know the meaning of each field, the following is specific to use when open. Just now we're talking about closing errors, so we'll first define a resource pool that has been shut down.

1	var errpoolclosed = errors. New ("resource pool has been closed. ")

Very concise, when we get resources from the resource pool, if the resource pool is closed, this error will be returned. The purpose of defining it individually is to differentiate it from other errors, so that we can differentiate it from a multitude of types when needed error ErrPoolClosed .

So we're going to Pool set up a function specifically for the creation, which is the factory function, which we named New .

12345678910

//create a resource pool func Span class= "title" >new (fn func () (IO. Closer, error) , size uint ) (* Pool, error) {if size <= 0 {return nil , errors. The value of New ()}return &pool{factory:fn,res: make (chan io. Closer, size),}, nil }

This function creates a resource pool that receives two parameters, one that fn creates a new resource, and one size that specifies the size of the resource pool.

In this function, the size size of the judgment, at least it can not be less than or equal to 0, otherwise it will return an error. If the parameters are normal, the size resource is saved by creating a buffered channel, and a pointer to the resource pool is returned.

With a well-created pool of resources, we can get resources from it.

 //get a resource from a resource pool  func   acquire   ()  Span class= "params" > (IO. Closer,error)   {select  {case  r,ok: = <- P.res:log. Println ( "Acquire: Shared resource" ) if !ok { return  nil , errpoolclosed}return  r, nil  default : Log. Println ( "Acquire: New Build Resource" ) return  p.factory ()}} 

1234567891011121314

AcquireMethod can fetch a resource from a resource pool, and if there is no resource, the calling factory method generates one and returns it.

Multiplexing is also used here select , as this function cannot be blocked, can be obtained, and cannot generate one.

The new knowledge here is that the channel receives a multi-parameter return, if it can be received, the first parameter is the value received, the second indicates whether the channel is closed. In the example ok , if the value is to indicate that the false channel is closed, true then the channel is normal. So here we make a judgment, if the channel is closed, return the channel close error.

There is a way to get the resources, there must be a corresponding method of releasing the resources, because after the resource is exhausted, the resource pool is returned for reuse. Before we explain how to dispose of resources, let's look at the method of closing the resource pool, because the method of freeing the resource will also be used.

Closing a resource pool means that the entire resource pool can no longer be used, then closes the channel that holds the resource and frees the resources in the channel.

12345678910111213141516171819

//Close resource pool, release resources func(P *pool)Close() {P.m.lock ()defer p.m.unlock ()if p.closed {returntrue//Close channel, do not write to Close (p.res)//Close the resource for r:=range p.res {r.close ()}} in the Channel

In this method, we use a mutex, because there is a field in which the tag resource pool is closed, which closed requires more than one goroutine operation, so we must ensure that the field is synchronized. Here, set the closing flag to true .

Then we close the channel, do not let the write, and we Acquire can also perceive the front of the channel is closed. After the channel, the resources in the channel begin to be freed because all the resources implement IO. Closer interface, so we call the Close method directly to release the resources.

The closing method has, let's see how the method of releasing resources is implemented.

12345678910111213141516171819 td>

func (P *pool) release {//guarantees that the operation and the action of the Close method are secure p.m.lock () defer p.m.unlock () //resource pool is closed, save this one not released resources, release can if p.closed {r.close () return }select { case p.res <-r:log. Println ( "resource released into the pool" ) default : Log. Println ( "resource pool full, release this resource" ) R.close ()}}

Releasing resources essentially sends the resources back to the buffer channel, which is simple, but for a more secure implementation of this method, we use mutexes to ensure closed the security of the flags, and this mutex has the benefit of not sending resources to a closed channel.

What is this for? Because the two methods of close and release are mutually exclusive, the change of the mark in the Close method closed can be sensed by the release method, so it will return directly without executing the following select Code, and will not send resources to a closed channel.

If the resource pool is not closed, continue to attempt to send resources to the resource channel, if it can be sent, it is equal to the resources back to the resource pool; If the resource pool is full and the resource cannot be returned to the resource pool, then we will shut down the resource that needs to be freed and discard it.

For this resource pool management step by step are implemented, and do a detailed explanation, the following look at the entire sample code, easy to understand.

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465 666768697071727374757677787980818283

Package Commonimport ("Errors" "io" "Sync" "log")//a secure resource pool, the managed resources must all implement IO. The close interface type Pool struct {m sync. Mutexres Chan io. Closerfactory func () (IO. Closer, error) closed Bool}var errpoolclosed = errors. New ("Resource pool has been closed. ")//Create a resource pool func New (fn func () (IO). Closer, error), size UINT (*pool, error) {if size <= 0 {return nil, errors. The value of New ("size" is too small.) ")}return &pool{factory:fn,res:make (chan io. Closer, size),}, nil}//gets a resource func (P *pool) acquire () from the resource pool (IO. Closer,error) {Select {case R,ok: = <-p.res:log. Println ("Acquire: Shared resources") if!ok {return Nil,errpoolclosed}return r,nildefault:log. Println ("Acquire: New Build Resource") return P.factory ()}}//close resource pool, free resource func (P *pool) close () {P.m.lock () defer p.m.unlock () if p.closed {return}p.closed = true//close channel, do not let write close (p.res)//Shut down channel resources for R:=range p.res {r.close ()}}func (P *pool) Release (R io. Closer) {//Ensure that the operation and the action of the Close method are secure P.m.lock () defer p.m.unlock ()//resource pool is closed, save this one not released resources, release can if p.closed {r.close () Return}select {case P.res <-r:log. PRINTLN ("The resources are released into the pool.") Default:log. Println ("Resource pool full, release this resource") R.close ()}}

"Go language Combat" reading notes, not to be continued, welcome to sweep code attention flysnow_org to the public or website http://www.flysnow.org/, the first time to see follow-up notes. If you feel helpful, share it with your friends and thank you for your support.

Well, resource pool management is written, and you know how the resource pool is implemented, now let's look at how to use this resource pool to emulate a database connection pool.

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465 66676869707172737475

PackageMainImport("Flysnow.org/hello/common""IO""Log""Math/rand""Sync""Sync/atomic""Time")Const(//simulation of the maximum GoroutineMaxgoroutine =5//size of the resource poolPoolres =2) func main() {//wait for task to completevarWG Sync. Waitgroupwg.add (Maxgoroutine) p, err: = Common. New (CreateConnection, Poolres)ifErr! =Nil{log. PRINTLN (ERR)return}//Simulate several goroutine use resource pools to query data at the same time forQuery: =0; Query < Maxgoroutine; query++ {Go func(q int) {dbquery (q, p) WG. Done ()} (query)}WG. Wait () log. Println ("Start closing resource pool") P.close ()}//Simulation database query func dbquery(query int, pool *common. Pool) {conn, err: = Pool. Acquire ()ifErr! =Nil{log. PRINTLN (ERR)return}deferPool. Release (conn)//Simulation QueryTime. Sleep (time. Duration (Rand. INTN ( +)) * time.millisecond) log. Printf ("%d queries, using a database connection with ID%d", query, Conn. (*dbconnection). ID)}//Database connectiontypeDbConnectionstruct{IDInt32//Connection flag}//Implement IO. Closer Interface func (db *dbconnection) Close() error {log. Println ("Close Connection", db.id)return Nil}varIdcounterInt32//methods for generating database connections for use by resource pools func createconnection() (IO. Closer, error) {//concurrency security, generating unique flags for database connectionsID: = Atomic. AddInt32 (&idcounter,1)return&dbconnection{id},Nil}

At this point we test the example of using a resource pool, first defining a struct dbConnection , which has only one field, which is used to make a unique tag. The dbConnection io.Closer interface is then implemented so that our resource pool can be used.

createConnectionThe function corresponds to a field in a resource pool that is factory used to create a database connection dbConnection , with a flag assigned to it.

Then we open at the same time 5 goroutine, simulate the concurrency of the database query dbQuery , Query method, first get the available database connection from the resource pool, and then release.

Here we will create 5 database connections, but we set the size of the resource pool is only 2, so after releasing 2 connections, the subsequent 3 connections will be released because the resource pool is full, and we will see the output of the print information can be seen.

Finally, after the resource connection pool is used, we will close the resource pool and use the resource pool Close method.

123456789101112131415161718192021

2017/04/17 22:25:08 acquire: New build resource 2017/04/17 22:25:08 acquire: New Build resource 2017/04/17 22:25:08 Acquire: New build resource 2017/04/17 22:25:08 acquire: New build resource 2017/04/17 22:25:08 acquire: New build resource 2017/04/17 22:25:08 2nd query, Using a database connection with ID 4 2017/04/17 22:25:08 The resource was released into the pool. 2017/04/17 22:25:08 4th query, using a database connection with ID 1 2017/04/17 22:25:08 resources released into the pool 2017 /04/17 22:25:08 3rd query, using database connection with ID 5 2017/04/17 22:25:08 resource pool is full, release this resource. 2017/04/17 22:25:08 close Connection 52017/04/17 22:25:09 1th query, using a database connection with ID 3 2017/04/17 22:25:09 the resource pool is full, release this resource. 2017/04/17 22:25:09 close connection 32017/04/17 22:25:09 No. 0 query, Using a database connection with ID 2 2017/04/17 22:25:09 the resource pool is full, release the resource. 2017/04/17 22:25:09 close connection 22017/04/17 22:25:09 start close resource pool 2017/04/17 22:25:09 Close Connection 42017/04/17 22:25:09 close connection 1

Here, we have completed the management of a resource pool and have used tests.
Resource object pools are used more frequently because we want to cache some objects for use so that they are efficient and do not call the GC very often, so go provides us with native resource pool management to prevent us from repeating the wheel, which is what we sync.Pool look at just our example, if sync.Poolimplementation.

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364

PackageMainImport("Log""Math/rand""Sync""Sync/atomic""Time")Const(//simulation of the maximum GoroutineMaxgoroutine =5) func main() {//wait for task to completevarWG Sync. Waitgroupwg.add (Maxgoroutine) P:=&sync. Pool{new:createconnection,}//Simulate several goroutine use resource pools to query data at the same time forQuery: =0; Query < Maxgoroutine; query++ {Go func(q int) {dbquery (q, p) WG. Done ()} (query)}WG. Wait ()}//Simulation database query func dbquery(query int, pool *sync. Pool) {Conn:=pool. Get (). (*dbconnection)deferPool. PUT (conn)//Simulation QueryTime. Sleep (time. Duration (Rand. INTN ( +)) * time.millisecond) log. Printf ("%d queries, using a database connection with ID%d", query, Conn.id)}//Database connectiontypeDbConnectionstruct{IDInt32//Connection flag}//Implement IO. Closer Interface func (db *dbconnection) Close() error {log. Println ("Close Connection", db.id)return Nil}varIdcounterInt32//methods for generating database connections for use by resource pools func createconnection() interface {} {//concurrency security, generating unique flags for database connectionsID: = Atomic. AddInt32 (&idcounter,1)return&dbconnection{id:id}}

Make minor changes, because the system library does not provide New this kind of factory function, so we use the literal to create one sync.Pool , notice the New field inside, this is a method to return any object, similar to our own implementation of the field in the resource pool, the factory meaning is the same, When there are no available resources, generate one.

Here we notice that there is no size limit to the resource pool of the system, that is, by default there is no limit and is limited by the memory size.

The method of obtaining and releasing the resource is Get the Put same as, and is very concise, returning any object interface{} .

12345

2017/04/17 22:42:43 the No. 0 query, using a database connection with ID 2, 2017/04/17 22:42:43 2nd query, using a database connection with ID 5 2017/04/17 22:42:43 4th query, Using a database connection with ID 1 2017/04/17 22:42:44 the 3rd query, using a database connection with ID 4 2017/04/17 22:42:44 1th query, using a database connection with ID 3

As for the resource pool of the system, we need to be aware that it caches objects that are temporary, and that the stored objects will be erased when the GC is next.

"Go language Combat" reading notes, not to be continued, welcome to sweep code attention flysnow_org to the public or website http://www.flysnow.org/, the first time to see follow-up notes. If you feel helpful, share it with your friends and thank you for your support.