Implementation of memcached cluster and automatic client hash to different servers

First, analyze some code when the Java client is started.

Memcached allows you to directly set multiple servers attributes to achieve multiple memcahced balancing. The corresponding attribute is weights, which literally means weight. After analyzing the code, it is the same as what I think.

The code for starting memcached is usually as follows:

Java code

1. Sockiopool pool = sockiopool. getinstance (poolname );
2. Pool. setservers (servers );
3. Pool. setweights (weights );
4. Pool. setinitconn (initconn );
5. Pool. setminconn (minconn );
6. Pool. setmaxconn (maxconn );
7. Pool. setmaxidle (maxidle );
8. Pool. setmaxbusytime (maxbusytime );
9. Pool. setmaintsleep (maintsleep );
10. Pool. setsocketto (socketto );
11. Pool. setsocketconnectto (socketconnectto );
12. Pool. setnagle (Nagle );
13. Pool. sethashingalg (sockiopool. new_compat_hash );
14. Pool. initialize ();
15. Memcachedclient client = new memcachedclient (poolname );

Both servers and weights are an array, that is, multiple servers can be set at the same time.

Then let's take a look at what Java code pool. initialize () has done.

1. Availpool = new hashmap <string, Map <sockio, long> (servers. length * initconn );
2. Busypool = new hashmap <string, Map <sockio, long> (servers. length * initconn );
3. Deadpool = new identityhashmap <sockio, integer> ();
4. Hostdeaddur = new hashmap <string, long> ();
5. Hostdead = new hashmap <string, date> ();
6. Maxcreate = (poolmultiplier> minconn )? Minconn: minconn/poolmultiplier; // only create up to maxcreate connections at once
7. If (log. isdebugenabled ()){
8. Log. debug ("++ initializing pool with following settings :");
9. Log. debug ("++ initial size:" + initconn );
10. Log. debug ("++ + min spare:" + minconn );
11. Log. debug ("++ + Max spare:" + maxconn );
12. }
13. // If servers is not set, or it empty, then
14. // Throw a runtime exception
15. If (servers = NULL | servers. Length <= 0 ){
16. Log. Error ("+++ trying to initialize with no servers ");
17. Throw new illegalstateexception ("++ trying to initialize with no servers ");
18. }
19. // Initalize our internal hashing Structures
20. If (this. hashingalg = consistent_hash)
21. Populateconsistentbuckets ();
22. Else
23. Populatebuckets ();

We can see that this is to open up some connection pool space, and then call the algorithm to execute populatebuckets () based on the hash algorithm we selected, or populateconsistentbuckets ();

The hash algorithm has four types of Java code.

1. // Native string. hashcode ();
2. Public static final int native_hash = 0;
3. // Original compatibility hashing algorithm (works with other clients)
4. Public static final int old_compat_hash = 1;
5. // New CRC32 based compatibility hashing algorithm (works with other clients)
6. Public static final int new_compat_hash = 2;
7. // MD5 based -- stops thrashing when a server added or removed
8. Public static final int consistent_hash = 3;

We usually use new_compat_hash, which ensures that wokrs with other clients

So let's take a look at what populatebuckets () has done.

Java code

1. This. Buckets = new arraylist <string> ();
2. For (INT I = 0; I <servers. length; I ++ ){
3. If (this. weights! = NULL & this. Weights. length> I ){
4. For (int K = 0; k <this. weights [I]. intvalue (); k ++ ){
5. This. Buckets. Add (servers [I]);
6. If (log. isdebugenabled ())
7. Log. debug ("++ added" + servers [I] + "to server bucket ");
8. }
9. }
10. Else {
11. This. Buckets. Add (servers [I]);
12. }
13. // Create initial connections
14. For (Int J = 0; j <initconn; j ++ ){
15. Sockio socket = createsocket (servers [I]);
16. If (socket = NULL ){
17. Break;
18. }
19. Addsockettopool (availpool, servers [I], socket );
20. }
21. }

Assume that the servers we set are 192.168.0.1: 44444 and 192.168.0.2: 22222.
Then we set weights to 5 and 3.
The value of the buckets list will eventually be
[
192.168.0.1: 44444,
192.168.0.1: 44444,
192.168.0.1: 44444,
192.168.0.1: 44444,
192.168.0.1: 44444,
192.168.0.2: 22222,
192.168.0.2: 22222.
192.168.0.2: 22222.
]
Then, the socket is created by Servers Based on the initcon initial connections.
So what is the purpose of this buckets?

It is called when we use set to store objects.

Java code

1. Sockiopool. sockio sock = pool. getsock (Key, hashcode );

Let's take a look at the pool. getsock code.

Java code

1. // Get initial Bucket
2. Long bucket = getbucket (Key, hashcode );
3. String Server = (this. hashingalg = consistent_hash)
4. ? Consistentbuckets. Get (bucket)
5. : Buckets. Get (INT) bucket );

Here is a code segment. Let's look at the Java code of getbucket.

1. Private long getbucket (string key, integer hashcode ){
2. Long Hc = gethash (Key, hashcode );
3. If (this. hashingalg = consistent_hash ){
4. Return findpointfor (HC );
5. }
6. Else {
7. Long bucket = HC % buckets. Size ();
8. If (bucket <0) bucket * =-1;
9. Return bucket;
10. }
11. }

Regardless of the key and hashcode, we can see that after a HC value is calculated, HC % buckets is directly implemented. size () is actually based on the number of buckets, the final value must be buckets. A value in the size () range.
Then the final server value is based on buckets. get (INT) bucket), then if we get the bucket is 3, we will get the list by referring to the value in the buckets above. get (3) = 192.168.0.1: 44444, so different servers are obtained based on the values set by weight. If weights is set to 10:1, 10 identical servers and 10 different servers are obtained in buckets, in the future, the server hash is likely to be the first server set in servers.

 

Address: http://bachmozart.javaeye.com/blog/211836