Implementation of memcached cluster and automatic client hash to different servers

Source: Internet
Author: User

First, analyze some code when the Java client is started.

Memcached allows you to directly set multiple servers attributes to achieve multiple memcahced balancing. The corresponding attribute is weights, which literally means weight. After analyzing the code, it is the same as what I think.

The code for starting memcached is usually as follows:

Java code
  1. Sockiopool pool = sockiopool. getinstance (poolname );
  2. Pool. setservers (servers );
  3. Pool. setweights (weights );
  4. Pool. setinitconn (initconn );
  5. Pool. setminconn (minconn );
  6. Pool. setmaxconn (maxconn );
  7. Pool. setmaxidle (maxidle );
  8. Pool. setmaxbusytime (maxbusytime );
  9. Pool. setmaintsleep (maintsleep );
  10. Pool. setsocketto (socketto );
  11. Pool. setsocketconnectto (socketconnectto );
  12. Pool. setnagle (Nagle );
  13. Pool. sethashingalg (sockiopool. new_compat_hash );
  14. Pool. initialize ();
  15. Memcachedclient client = new memcachedclient (poolname );

Both servers and weights are an array, that is, multiple servers can be set at the same time.

Then let's take a look at what Java code pool. initialize () has done.

  1. Availpool = new hashmap <string, Map <sockio, long> (servers. length * initconn );
  2. Busypool = new hashmap <string, Map <sockio, long> (servers. length * initconn );
  3. Deadpool = new identityhashmap <sockio, integer> ();
  4. Hostdeaddur = new hashmap <string, long> ();
  5. Hostdead = new hashmap <string, date> ();
  6. Maxcreate = (poolmultiplier> minconn )? Minconn: minconn/poolmultiplier; // only create up to maxcreate connections at once
  7. If (log. isdebugenabled ()){
  8. Log. debug ("++ initializing pool with following settings :");
  9. Log. debug ("++ initial size:" + initconn );
  10. Log. debug ("++ + min spare:" + minconn );
  11. Log. debug ("++ + Max spare:" + maxconn );
  12. }
  13. // If servers is not set, or it empty, then
  14. // Throw a runtime exception
  15. If (servers = NULL | servers. Length <= 0 ){
  16. Log. Error ("+++ trying to initialize with no servers ");
  17. Throw new illegalstateexception ("++ trying to initialize with no servers ");
  18. }
  19. // Initalize our internal hashing Structures
  20. If (this. hashingalg = consistent_hash)
  21. Populateconsistentbuckets ();
  22. Else
  23. Populatebuckets ();

We can see that this is to open up some connection pool space, and then call the algorithm to execute populatebuckets () based on the hash algorithm we selected, or populateconsistentbuckets ();

The hash algorithm has four types of Java code.

  1. // Native string. hashcode ();
  2. Public static final int native_hash = 0;
  3. // Original compatibility hashing algorithm (works with other clients)
  4. Public static final int old_compat_hash = 1;
  5. // New CRC32 based compatibility hashing algorithm (works with other clients)
  6. Public static final int new_compat_hash = 2;
  7. // MD5 based -- stops thrashing when a server added or removed
  8. Public static final int consistent_hash = 3;

We usually use new_compat_hash, which ensures that wokrs with other clients

So let's take a look at what populatebuckets () has done.

Java code
  1. This. Buckets = new arraylist <string> ();
  2. For (INT I = 0; I <servers. length; I ++ ){
  3. If (this. weights! = NULL & this. Weights. length> I ){
  4. For (int K = 0; k <this. weights [I]. intvalue (); k ++ ){
  5. This. Buckets. Add (servers [I]);
  6. If (log. isdebugenabled ())
  7. Log. debug ("++ added" + servers [I] + "to server bucket ");
  8. }
  9. }
  10. Else {
  11. This. Buckets. Add (servers [I]);
  12. }
  13. // Create initial connections
  14. For (Int J = 0; j <initconn; j ++ ){
  15. Sockio socket = createsocket (servers [I]);
  16. If (socket = NULL ){
  17. Break;
  18. }
  19. Addsockettopool (availpool, servers [I], socket );
  20. }
  21. }

Assume that the servers we set are 192.168.0.1: 44444 and 192.168.0.2: 22222.
Then we set weights to 5 and 3.
The value of the buckets list will eventually be
[
192.168.0.1: 44444,
192.168.0.1: 44444,
192.168.0.1: 44444,
192.168.0.1: 44444,
192.168.0.1: 44444,
192.168.0.2: 22222,
192.168.0.2: 22222.
192.168.0.2: 22222.
]
Then, the socket is created by Servers Based on the initcon initial connections.
So what is the purpose of this buckets?

It is called when we use set to store objects.

Java code
  1. Sockiopool. sockio sock = pool. getsock (Key, hashcode );

Let's take a look at the pool. getsock code.

Java code
  1. // Get initial Bucket
  2. Long bucket = getbucket (Key, hashcode );
  3. String Server = (this. hashingalg = consistent_hash)
  4. ? Consistentbuckets. Get (bucket)
  5. : Buckets. Get (INT) bucket );

Here is a code segment. Let's look at the Java code of getbucket.

  1. Private long getbucket (string key, integer hashcode ){
  2. Long Hc = gethash (Key, hashcode );
  3. If (this. hashingalg = consistent_hash ){
  4. Return findpointfor (HC );
  5. }
  6. Else {
  7. Long bucket = HC % buckets. Size ();
  8. If (bucket <0) bucket * =-1;
  9. Return bucket;
  10. }
  11. }

Regardless of the key and hashcode, we can see that after a HC value is calculated, HC % buckets is directly implemented. size () is actually based on the number of buckets, the final value must be buckets. A value in the size () range.
Then the final server value is based on buckets. get (INT) bucket), then if we get the bucket is 3, we will get the list by referring to the value in the buckets above. get (3) = 192.168.0.1: 44444, so different servers are obtained based on the values set by weight. If weights is set to 10:1, 10 identical servers and 10 different servers are obtained in buckets, in the future, the server hash is likely to be the first server set in servers.

 

Address: http://bachmozart.javaeye.com/blog/211836

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.