Turn: memcached Java client spymemcached consistent hash algorithm

Transferred from: http://colobu.com/2015/04/13/consistent-hash-algorithm-in-java-memcached-client/memcached A consistent hash algorithm for Java client spymemcached

Recently saw two articles, one is a strange but silent Unified Lake MurmurHash, another is Zhang Yang consistent hashing algorithm and its application in distributed system. Although I have been using Memcached's Java client spymemcached in my project for several years, the details of its consistent hashing algorithm have never been studied carefully. Take this opportunity to take a special look at its source code.

We know that the memcached itself does not provide a distributed function, the general client will implement a consistent hash algorithm, based on the key value to calculate which node should be accessed.

Implementation of Ketama Hash

Spymemcached implements several HASH algorithms: Native_hash,crc_hash,fnv1_64_hash,fnv1a_64_hash,fnv1_32_hash,fnv1a_32_hash,ketama_hash.
Compared with the previous hash algorithms, the KETAMA hash algorithm can distribute the virtual node of the server to the ring fairly evenly, it is a hash algorithm based on MD5 hash.
The following class is my streamlined spymemcached Ketamanodelocator class for testing the distribution of generated virtual nodes, which prints out the interval between two virtual nodes. If the interval is more homogeneous, we believe that the key value computed using the same hash algorithm should be evenly spaced on each node.

Spymemcached The virtual node is calculated for each node using the 节点地址 + "-i" format, I the maximum number of virtual nodes per node, the default is 160.

1234567891011121314151617181920212223242526272829303132333435

Package com.colobu.consistenthashing;import java.util.List;import Java.util.TreeMap;public class Ketama { Public Treemap<long, node> hashnodes;Public HashAlgorithm HashAlgorithm;protected void setketamanodes (list<node> nodes) { Treemap<long, node> newnodemap =New Treemap<long, node> ();int numreps = ; For (Node node:nodes) {if (HashAlgorithm = = Hashalgorithm.ketama_hash) {For (int i = 0; i < numreps/ 4; i++) { byte[] Digest = HASHALGORITHM.COMPUTEMD5 (Node.getname () + "-" + i); For (int h = 0; h < 4; h++) { Long k = ((long) (digest[3 + H * 4] & 0xFF) << | ((long) (digest[2 + H * 4] & 0xFF) << + ) | ((long) (digest[1 + H * 4] & 0xFF) << 8) | (DIGEST[H *4] & 0xFF); Newnodemap.put (k, node); }}}else {For (int i = 0; i < numreps; i++) { Newnodemap.put (Hashalgorithm.hash (node +"-" + i), node);}}} hashnodes = Newnodemap;}}

Write a test class to see the distribution of virtual nodes:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465 66676869

Package com.colobu.consistenthashing;import java.util.ArrayList;import Java.util.Iterator;import java.util.List;import java.util.Map.Entry;public class Main { public static void main (string[] args) { //system.out.println ("Test Ketama hash");//testketama ();//system.out.println ("\r\n\r\n Test native Hash");//testhash (hashalgorithm.native_hash);//system.out.println ("\r\n\r\n Test CRC hash");//max=32767//testhash (hashalgorithm.crc_hash);//system.out.println ("\r\n\r\n Test Fnv1_64_hash");//testhash (hashalgorithm.fnv1_64_hash);//system.out.println ("\r\n\r\n Test Fnv1a_64_hash");//testhash (hashalgorithm.fnv1a_64_hash);//system.out.println ("\r\n\r\n Test MurmurHash");//testhash (hashalgorithm.murmurhash_32);System.out.println ("\r\n\r\n Test MurmurHash 64"); Testhash (hashalgorithm.murmurhash_64);}private static void Testhash (hashalgorithm hash) { Ketama Ketama =new Ketama ();Ketama.hashalgorithm = hash; list<node> nodes =new Arraylist<> ();For (int i=0; i< ; i++) { Nodes.Add (New Node ("name-" + i)); } ketama.setketamanodes (nodes); Iterator<entry<long, node>> it = Ketama.hashNodes.entrySet (). Iterator (); Entry<long, node> prior = It.next ();While (It.hasnext ()) {Entry<long, node> current = It.next (); System.out.println ("interval:" + (Current.getkey ()-Prior.getkey ()) + "=" + current.getkey () + "-" + Prior.getkey ()); Prior = current;} }private static void Testketama () { Ketama Ketama =new Ketama ();Ketama.hashalgorithm = Hashalgorithm.ketama_hash; list<node> nodes =new Arraylist<> ();For (int i=0; i< ; i++) { Nodes.Add (New Node ("name-" + i)); } ketama.setketamanodes (nodes); Iterator<entry<long, node>> it = Ketama.hashNodes.entrySet (). Iterator (); Entry<long, node> prior = It.next ();While (It.hasnext ()) {Entry<long, node> current = It.next (); System.out.println ("interval:" + (Current.getkey ()-Prior.getkey ()));Prior = Current;}} }

The actual results see the Ketama algorithm is still good.

Adding MurmurHash algorithm

Jiangnan White in that article introduced the MurmurHash algorithm, open source Chinese community also translated a Hash function overview of the popular Science article.
What happens if we add the MurmurHash algorithm to spymemcached. I haven't tested it for performance, but it's a good one from a distribution point of view.
There are several MurmurHash implementations on the internet, such as guava, Cassandra and so on. I don't want to introduce a third party package, so I copied the implementation of the Viliam Holub directly.

Add the MurmurHash enumeration type to the HashAlgorithm algorithm.

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465 6667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611 7118119120121122123124125126127128129130131132133134135136137138139140141142143144

Package com.colobu.consistenthashing;import java.security.MessageDigest;import java.security.NoSuchAlgorithmException;import java.util.zip.CRC32;public enum HashAlgorithm { /*** Native Hash (String.hashcode ()). */Native_hash,/*** Crc_hash as used by the Perl API. This would be more consistent both * across multiple API users as well as Java versions, but is mostly likely * significant Ly slower. */Crc_hash,/*** FNV hashes is designed to being fast while maintaining a low collision rate. * The FNV speed allows one to quickly hashes lots of data while maintaining a * reasonable collision rate. * *@see <a href= "http://www.isthe.com/chongo/tech/comp/fnv/" >FNV* Comparisons</a> *@see <a href= "Http://en.wikipedia.org/wiki/Fowler_Noll_Vo_hash" >FNV at* wikipedia</a> * */Fnv1_64_hash,/*** Variation of FNV. */Fnv1a_64_hash,/*** 32-bit FNV1. */Fnv1_32_hash,/*** 32-bit fnv1a. */Fnv1a_32_hash, murmurhash_32, murmurhash_64,/*** md5-based hash algorithm used by Ketama. */Ketama_hash;private Static final long fnv_64_init = 0xcbf29ce484222325l; private Static final long fnv_64_prime = 0x100000001b3l; private Static final long fnv_32_init = 2166136261L; private Static final long fnv_32_prime = 16777619; private static messagedigest md5digest = null; Static { try {Md5digest = Messagedigest.getinstance ("MD5");}catch (NoSuchAlgorithmException e) { throw New RuntimeException ("MD5 not supported", e); } }/*** Compute the hash for the given key. * *@return A positive integer hash*/public Long hash (final String k) { Long RV = 0; int len = K.length (); switch (this) { Case Native_hash:RV = K.hashcode ();Break ; Case Crc_hash: //Return (CRC32 (Shift) >>) & 0x7fff;CRC32 CRC32 =new CRC32 ();Crc32.update (K.getbytes ()); RV = (Crc32.getvalue () >>) & 0x7fff; Break ; Case Fnv1_64_hash: //Thanks to [e-mail protected] for the pointerRV = Fnv_64_init;For (int i = 0; i < len; i++) { RV *= fnv_64_prime; RV ^= K.charat (i); }Break ; Case Fnv1a_64_hash:RV = Fnv_64_init;For (int i = 0; i < len; i++) { RV ^= K.charat (i); RV *= fnv_64_prime; }Break ; Case Fnv1_32_hash:RV = Fnv_32_init;For (int i = 0; i < len; i++) { RV *= fnv_32_prime; RV ^= K.charat (i); }Break ; Case Fnv1a_32_hash:RV = Fnv_32_init;For (int i = 0; i < len; i++) { RV ^= K.charat (i); RV *= fnv_32_prime; }Break ; Case murmurhash_32:RV = Murmurhash.hash32 (k);Break ; Case murmurhash_64:RV = murmurhash.hash64 (k);Break ; Case Ketama_hash: byte[] Bkey = ComputeMd5 (k);RV = ((long) (bkey[3] & 0xFF) << | ((long) (bkey[2] & 0xFF) << | ((long) (bkey[1] & 0xFF) << 8) | (bkey[0] & 0xFF); Break ; Default: assert false; }return RV & 0xffffffffL; / * Truncate to 32-bits * /}/*** Get The MD5 of the given key. */public static byte[] computeMd5 (String k) { MessageDigest MD5;try {MD5 = (messagedigest) md5digest.clone (); }catch (Clonenotsupportedexception e) { throw New RuntimeException ("clone of MD5 not supported", e); } md5.update (K.getbytes ());return Md5.digest ();}}

The actual results see MurmurHash is also quite homogeneous.

The realization of xmemcached

Xmemcached is another memcached Java client that implements a hash algorithm similar to spymemcached. Only a few new HASH algorithms have been added: Mysql_hash,elf_hash,rs_hash,lua_hash,one_at_a_time.

Twemproxy

Twemproxy is a memcahced gateway program. It implements the following hash algorithms.

• One_at_a_time
• Md5
• Crc16
• CRC32 (CRC32 implementation compatible with libmemcached)
• CRC32A (correct CRC32 implementation as per the spec)
• Fnv1_64
• Fnv1a_64
• Fnv1_32
• Fnv1a_32
• Hsieh
• Murmur
• Jenkins

Turn: memcached Java client spymemcached consistent hash algorithm