Introduction to MC & amp; Redis & amp; Ehcache

Introduction to MC & Redis & Ehcache

I haven't updated my blog for a long time. I hope I can continue writing my blog later. Now I find this is a great process. I 've sorted out the usage and introduction of cache a long time ago. Now I post it as my personal notes:

 

Introduction to Memecached

 

Many Web applications save data to RDBMS. The application server reads data from the data and displays it in the browser. However, as the data volume increases and access is concentrated, the burden on RDBMS increases, the database response deteriorates, and the website display latency. In this case, memcached is ready to use. Memcached is a high-performance distributed memory cache server. The general purpose is to reduce the number of database accesses by caching database query results, so as to speed up dynamic Web applications and improve scalability.

Storage Method

To improve performance, data stored in memcached is stored in the memory storage space built in memcached. Because the data only exists in the memory, restarting memcached and the operating system will cause all data to disappear. In addition, when the content capacity reaches the specified value, the unused cache is automatically deleted based on the LRU (Least Recently Used) algorithm. Memcached itself is a server designed for caching, so it does not take the permanent data into consideration.

 

MC & Spring

Spring Configuration:

<Beannamebeanname = "memcachedClient" class = "net. rubyeye. xmemcached. utils. XMemcachedClientFactoryBean "destroy-method =" shutdown "> <property name =" servers "value =" $ {VIPCPS_MC_S1_HOST }:$ {VIPCPS_MC_S1_PORT} $ {VIPCPS_MC_S2_HOST }: $ {audience }$ {VIPCPS_MC_S3_HOST }:$ {VIPCPS_MC_S3_PORT }$ {VIPCPS_MC_S4_HOST }:: {audience }$ {audience }:$ {VIPCPS_MC_S5_PORT} "></property> <! -- Nio connection poolsize. The connection pool is not recommended for performance testing. --> <property name = "connectionPoolSize" value = "1"> </property> <! -- Distributed strategy --> <property name = "sessionLocator"> <bean class = "net. rubyeye. xmemcached. impl. ketamaMemcachedSessionLocator "> </bean> </property> </bean>

Redis Introduction

Redis is a key-value storage system. Similar to Memcached, Memcached supports more storage value types, including string, list, set, and zset) and hash (hash type ). These data types support push/pop, add/remove, Intersection Set and difference set, and more abundant operations, and these operations are atomic. On this basis, redis supports sorting in different ways. Like memcached, data is cached in the memory to ensure efficiency. The difference is that redis periodically writes the updated data to the disk or writes the modification operation to the append record file, and on this basis implements master-slave (master-slave) synchronization.

Redis is a high-performance key-value database. The emergence of redis largely compensates for the shortage of key/value storage such as memcached, and can play a good complementary role in relational databases in some cases. It provides Python, Ruby, Erlang, and PHP clients for ease of use. [1]

Redis supports master-slave synchronization. Data can be synchronized from the master server to any number of slave servers. The slave server can be the master server associated with other slave servers. This allows Redis to perform single-layer tree replication. Data can be written intentionally or unintentionally from a disk. Because the publishing/subscription mechanism is fully implemented, you can subscribe to a channel and receive the complete message publishing records of the master server when synchronizing the tree from the database anywhere. Synchronization is helpful for the scalability and data redundancy of read operations.

Storage Method

Redis storage is divided into three parts: memory storage, disk storage, and log files. The configuration file contains three parameters for configuration. Save seconds updates, save configuration, specifies how many update operations are performed within the time range, the data is synchronized to the data file. This can be used with multiple conditions. For example, three conditions are set in the default configuration file. Appendonly yes/no, appendonly configuration, indicating whether to record logs after each update operation. If not enabled, data may be lost for a period of time during power failure. Because redis synchronizes data files according to the above save conditions, some data will only exist in the memory for a period of time. Appendfsync no/always/everysec, appendfsync configuration, no indicates that the data cache is synchronized to the disk by the operating system, and always indicates that the data is written to the disk by manually calling fsync () after each update operation, everysec indicates that synchronization is performed once per second.

 

Redis is a replacement of Memched.

 

Consistent Hash (server cluster)

Determine the storage server by using an operation between the key value and the hash value that the client represents.

To solve the load balancing problem, when there are too few server nodes, we need to construct a virtual server node.

The following vulnerabilities of simple hash are solved:

1) when adding or deleting a market point, the update efficiency is low. In the systemStorage NodeWhen the number increases or decreases, the ing formula changes to hash (object) % (N ± 1), which changes the ing positions of all obiect, the ing position of the entire system data object needs to be re-computed, and the system cannot respond to external access normally, which will cause the system to crash.

2) poor balance, without considering the performance differences between nodes. Due to the improvement of hardware performance, the newly added nodes have better carrying capacity. How can we improve the algorithm to make better use of the node performance.

3) The monotonicity is insufficient. An important indicator for measuring data distribution technology is monotonicity. monotonicity means that if some content has been allocated to the corresponding buffer through hash calculation, and when a new buffer is added to the system, the hash result should ensure that the original allocated content can be mapped to the new buffer instead of other buffers in the old buffer set.

 

Redis & Spring

<! -- Default --> <bean id = "jedisPoolConfig" class = "redis. clients. jedis. jedisPoolConfig "> <property name =" maxActive "value =" $ {redis. pool. maxActive} "/> <property name =" maxIdle "value =" $ {redis. pool. maxIdle} "/> <property name =" maxWait "value =" $ {redis. pool. maxWait} "/> <property name =" testOnBorrow "value =" $ {redis. pool. testOnBorrow} "/> </bean> <bean id =" stringRedisTemplate0 "class =" org. springframework. data. redis. Core. stringRedisTemplate "p: connection-factory-ref =" jedisConnectionFactory0 "scope =" prototype "> </bean> <bean id =" stringRedisTemplate1 "class =" org. springframework. data. redis. core. stringRedisTemplate "p: connection-factory-ref =" jedisConnectionFactory1 "scope =" prototype "> </bean> <! -- Custom --> <bean id = "jedisConnectionFactory0" class = "org. springframework. data. redis. connection. jedis. jedisConnectionFactory "> <property name =" hostName "value =" $ {VIPCPS_REDIS_HOST_0} "/> <property name =" port "value =" $ {VIPCPS_REDIS_PORT_0} "/> <property name = "poolConfig" ref = "jedisPoolConfig"/> </bean> <bean id = "jedisConnectionFactory1" class = "org. springframework. data. redis. connection. jedis. jedisConnectionFactory "> <property name =" hostName "value =" $ {VIPCPS_REDIS_HOST_1} "/> <property name =" port "value =" $ {VIPCPS_REDIS_PORT_1} "/> <property name = "poolConfig" ref = "jedisPoolConfig"/> </bean> <bean id = "redisTemplateRouter" class = "com. vipshop. unionweb. framework. redis. redisTemplateRouter "> <constructor-arg> <list> <ref bean =" stringRedisTemplate0 "/> <ref bean =" stringRedisTemplate1 "/> </list> </constructor-arg> </bean>

Java Implementation

 

Public class detail {privatestatic finalLogger logger = LoggerFactory. getLogger (logging. class); privatestatic CRC32 crc32 = newCRC32 ();/*** number of services */privatestatic intSERVER_COUNT = 1; privatestatic List
 
  
RedisTemplates = newArrayList
  
   
(); PublicRedisTemplateRouter (List
   
    
RedisTemplates) {RedisTemplateRouter. redisTemplates = redisTemplates; SERVER_COUNT = redisTemplates. size ();} publicstatic List
    
     
GetRedisTemplates () {returnredisTemplates;} publicstatic StringRedisTemplategetRouteRedis (String key) {crc32.reset (); crc32.update (key. getBytes (); longcrc32Value = crc32.getValue (); intindex = Math. abs (int) (crc32Value % SERVER_COUNT); logger. debug ("route to redis [key :{}, index :{}, crc32 :{}]", new Object [] {key, index, crc32Value}); returnredisTemplates. get (index );}}
    
   
  
 

 

Usage:

 

RedisTemplate template= RedisTemplateRouter.getRouteRedis(key);Object result = template.opsForValue().get(key);

 

Ehcache

It is a pure Java in-process cache framework, and the cache memory and disk storage can be scaled to several GB. Ehcache is a widely used open source Java distributed cache; supports multiple elimination algorithms such as LRU, LFU, and FIFO, cache pages, objects, and data, and cluster/distributed cache;

 

Spring configuration of object Cache

The object cache adds the queried data to the cache. It is obtained directly from the cache next time it is queried, rather than from the database.

 

<Bean id = "cacheManagerFactory" class = "org. springframework. cache. ehcache. ehCacheManagerFactoryBean "> <property name =" configLocation "value =" classpath: com/config/ehcache. xml "/> </bean> <! -- Supports cache annotation --> <cache: annotation-driven/> <! -- The default value is cacheManager to configure the cache manager --> <bean id = "cacheManager" class = "org. springframework. cache. ehcache. ehCacheCacheManager "> <property name =" cacheManager "ref =" cacheManagerFactory "/> </bean>

 

Ehcahe configuration instance

 

<EhcacheupdateCheckehcacheupdateCheck = "false" monitoring = "autodetect" dynamicConfig = "true"> <diskStorepathdiskStorepath = "java. io. tmpdir "/> <defaultCache maxElementsInMemory =" 10000 "eternal =" false "temperature =" 120 "timeToLiveSeconds =" 0 "overflowToDisk =" true "maxElementsOnDisk =" 10000000 "diskPersistent =" false "diskExpiryThreadIntervalSeconds =" 120 "memoryStoreEvictionPolicy =" LRU "/> <! -- Channel integrated report cache --> <cachenamecachename = "channelSummaryCache" maxElementsInMemory = "100" eternal = "false" leading = "120" timeToLiveSeconds = "120" overflowToDisk = "false" maxElementsOnDisk = ""10000000" diskPersistent = "false" diskExpiryThreadIntervalSeconds = "120" memoryStoreEvictionPolicy = "FIFO"/> <! -- Configure the custom cache maxElementsInMemory: Maximum number of objects allowed to be created in the cache eternal: whether the objects in the cache are permanent. If yes, the timeout setting is ignored and the objects never expire. TimeToIdleSeconds: the cache data passivation time, that is, the maximum time interval between two access times before an element disappears, which is valid only when the element is not permanently resident, if the value is 0, it means that the element can pause for an infinite period of time. TimeToLiveSeconds: The survival time of cached data, that is, the maximum time interval between an element construction and extinction. This is only valid when the element is not permanently resident, if the value is 0, it means that the element can pause for an infinite period of time. OverflowToDisk: whether to enable disk cache when the memory is insufficient. MemoryStoreEvictionPolicy: The elimination algorithm after the cache is full. -->

 

Java

There are three cache Annotations:

@ Cacheable (value = "accountCache"). This comment indicates that when this method is called, it will be queried from a cache named accountCache. If no, the actual method (query database) is executed, and the execution result is saved to the cache. Otherwise, the objects in the cache are returned.

@ CacheEvict annotation to mark the method for clearing the cache. When this method is called, the cache is cleared.

@ CachePut annotation. This annotation ensures that the method is executed and the return value of the method is recorded in the cache for Synchronous updates between the cache and the database.

# With Parameters

 

@CachePut(value=" channelSummaryCache",key="#startTime.getTime().toString().concat('_').concat(''+#endTime).concat('_')" +".concat(''+#commissionType).concat('_').concat(#userId).concat('_'+#channelTag)")PublicList
 
  summaryGroupByChannel(Integer userId,Date startTime, Date endTime, Short commissionType, String channelTag) {……}