Scaling Memcache at Facebook

The recent group is writing Memcache's operation and maintenance manual, around memcache operations in the study of the collation of data found a non-jumping, that is, Facebook a few years ago on the operation of Memcache summary.

The relevant primary information is as follows:

1 Facebook engineers share the video https://www.youtube.com/watch?v=6phA3IAcEJ8 [please FQ]

2 Facebook PPT https://www.usenix.org/sites/default/files/conference/protected-files/nishtala_nsdi13_slides.pdf

3 Facebook paper "Scaling Memcache at Facebook" https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_ Update.pdf

The following video and papers around these materials to do a summary, just for not accustomed to see English students to do abstract.

Why Facebook has the need for massive scaling memcache

Facebook is a feature of this SNS Internet product:

1. Near real-time communication real-time communication (release, reply, private messages)

2. Aggregate content on-the-fly from multiple sources aggregating the contents of multiple data sources (avatar, status, picture wait)

3. Be able to access and update very popular shared content (Hotspot star contents)

4. Scale to process millions of user requests per second (high user base access)

The technical requirements resulting from this feature are:

Support A very heavy read load (read request is much larger than write request---cache)

? Over 1 billion Reads/second

? Insulate backend services from high read rates

Geographically distributed (Geographical division)

Support a constantly evolving product (continuous integration and release)

? System must bes flexible enough to support a variety of use cases

? Support rapid deployment of new features

Persistence handled outside the system (conformance requirements)

? Support mechanisms to refill after updates

Based on Facebook's product features and technical requirements, Facebook has chosen Memcache and has the need for scaling memcache

Baixia said: In fact, Facebook's cache requirements represent the vast number of SNS product cache features, similar product requirements such as more UGC, cloud music and other reading scenes larger than the writing scene.

However, mention memcache can not ignore the Redis pea Pod chose Codis (Proxy based Redis cluster solution supporting pipeline and scaling dynamically) to solve the Dynamically scaled, massively cached.

There will be an indispensable share of the Redis-related caching solution, in fact, when it comes to caching, after focusing on choosing Memcache or Redis, OPS or architects should think about such a problem; "How to build a massively scalable cache cluster that maintains a dynamic scale" This article looks at how Facebook explored and practiced this aspect a few years ago.

Memcache General Use Posture:

Posture 1: Read: Read if Miss back to the source database and set

Posture 2: Write: Update the database and delete the cache

Note: Why use delete instead of set to resolve data update

Consider that the delete is better idempotent than set and

Issue 1: Possible stale sets (dirty write)

WebA WebB is a two front-end colleague who operates a cache object the database content for the MC cache object is DB

WebA operation database set to Db_a

WebB operation database set to Db_b

WebB operation Cache set to Mc_b

WebA operation Cache set to Mc_a

This is what you will find that the content of the DB is Db_b MC content is mc_a the database cache is inconsistent

How to solve this problem, the paper has written, but I see this side should be opened memcache CAs function to solve.

Problem 2 exists Thundering herds (surprise swarm effect) cache avalanche

In layman's terms, the cache avalanche, when a hotkey fails, all caches in accordance with posture 1 strategy will be back to the source, which causes the database pressure instantaneous, severe may overwhelm the database.

Facebook gives the strategy:

clients given a choice of using a slightly stale value or waiting is to buy using outdated content, or wait a while.

This blog has also done some discussion [http://www.php.cn/php-weizijiaocheng-89275.html]

Baixia says: How to avoid 1: Lock or queue to control the number of threads that read the database write cache. For example, a key allows only one thread to query the data and write the cache, and other threads wait. (using ZK to do distributed locks) This is a common way to limit the application layer of database access. 2: Different keys, set different expiration time, so that the cache fails to the point of time as evenly as possible. 3: Do two-level cache, A1 for the original cache, A2 for the copy cache, A1 failure, can access the A2,A1 cache expiration time set to short-term, A2 set to long (this point is added). Multi-level caching can exist in strategy 4: For the hot key to break down processing such as a key_001 key_002 to distinguish.

Question 3:incast congestion

The problem is that webserver reads a lot of memcache at once, and at the same time returns to the network port to burst or to produce a packet loss.

Resolution strategy: Facebook is a mechanism to introduce a similar TCP-like sliding window. I think some of the better MEMCAHCE clients should have a solution.

Issue 4 Scaling issues

The above for the small-scale scenario is not a problem, but if the scale expanded, such as the cache cluster more, the database is also more, then to the management problems, especially the size of the FB cache

Solution:

Cached data must is invalidated after database updates

? Solution:tail the MySQL commit log and issue deletes BasedOn transactions that has been committed
? Allows caches to being resynchronized in the event of a problem

Facebook internal Usage Mcsqueal This component is used to disable caching based on Commitlog.

Baixia says:

NetEase also synchronizes the cache in a similar way, and our components are called NDC, which of course is not open source. In fact, subscribe to Commitlog and then run to Kafka have related programs to consume Kafka log and then sync the cache.

Open source, Ali Open source Otter components, the author yy can be otter related Commitlog synchronization to achieve the relevant needs.

Sync across Centers

With MySQL-based Commitlog cache synchronization, synchronizing across data centers is the same as just setting up a library.

The problem now is that when webserver is written, it needs to write master and fail the local memcache

However, if there is another request to read the cache at this time, the discovery is empty, then the set cache rescue problem is found from replicadb, because Masterdb has not synchronized the data because of the delay problem

Solution:

Add a flag variable Remote-marker solve the problem if you have a set variable read from Masterdb instead of from replicadb

Best practice recommendations from Facebook:

(1) Separating cache and persistent storage systems allows us to independently scale them. Try to split the cache and storage system for idempotent expansion

(2) Features that improve monitoring, debugging and operational efficiency is as important as performance. Can not ignore monitoring D ebug ease of operation
(3) Managing stateful are operationally more complex than stateless ones. As a result keeping logic in a stateless client helps iterate on features and minimize disruption. Stateful hard to maintain, try to keep the client stateless
(4) The system must support gradual rollout and rollback of new features even if it leads to temporary heterogeneity of FE  Ature sets. Support rollback
(5) Simplicity is vital. Kiss principle Keep it simple and stupid

In fact, Facebook's mcrouter tool is perfect throughout the best practice recommendations, and I'll continue to update the blog post about the use of Mcrouter

Https://github.com/facebook/mcrouter

Play a little bit:

For Memcache's operations, I want to do it.

Requirement 1: Master-slave high availability, cache downtime does not cause penetration
Requirement 2: Smooth expansion of hot and cold clusters, when people expand new cache nodes or replace cache upgrade cache Stop cache
Requirement 3: Dynamically load the configuration without rebooting
Requirement 4: Multilevel cache, can support multilevel such as Local
Requirement 5: Flexible multi-cluster scheduling, multi-cluster traffic scheduling, prefix scheduling, etc.

These requirements Facebook Open-source mcrouter can be implemented

Advertising time

NetEase Operations and Account Center is recruiting including application operations, system operations, database operations, operations and development related positions:

Interested in joining partners please resume [email protected]

Scaling Memcache at Facebook