Go deep into Facebook message Application Server

Key points:

1. Facebook unified messaging system (email, SMS, chat, message, etc );
2. Hbase is used as the backend storage facility. Each user data is stored in a single row of hbase, and each object (folder, topic, message, and so on) is stored in its own hbase column;
3. It involves the image processing infrastructure of haystack;
4. Use Apache Lucene to maintain the reverse index list;
5. Mirrors the real-time chat and inbox information of about 10% of users to the test cluster, and performs the test through dark launch.

Facebook messages is one of the most technically challenging products we have ever created.

When we published Facebook messages, we mentioned that we needed to build a dedicated application server to manage its infrastructure.

We recently discussed how to deal with all the communications from email, SMS, Facebook chat, and inbox.

Today, we will go deep into the core of the Message application server.

Application Server business logic

The Application Server integrates many Facebook services and protection (shields) from the complexity of various terminals. It provides a simple interface for the client to process standard messages, including creating, reading, deleting, updating messages, and inbox.

The following is the process of each part.

When you create a new message or reply to a message, the application server sends the message to the recipient on behalf of the sender. If the recipient uses its email address, the server obtains the attachment (if any) through haystack, constructs an HTML subject, and creates an rfc2822 message.

Output stream 

When a message is sent to a user, if the address is a reply processing address, the server obtains the correct recipient information from the existing email address and input message. The server finally transmits the message to the user's mailbox, runs all necessary preprocessing and post-processing processes, and decides the message routing based on folders and topics of multiple signals.

Input stream 

When reading a message, the server obtains multiple statistics about the user's mailbox, such as capacity, message, subject, and reply count, and number of friends. At the same time, it also obtains statistics and attributes related to folders, lists of topics (folders, attributes, authors, keywords, etc.) for various search conditions, theme attributes, and other messages for this topic.

When a message is deleted, the server marks the message and topic to be deleted. An offline task clears the message content.

When a message or topic is updated, the server changes the attributes of the message or topic, such as the reading and arrival statuses and tags. It also processes requests from multiple users to subscribe to and unsubscribe to topics.

Manage group topics

Facebook messages uses a chat room model to manage group message topics. Users can join (subscribe) and quit (cancel subscription ).

This model is required when the email address is the specified recipient of this topic. The application server creates a reply processor, similar to the chat room ID. When an email recipient replies to a topic, the message is sent to the reply processor address.

To optimize read performance and simplify migration and backup processing, message topics are stored in a denormalized mode. As a result, each user has a copy of topic metadata and messages. The server broadcasts subscriptions and unsubscribes to events, and synchronizes topic metadata subscribed and replied to by all recipients in a decentralized manner. The server also manages situations where users still use the old inbox or subscribe through their email addresses.

Cache user metadata

When a user accesses the inbox, the app server loads the most frequently used user metadata (also called activity metadata) and saves them in the recently used cache (a least recently used cache, that is, LRUAlgorithm). Subsequently, requests of the same user are quickly processed through a small amount of hbase queries.

We need to reduce hbase queries because hbase does not support join. to process a Read Request, the server may need to search Multiple indexes and match metadata and message bodies in a separate hbase query. Hbase optimization is embodied in write operations rather than reading data. user behavior usually has good time and regionality (good temporal and spatial locality), so the cache can help solve this problem and improve performance.

We have also made a lot of efforts to improve the cache effectiveness by reducing user memory usage and transferring to the fine-grained mode. We can cache 5%-10% of the user volume and 95% of the Active metadata cache hit rate. We cache frequently accessed data on the global memcache layer (for example, displaying the number of unread messages on the Facebook homepage ). When a new message arrives, the application server marks the cache as (dirties) (Note: dirties indicates the data modified but not written to the data file ).

Synchronization

Hbase provides limited support for transaction isolation. Multiple updates for the same user may occur simultaneously. To resolve potential conflicts between them, we use the application server as the synchronization point of user requests. A user provides services by a dedicated server at any given time. In this way, the same user request can be synchronized and executed in a completely isolated way (fashion) on the application server.

Storage Mode

MTA proxy feature attachments and a large number of message entities, which are stored in haystack before they can reach the application server. However, metadata, including indexed data and small message bodies, is stored in hbase and maintained by the application server. Each user's inbox is independent of any other user; user data is not shared in hbase ). Each user data is stored in a separate row of hbase, which contains the following parts:

Metadata entity and Index

Metadata objects include attributes of inbox objects, such as folders, topics, and messages. Each object is stored in its own hbase column. Unlike relational databases (RDBMS), hbase does not provide local support for indexing. We maintain secondary indexes at the application level and store them in separate columns as key/value pairs.

For example, to query "loading unread threads on the second page of the other folder," the application server first searches for the metadata index to obtain a list of qualified topics, then retrieve the metadata entity of the specified topic and construct a response with their attributes.

As we mentioned earlier, caching and effective pre-loading can reduce the amount of hbase queries for better performance.

Activity Log

Any updates (such as posting and deleting messages, marking the topic as read, etc.) in the user's mailbox are immediately added to the column in chronological order. This is called an Action Log ). Small message entities are also stored in activity logs.

We can construct or restore the current status of the user's mailbox by replaying activity logs. We use the ID of the last activity log to play back the metadata entity and the version of the index. When a user's mailbox is loaded, the app server compares the metadata version and the last activity log ID. If the metadata version is delayed (lags behind), the email content is updated.

The storage of activity logs brings great flexibility at the application level:

• We can seamlessly switch to a new mode through playback of Activity logs and generate new metadata entities and indexes through an offline mapreduce task or online application server itself.
• We can execute a large number of hbase asynchronous writes in a batch to save network bandwidth and reduce hbase compression costs.
• It is a standard protocol for exchanging persistent data with other components. For example, we write activity logs to the record log for application-level backup. This porting pipeline converts users' old inbox data to Activity logs and generates metadata and indexes through offline mapreduce.

Search Index

To support full-text retrieval, we maintain a reverse index from the keyword to the matching message. When a new message arrives, we use Apache Lucene to parse and convert it into a (keyword, Message ID, positions) tuples (tuples, then, add the data to the hbase column in ascending order. Each keyword has its own column. All messages, including chat records, emails, and text messages, are indexed in real time.

Dark launchTest

Application Server is a brand new software built from scratch, so we need to monitor its performance, reliability, and scalability before pushing it to 0.5 billion users. We initially developed a stress testing robot (robot) to generate simulated requests, but we found that such results may be affected by other new factors, such as the message length, distribution of different types of requests, distribution of user activity, and so on.

To simulate a real product load, we created a dark launch, which mirrors the real-time chat and inbox information of about 10% users to the test cluster. Dark launches helps us discover more performance problems and identify bottlenecks. We also use it as a persuasive indicator to evaluate many of our improvements. Next, we will continue to work hard to provide a new messaging system for all our users.