Sina Weibo platform architecture for hundreds of millions of users

Sina Weibo platform architecture for hundreds of millions of users

Preface Sina Weibo published March 2014 million monthly active users (MAU) in 0.143 billion and 2014 million Weibo posts in the first minute of the new year in 808298. such a huge user scale and business volume, high availability (HA), High concurrent access, and low latency are required. The first generation of Weibo platform architecture is LAMP, the database uses MyIsam, the backend uses php, and the cache is Memcache. With the growth of application scale, the second-generation architecture has modularized, serviced, and componentized business functions, and the background system has been replaced with Java from php to form an SOA architecture, it has supported the business development of the Weibo platform for a long time. On this basis, after a long period of reconstruction, online operation, thinking and precipitation, the platform formed a third-generation architecture system. Let's first look at the core business diagram of Weibo (as shown below). Is it very complicated? However, this is a simplified business diagram that cannot be simplified. the third-generation technology system aims to ensure fast, efficient, and reliable release of new product features on Weibo's core business. Third-generation technical system The third-generation technology system of Weibo platform uses orthogonal decomposition to establish a model: in the horizontal direction, it adopts a typical three-level layered model, that is, the interface layer, service layer and resource layer; in the vertical direction, further subdivided into business architecture, technical architecture, monitoring platform and service governance platform. The overall architecture of the platform is as follows: As shown in, orthogonal decomposition divides the entire graph into 3*4 = 12 regions, and each region represents the intersection of a horizontal dimension and a vertical dimension, define the core functions of the region accordingly. for example, area 5 mainly completes the technical architecture of the service layer. The following describes in detail the design principles of the horizontal and vertical directions, especially the technical components in 4, 5, and 6 and their roles in the entire architecture system. Horizontal hierarchy The horizontal dimension is very basic in the design of large and medium-sized internet backend business systems and is embodied in each generation of technical systems on the platform. Here is a brief introduction to pave the way for the subsequent vertical dimension extension: [Size = 15.4545450210571px]1,The interface layer interacts with Web pages and mobile clients, and defines a unified interface specification. The three most core interface services of the platform are content (Feed) services, user relationship services, and communication services (single private message, group chat, and group chat ). [Size = 15.4545450210571px] [Size = 15.4545450210571px]2,The service layer mainly modularize and servitize core services. here there are two types of services, one being atomic services, which are defined as service modules independent of any other services, for example, common short chain services and sender services belong to this category. As shown in the figure, swimming channels are isolated to indicate their independence. The other type is composite services. services are completed through the combination of various atomic services and business logic, such as Feed services and communication services. in addition to their own business logic, it also relies on short chain, user, and sender services. [Size = 15.4545450210571px] [Size = 15.4545450210571px]3,The resource layer mainly stores data models, including general cache resources Redis and Memcached, as well as persistent database storage MySQL, HBase, distributed file system TFS, and Sina S3 services. The horizontal hierarchy has a feature that the dependencies are from top to bottom, and the services on the upper layer are dependent on the lower layer. this creates a simple and direct dependency relationship. In contrast to the layered model, Weibo servers mainly include three types: front-end servers (providing API services) and queue servers (processing upstream business logic, mainly writing data) and storage (mc, mysql, mcq, redis, HBase, etc ). Vertical extension technical architecture With the development and optimization of the business architecture, the platform R & D has implemented many excellent middleware products to support core businesses. These middleware products are generated by the business Drive. as the technical components become richer and richer, A complete platform technical framework is formed, which greatly improves the product R & D efficiency and business operation stability of the platform. Different from the relationship between the upper layer and the lower layer in the horizontal direction, the vertical direction uses the technical framework as the foundation support point to drive the impact on the business architecture, monitoring platform, and service governance platform on both sides. The following describes the core components. Interface Layer Web V4 framework The interface framework simplifies and standardizes the business interface development work, packs common interface layer functions into the framework, and adopts the Spring aspect-oriented (AOP) design concept. The interface framework is developed based on Jersey and defines the interface (url and parameter) based on annotation. it has the built-in Auth, frequency control, access log, and downgrade functions to support the interface layer monitoring platform and service governance, there are also automated Bean-json/xml serialization. Service layer framework The service layer mainly involves the RPC remote call framework and message queue framework, which are the two most widely used frameworks of the Weibo platform in the service layer. MCQ Message Queue Message queue provides a first-in-first-out communication mechanism. in the platform, the most common scenario is to asynchronously write data operations into the queue, and the queue processing program reads data in batches and writes data to the database, the asynchronous mechanism provided by message queue accelerates the response time of the front-end machine. second, batch DB operations also indirectly improve the performance of DB operations. in another application scenario, the platform uses message queue, provides real-time data to search, big data, and commercial operation departments. The MCQ (SimpleQueue Service Over Memcache) message queue Service is widely used in the Weibo platform. based on the MemCache protocol, message data is persistently written to BerkeleyDB and only two commands are available: get/set, at the same time, it is also very easy to monitor (stats queue), there are a wide range of client libraries, online operation for many years, the performance is much higher than the general MQ. Motan RPC framework The underlying communication engine of Weibo's Motan RPC service uses the Netty network framework. the serialization protocol supports Hessian and Java serialization. the communication protocol supports Motan, http, tcp, mc, and so on, the Motan framework is widely used internally and has mature technical solutions and robustness in terms of system robustness and service governance, the High Availability and Load Balance policies are implemented based on the Config configuration management service (supports flexible FailOver and FailFast HA policies, as well as Load Balance policies such as Round Robin, LRU, and Consistent Hash ), in terms of service governance, complete service call chain data, service request performance data, Response Time (Response Time), QPS, and standardized Error and Exception log information are generated. Resource Layer framework The resource layer has many frameworks, including Key-List DAL middleware for MySQL and HBase encapsulation, customized counting components, and proxies for distributed MC and Redis, I have many experiences in these fields. here I will share the object library and SSD Cache components of the platform architecture. Object Library Object library supports convenient serialization and deserialization of object data in Weibo: during serialization, the object in JVM memory is serialized into HBase and a unique ObjectID is generated. when you need to access this object, read through ObjectID. the object library supports any type of objects and supports PB, JSON, and binary serialization protocols, the largest application scenario in Weibo defines videos, images, and articles referenced in Weibo as objects. dozens Of object types are defined and standard object metadata Schema is abstracted, the object content is uploaded to the object storage system (Sina S3), and the object metadata is saved in Sina S3. SSDCache With the popularization of SSD hard disks and superior IO performance, SSD hard disks are increasingly used to replace traditional SATA and SAS disks. There are three common application scenarios: 1) replace the hard disks of mysql databases, currently, the community does not have a MySQL version optimized for SSD. in this case, upgrading the SSD hard disk can also increase IOPS by about 8 times; 2) replacing the Redis hard disk to improve its performance; 3) it is used in CDN to accelerate loading of static resources. The Weibo platform applies SSD to distributed Cache scenarios. the traditional Redis/MC + Mysql method is extended to the Redis/MC + SSD Cache + Mysql method, and the SSD Cache is used as the L2 Cache, first, it reduces the cost and capacity of MC/Redis, and solves the database access pressure caused by database penetration. Vertical monitoring and service governance As the service scale and business become more and more complex, it is difficult for business architects to accurately describe the dependencies between services, and service management and O & M become more and more difficult. in this context, based on google's dapper and twitter's zipkin, the platform has implemented its own large-scale distributed tracing system WatchMan. WatchMan large-scale distributed tracing system Like other large and medium-sized Internet applications, the Weibo platform consists of a large number of distributed components. after a user reaches the application server through each HTTP request of a browser or mobile client, it passes through many business systems or system components and leaves a footprint (footprint ). However, the scattered data is of limited help for troubleshooting or process optimization. For such a typical cross-process/cross-thread scenario, it is particularly important to collect and analyze such logs. On the other hand, collecting performance data for each footprint and performing traffic control or degradation for each subsystem based on policies is also an important factor to ensure the high availability of the Weibo platform. It is necessary to track the complete call link of each request, collect performance data of each service on the call link, and track all errors and exceptions in the system; through computing performance data and comparison of performance indicators (SLAs) and then feedback to the control flow, based on these goals, the Watchman system of Weibo was born. A core principle of the system design is non-invasivenss (non-invasivenss): as a non-business component, the system should be as little or no intrusion into other business systems as possible to maintain transparency to users, this greatly reduces the burden on developers and the access threshold. Based on this consideration, all log collection points are distributed in the technical framework middleware, including the interface framework, RPC framework, and other resource middleware. WatchMan is a framework built by a technical team. in all business scenarios, O & M is based on this system to improve the monitoring platform, and business and O & M use this system together to complete distributed service governance, this includes service expansion and contraction, service downgrade, traffic switching, Service release, and gray scale. End Now, the technical framework plays an increasingly important role in the platform, driving the platform's technical upgrades, business development, and system O & M services. This article is limited by space restrictions and has not been introduced, we will continue to introduce the design principles and system architecture of Core middleware in the future. About the Author Wei Xiangjun (@ Wei Xiangjun _ Weibo), graduated from Beijing University of Posts and Telecommunications, and is currently a Weibo platform architect. he has successively worked in technology R & D at Microsoft, Jinshan cloud, and Sina Weibo, focuses on system architecture design, audio and video communication systems, distributed file systems, data mining, and other fields. [Size = 15.4545450210571px] this article