Stack Overflow 2016 Latest Architecture Quest

This article mainly reveals the technical architecture of Stack Overflow as of 2016.

First give an intuitive data, let everyone have a preliminary impression.

Compared to November 2013, Stack Overflow in February 2016, the statistical data has a large change, the following gives the February 09, 2016 Day data, as follows:

• Number of HTTP requests 209,420,973 (+61,336,090)
• Page load times 66,294,789 (+30,199,477)
• HTTP traffic is sent with 1,240,266,346,053 (+406,273,363,426) bytes (1.24 TB)
• Total Data received 569,449,470,023 (+282,874,825,991) bytes (569 GB)
• Total data sent 3,084,303,599,266 (+1,958,311,041,954) bytes (3.08 TB)
• Number of SQL queries (HTTP requests) 504,816,843 (+170,244,740)
• Redis hits 5,831,683,114 (+5,418,818,063)
• Elastic Number of queries 17,158,874 (data not included in 2013)
• Number of label engine requests 3,661,134 (+57,716)
• SQL query takes 607,073,066 (+48,848,481) milliseconds (168 hours)
• Redis hit Time 10,396,073 (-88,950,843) milliseconds (2.8 hours)

It's hard to imagine that the. NET technology architecture can reduce processing time by 757 hours (compared to 2013) with 61 million requests per day. These improvements have benefited not only from the upgrading of hardware equipment in the early 2015, but also with the performance optimization of the software.

So what has changed in hardware over the last two years? The following is a list of hardware so far:

• 4 database servers (Microsoft SQL Server), of which two update hardware configurations
• 11 Web Server (IIS), hardware configuration updated
• 2 distributed cache and Message Processing server (REDIS), hardware configuration updated
• 3 Application servers (with tag engine capability), of which two are new hardware configurations
• 3 Search servers (ElasticSearch), configured with 2013 years
• 4 Load Balancer servers (HAProxy), with two newly added CDN accelerator services to support CloudFlare
• 2 Network switches (each Cisco Nexus 5596 + Fabric extenders and upgrade Nic 10Gbps) 2 Fortinet 800C (alternative to 2 Cisco 5525-x ASA firewalls)
• 2 Ciso ASR-1001 routers (replaces 2 Cisco 3945 routers)
• 2 Ciso asr-1001-x Routers

What do you need to do to support the Stack Overflow operation? In fact, compared with 2013, there is no significant change, just do the above mentioned hardware upgrade and program performance optimization.

The existing system is generally not completely isolated, Stack Overflow is not listed outside. A picture wins thousands of words, the following gives the overall structure of the Stack Overflow. This article gives only the highlights of the logical architecture of hardware finishing, and the detailed hardware details will be presented in the next article.

Figure 1 shows a physical picture of rack a (upgraded in February 2015).

Figure 1

Now, let's give the logical architecture diagram of the main system, 2.

Figure 2

Basic rules

the general rules of the global are given first :

• Everything needs to be backed up
• At least 2 x 10Gbps bandwidth for all servers and network switches
• Two power supplies for all servers (with UPS power reserve)
• All servers are redundant on racks A and B
• All servers and services are offsite (New York room and Colorado State room)

　　Network Services

First, users go to the Stack Overflow website to browse through the Internet. To allow users to browse the site faster, Stack Overflow uses CloudFlare CDN acceleration. The CloudFlare service is used here because their CDN servers are all over the world.

Immediately after that, the user's HTTP traffic passes through the four ISP providers (level 3,zayo,cogent and Lighttower), over four routers. Stack Overflow uses standard Border Gateway Protocol (BGP) to equalize all traffic so that users can open the site more efficiently. Nick Craver, an engineer at Stack Overflow, recommends using one of the two Gbps MPLS in an offsite data center to quickly recover and replicate data in the event of an emergency.

　Load Balancing (HAProxy)

Load Balancer uses the HAProxy 1.5.15 and CentOS 7 and joins the Secure Transport Layer Protocol (TLS/SSL) in HAProxy. Subsequent upgrades are HAProxy to version 1.6 to support HTTP/2.

The load balancer comes with 2 pairs of 10Gbps networks. Stack Overflow effectively resolves Secure Sockets Layer (SSL) issues by adding memory. Cache Secure Transport Layer Protocol (TLS) sessions to memory for reuse, which reduces the amount of repetitive computations for the same client connection, and the speed and cost of reaching an elevation session. Moreover, the RAM is quite cheap, and achieves a double win effect.

The settings for the load balancer are fairly straightforward. They listen to the various IPs and route the distribution. Stack Overflow also does a load balancer current limit and monitors HAProxy logs to make timely alarms.

Web Layer Schema (IIS 8.5,asp.net MVC 5.2.3, and. Net 4.6.1)

Stack Overflow through the Load Balancer layer to import traffic to 9 Web servers ("primary" server), the other two to do site metadata and other environmental management. In addition to meta.stackoverflow.com and meta.stackexchange.com, the stack Overflow, careers, and stack Exchange website businesses are running on the "primary" server.

In the monitoring platform Opserver can be seen, Stack Overflow in the WEB layer distribution, see Figure 3

Figure 3

A more intuitive look at the corresponding Web server graphic display, see Figure 4

Figure 4

The service layer (iis,asp.net MVC 5.2.3, Net 4.6.1, and HTTP. SYS)

In the overall logical frame composition can be clearly seen, next to the WEB layer is the service layer (deployed on Window server Windows 2012R2). There are two important features: the tag Application server (HTTP. sys) and the API (based on IIS). In order to improve these two services do a lot of redundancy, but not more than 9 times times the redundancy. For example, loading all pages from a database and corresponding tags changes (every n minutes (currently set to 2 minutes) is time consuming. It only needs to be loaded three times to ensure security. The Stack Overflow is also optimized at the hardware level. Tag App service is a more complex topic, here simply, when you visit/questions/tagged/java use Tag app service. All/search and navigation are also used by these data services.

Cache & Publish/subscribe (Redis)

Stack Overflow uses Redis,redis server 256GB of memory in the cache layer and is deployed in a master/slave structure, despite the 160 million ops per month, the CPU utilization per instance is below 2%.

Figure 5

The Redis server has a l1/l2 cache and the HTTP cache settings for the Web service are in the first-level cache L1, and the Redis cache is L2 in level two cache. Redis values in the level two cache are stored in the PROTOBUF format and parsed with Protobuf-dot-net when the user accesses the misses in the first-level cache L1. The Stackexchange.redis used by Redis customers (Stack Overflow internally implemented and open source). If the Web service misses both L1 and L2 level two caches, it will go directly to the original data source (for example, database queries, API callbacks, etc.) and cache the obtained results locally and in Redis, when the other service misses the L1 cache and goes to level two cache L2/redis. Save access time to call database queries or API callbacks.

Most running question and answer sites have their own l1/l2 cache, with L1 cache Key prefixes, L2/redis cache database IDs.

Although Redis is primarily used for caching, but also plays a role in consuming and subscribing, Redis can push a message and then other subscribers to subscribe to the message (including downstream Redis from the library in the subscription message).

　Websockets (NetGain)

Websockets real-time push messages (e.g., top bar notifications, polls, new answers and comments) to the user.

The Sockets server runs at the Web layer, and NetGain is a lightweight, high-performance, real-time, open source messaging middleware implemented by Stack Overflow. Peak times up to 500,000 concurrent websocket connections.

Shows a week of websocket concurrency:

Figure 6

　　Search (Elasticsearch)

Nick Craver, the engineer at Stack Overflow, says there is no exciting part of the search layer. The Web tier employs Elasticsearch 1.4 and internally implements a high-performance stackexchange.elastic client, which is not open source. Stack Overflow uses Elastic to inquire about questions and answers.

Each data center has a Elasticsearch cluster with three nodes, each with its own index. All three Elasticsearch clusters use SSD storage, 192GB memory and dual 10Gbps NICs.

Stack Overflow uses Elasticsearch instead of the previous SQL full-row index, the main factor being: Elasticsearch scalability and low cost.

Database (SQL Server)

SQL Server is the only source database for Stack Overflow, and all Elastic and Redis data comes from SQL Server. Using Microsoft's SQL Server monitoring component, AlwaysOn availability Groups deploys two SQL Server clusters. Each cluster has one main library, one data backup in New York, and another data backup in the Colorrado data center. All backups are asynchronous replication.

First cluster hardware configuration: Dell R720XD Server, 384G memory, 4TB SSD storage, dual 12-core CPU, second cluster hardware configuration: Dell R730XD Server, 768G memory, 4TB SSD storage, dual 8-core CPU.

All databases in the past 24 hours CPU monitoring Figure 7, most of the time the CPU utilization is low, occasionally doing the cache task is higher. In the figure ny-sql02 and 04 are the main libraries, and 01 and 03 are the backup libraries.

Figure 7

Throughout the full text, Stack Overflow's overall architecture has not adopted those very high-end technologies, but it has created an IT industry's most popular question and answer site, which is very good. Each of these technologies is deeply researched and open source is shared with the community, and domestic companies can get some inspiration from it.

Stack Overflow 2016 Latest Architecture Quest