Pinterest on combat experience: how to achieve a zero to tens of billions of monthly visit in two years

Pinterest has maintained an exponential growth that will be doubled every 1.5 months. In two years, they achieved PV from 0 to tens of billions of, from the beginning of the two founder plus an engineer to now more than 40 engineers, from a small MySQL server to 180 web Enigne, 240 API Enigne, 88 MySQL DB (Cc2.8xlarge, each DB will be configured with a slave node), 110 Redis Instance and 200 Mmecache Instance.

In a keynote speech called "Scaling Pinterest", Pinterest's Yashwanth Nelapati and Marty Weiner told us about this dramatic process. Of course, the expansion to the current scale, Pinterest in many choices inevitably took a lot of detours, and Todd Hoff that the most valuable experience should be attributed to the following two points:

If your architecture is dealing with the problems of growth, simply invest in more hosts, your architecture is full of gold. When you put things to the limit, these technologies will fail in different ways, which leads to a particular preference for tools: Mature, simple, excellent, well known, more user-favored, better supported, stable and outstanding performance, usually without fault, and free. Using these standards, they chose MySQL, SOLR, Memcache, Redis, Cassandra, and abandoned MongoDB.

The same two points are related, and tools that conform to the second principle can be extended by putting in more hosts. Even if the load increases, the project will not have a lot of failures. Even if there is a problem that is difficult to solve, at least one community can find a solution to the problem. Once you choose an overly complex and picky tool, the road to expansion will be full of thorns.

It should be noted that all of the tools they choose depend on adding fragmentation to scale, not through clustering. It also explains why fragmentation is better than clusters and how to slice them, and these ideas may be unheard of before.

Here's a look at the phased timeline for Pinterest expansion:

Project background

Pins is a picture of other bits and pieces of information that shows important information about the customer and links to its location. Pinterest is a social network where you can follow (focus) on others and board. Database: Pinterest users have board, and each board contains Pin;follow and Repin interpersonal and authentication information.

1. March 2010 Release-Looking for the times of real

At that time, you don't even know what kind of product you need to build. You have ideas, so you quickly iterate and evolve. And eventually you'll get some very small MySQL queries that you've never done in real life.

Some numbers in the early stages of Pinterest:

2 founder 1 Engineers Rackspace 1 Small Network engine 1 small MySQL database November 2011

is still small, the number of products that evolve through user feedback is:

Amazon EC2 + S3 + cloudfront 1 NGinX, 4 Web engines (for redundancy, not all load) 1 MySQL DB + 1 Read Slave (for master node failure condition) 1 task Queue + 2 task P Rocessors 1 MongoDB (for counting) 2 engineers

2. Through the 2011-The age of experimentation

The pace of crazy growth, basically doubled every 1.5 months.

As you grow so fast, every day of the week you may need to break or throw something away. At this time, they read a large number of papers, all of which illustrate the need to add only one host problem will be resolved. They started to add a lot of technology and then had to give it up. So there were some very strange results: Amazon EC2 + S3 + cloudfront 2NGinX, Web engines + 2 API engines 5 functionally sharged MySQL DB + 9 Read Slaves 4 Cassandra Nodes membase Nodes (3 separate clusters) 8 Memcache Nodes Redis Nodes 3 Task routers + 4 task processors 4 Elasti C Search Nodes 3 Mongo clusters 3 Engineers 5 primary database technology, only for the independence of the data. The growth was so fast that MySQL was on the run, and all the other technologies reached the limit. When you use things to the limit, these technologies will go wrong in different ways. Begin to discard some of the technology and reflect on what you need, basically redo all the architecture.

3. February 2012-The Age of maturity

After

all the schemas, the system presents the following status: Amazon EC2 + S3 + Akamai, Elb Web engines + API engines the MySQL DBs (m1.xlarge) +, each database is equipped with slave node 59 Redis Instances Memcache Instances 1 redis Task Manager + task processors sharded SOLR 6 engineers are now using the technology that is fragmented MySQL, Redis, Memcache and SOLR, a bit of it is that these technologies are simple and mature. The growth of the network is still at its previous pace, and the iphone is moving higher.

4. October 12, 2012-Harvest season

About 4 times times the size of January.

Now the data are: Amazon EC2 + S3 + Edge Cast,akamai, Level 3 180 Web engines + the API engines the MySQL DBs (Cc2.8xlarge), also each database has a slave node 110 Redis Instances Memcache Instances 4 redis Task Manager + task processors sharded SOLR 40 engineers (still growing) it should be noted that today's architecture is approaching perfection, Responding to growth requires more than just putting in more hosts. The current has started moving to SSD

(Responsible editor: admin)