Sina Weibo architecture and relational storage

The web of the hastily organized

1. Build a scalable microblogging architecture Timyang Sina Weibo technology architect

2. From blogs to Weibo

3. Blog function posted view message Contentmanager System

4. Blog technology, LAMP MySQL master/slave Memcached PHP CDN

5. Weibo Weibo, product real-time focus on relationship information aggregation

6. Information Aggregation

7. Information aggregation microblogging two kinds of information aggregation design mode push (push) pull

8.Push Weibo as mail inbox: Received Weibo Outbox: Posted Weibo: Save to all fans Inbox (Heavy) View: Direct access to inbox (light)

9.Push (figure) User A updateaction Inbox (Append to 1 ' s hometimeline) Inbox (Append to 2 's home timeline) Inbox (Append to 3 ' shome timeline) followers of User A = 1, 2, 3

10.Push Advantages: Simple implementation, preferred disadvantage: distribution volume

11.Pull Published: Save to Yourself Outbox (light) View: All objects of interest inbox (Heavy)

12.Pull user I Get home_timeline Outbox (statuses sent by A) Outbox (statuses sent by B) Outbox (statuses sent by C) User I ' Sfollowing List = A, B, C

13.Pull Advantages: Save Storage Disadvantage: Large computational capacity

14. Weibo is a message distribution system that can be pushed or pulled to achieve

15. Architectural challenges: peaks-such as New Year's Eve, Spring Festival

16. Volume of requests if published 50 million/day average: 578 bar/S Design system capacity: 2,000?

17.IO bottleneck Peak: 5,000–10,000? 100,000?

18. Consequence Latencydb read Timeout front-end timeout (503 error) solution.

19. Asynchronous design out of sync wait for messages to be stored in Message Queuing (MessageQueue) Lightweight Publishing

20.MQ Products Kestrel by Twitter RabbitMQ, an Erlang Queue Servermemcacheq on the massive use of Sina Weibo projects

21.Memcacheq based on BerkeleyDB, stable and reliable memcachedprotocol, rich clientlibrary easy to monitor (Statsqueue) only 2 commands: Get/set

22. Avoid single point of failure core service, need to avoid separate failure methods use multiple Memcacheq pool Get operations: poll all server set operations: Randomly select a design that does not require additional complex "architectures"

Common advantages of the 23.MQ approach offlinework application of unbalanced demand-volume decoupling asynchronous communication principle

24. Using the MQ principle to calculate overhead is greater than message distribution overhead

25. Architectural challenges: real-time

26. The more important events, the more you want real-time sex

27.The value of the tweet decreases exponentially with time Johnkalucki, Twitter http://t.sina.com.cn/pub/star#a_ty

28. Solution cache Ramis the new disk

29.Local Cache Memcached Database Buffer/cache

30.LAMP, cache= optional Layer Cache Center after the new problem

31. Capacity problem TB-level ideas: compression Quicklzlzo without gzip

32. Single point problem single point of failure, SIGSEGV how to deal with 1. Consistent Hash 2. Read-through Cache

33.Consistent Hash Principle Advantage oscillation minimum

34.read-through Cache

35.read-through and Write-through products or projects MySQL memcachedudf Cache money for Ruby on Rails or wrap a proxy fo R the DB driver,in any language

36.Evictions problem Evections:cache data is kicked performance nightmare latency one of the sources of production

37. How to avoid evictions planning cache capacity separate permanent data from temporary data without using random characters as key

38.Multiget problem whenmemcached servers is CPU bound, adding more memcached serversdoesn ' t help serve more requests. -Jeff Rothschild, Vice presidentof technology at Facebook

39.Cache Challenge: Multigethole

40. Workaround Memcachedreplication

41. Architectural Challenges: mass storage

42. Architectural challenges: Domestic network bandwidth issues

43. Geographic distribution for the following reasons, the need for distributed deployment access speed IDC the core of the unavailable fault distribution is the data distribution

44. Data geographic distribution principle Master-slavemaster-master 2pc/3pc Paxos http://timyang.net/data/multi-idc-design/

45. Geographical distribution of the scheme Mysqlmaster/slave Dynamo/cassandra PNUTS

46. Architecture Challenge: API Traffic

47. Take Sina Weibo open platform as an example