Original address: http://2014.54chen.com/blog/2014/03/05/ihaveadream/
Overall thinking
Summing up these years of experience, the direction of the evolution of the framework to choose, roughly to achieve the following objectives:
- Rapid development deployment (five minutes write out a tested Hello World and can be accessed/invoked and accessible on the public network)
- Natural scalability (business layer stateless, as far as possible all the last)
- Automation (out of memory, in addition to the alarm, you should automatically add a machine to enter; New project, the base code should not write, automatically generated can be)
- Framing (a common-level thing as framed as possible, a layer like log, counter, trace, should not need to be developed and then write a line of code is opened by default)
- Quantization (all calls should have percentile with RPS, transparent feedback quality of service, tracking system can simulate the user inside the system)
- Isomorphism (because the cost of two sets is huge, the whole should be more and more in the same language)
- Hardware virtualization (the entire platform should be transparent to the process under hardware conditions)
- Versioning, grayscale (all software should have a clear version on the line, and the online process is a bit grayscale)
Pure hand-painted took some time, this article is described in this diagram from top to bottom.
User
In the mobile Internet environment, the user will be divided into good network users and bad network users, we want to do all the bad network users do everything possible to provide the right link and reliable DNS.
Access Layer
At the code level of the access layer, you need to prepare the Client-server suite, which means that you need an understanding of android\ios at the same time. and server-side development team, specifically to create a network suite.
- The goal of this layer is to let client developers no longer care about network protocol issues.
Business Access Layer
The goal of this layer is to well-informed mobile deployment of traffic, often everyone's plan is LVS, or F5. Bigger on the upper point, and then some traffic analysis equipment, when there is a sudden increase in use to find problems.
Business Layer
In a unified business framework, to complete the various flexible organization of the biz logic, here is related to the impact of heterogeneous systems on a large company.
- If 80% of the people are using Java, or the full use of Java, because it means that the remaining 20% in other languages of the students, it is possible to the 80% of the students to do the basis of all realized once.
- Heterogeneous will inevitably lead to some modules do not work perfectly, such as the subsequent RPC, configuration management, monitoring alarm, tracking system and so on.
RPC Framework and queues
The two work together to achieve the data in IDC, the difference is that one is synchronous, one is asynchronous.
- The unified RPC Framework benefits do not have to be spoken.
Configuration Management
Zookeeper elected the best role, on the point of the scale of the service will basically have ZK figure.
Log system
A unified logging system makes it easier to get all the data you need for future development. The characteristics of the log system: fast, tolerant network errors, simple deployment, stable process, can be horizontally extended.
- scribe and Kafka are good choices.
- The final log here may need to be placed in HDFs or hbase for hive queries, otherwise it is not easy to use the log after a large amount of data.
Monitoring alarm System
Ganglia and Nagios are still the best open source management software.
- Consider the need to monitor and alert public perfcounter that are recorded in the business framework by default.
Tracking System
The tracking system is an essential tool when there is a bug in the system that can be used for quick debug and when more and more services are available.
- Twitter's Zipkin is a good open source implementation, but it can be processed in your own code.
PAAS Agent Daemon
A unified operations platform for the client program, this program is responsible for: reporting hardware and network data to the monitoring system, starting and stopping the application, to the monitoring system and the PAAs platform to deliver the operational status of the application.
Storage platforms
This layer includes all hosting service for the heavy state.
- memcached cluster, using a unified consistent hash client, all cache servers for unified management, calculate hit rate, usage, real-time add memory.
- DBMS cluster, which uses a unified database to divide the surface layer, dynamically senses and switches faults. Common items such as mysqlproxy, Amoeba.
- HBase cluster, a standalone storage availability guarantee, is itself a cluster designed for high availability.
PAAS Resource Control Layer
The goal is to get real-time feedback on how much memory is inside the entire or multiple IDC, whether the CPU is adequate, and how many machines are needed next.
- Virtualization is a key challenge. Common open Source software: Docker, Warden, LXC.
- Resource control CPU available cgroups, disk available aufs, etc., these solutions are already included in the general virtualization scenario.
PAAs user interface Layer
This layer is primarily intended for operations and developers, such as on-line services, adding and removing machines.
- In addition to the Web interface, there should be support for CLI mode.
Automatic deployment Layer
It is generally done after the Hudson CI (continuous build) is completed, but an automated deployment must be a very reliable test framework, and the test code is reliable, otherwise it is a tragedy.
Test framework
Borrow some high-level frameworks to write less code, such as Jmockit, Spring-test, and so on.
Compiling tools
Java's maven is the choice. Compiled package warehouse, recommended Nexus.
Code generation
Developers do not need to repeat the operation, as long as the framework is fixed, all the code should be able to be generated. Just take the effort to modify the core logic.
- Here is more abstract, can use the method such as to do a maven-plugin, so that all engineers will use.
- Continue to upgrade this tool to include more possible code patterns.
Code Quality
After the engineer's code is complete, run the static analysis, you can find some problems in advance, can be made into regular mode, with continuous integration.
- Recommended Hudson + maven + sonar three swords in unity.
Code and general Systems
- Code-managed: Gitlab is a good tool like GitHub (more and more like it).
- Codereview: You can use Gitlab's merge request directly or open source Reviewboard.
- Knowledge Management: There's nothing to say, MediaWiki.
- Demand and Bug:jira.
- Fault management: This no open source project, Post-mortem system, is a record of the cause of the failure of the systems, the next time a failure comes, the system to conduct a questionnaire survey and reflection.
PAAS for DEV & TEST
- The development phase requires that the previously mentioned CLI be published to its own development machine (which also requires PAAs to be able to open a new development machine easily).
- The test phase requires a more high-availability environment than the development phase, and the reliability of the underlying tools should always be improved, the development environment should not be left out of development, but instead used as a development environment in a test environment, and the reason for such events in reality is that deployment is not perfect.
