Build a PHP 1 billion request/Week website based on the official name of Symfony2 and Redis

[Editor's note] If you are still using Symfony2 and Redis, you cannot use Redis as the primary storage. Because Symfony2 has many features, it runs very slowly, it may be worth looking at how to build a high-request website for Octivi. Although there is no underlying details, it details the macro features based on the two applications and the Symfony2 during development.

[Editor's note] If you are still using Symfony2 and Redis, you cannot use Redis as the primary storage. Because Symfony2 has many features, it runs very slowly, it may be worth looking at how to build a high-request website for Octivi. Although there is no underlying details, it details the macro features based on the two applications and the Symfony2 during development.

[Editor's note] If you are still using Symfony2 and Redis, you cannot use Redis as the primary storage. Because Symfony2 has many features, it runs very slowly, it may be worth looking at how to build a high-request website for Octivi. Although there is no underlying details, it details the macro features based on the two applications and the Symfony2 features during development.

Subscribe to the "CSDN Big Data" public account for free to learn about the latest Big Data progress in real time!

CSDN big data focuses on sharing and discussing big data information, technologies and experiences, provides big data views such as Hadoop, Spark, Imapala, Storm, HBase, MongoDB, Solr, machine learning, and intelligent algorithms, big data technology, big data platform, and Big Data practices, big Data industry information and other services.

The following is a translation:

Some people say that Symfony2 is as slow as other complex frameworks, but we think it all depends on the user's own. This article describes the software architecture details of applications that execute more than 1 billion requests per week based on Symfony2.

The following shows the social feedback after the tweeting:

This article describes applications based on Symfony2 and Redis. There will not be too many details here. On the contrary, we will show you the macro features of these applications and the Symfony2 features during development.

For low-level Symfony2 performance optimization practices, we have written a special article-master the Symfony2 performance series-Internals and Doctrine

The first is some data about the described application.

Performance Statistics from a single program node:

• Symfony2Instance processing per second700Request, the average response time of each request is 30 ms
• VarnishProcess more than 12000 requests per second (obtained through stress testing)

Note: As described below, the entire platform includes many such nodes

RedisMeasurement:

• Over 0.16 billion keys (98% of which are permanently stored );
• 89% hits-that is, only 11% of transactions reach the MYSQL server.

Stack structure

Application

All traffic will flow into HAProxy, and HAProxy will distribute the traffic to the application server.

The application instance is named Varnish Reverse Proxy.

We keep Varnish highly available on every application server-No spof. A single Varnish allocation of traffic may cause risks. The detached Varnish instance may reduce the cache hit, but we can accept this. We need higher availability than performance, but you can see from these figures that performance is not a problem.

Application Server Configuration:

• Xeon E5-1620@3.60GHz, 64 gb ram, SATA
• Apache2(We don't even need nginx)
• PHP5.4.X operates in PHP-FPM, along with APC

Data Storage

We use Redis and MySQL to store data. Their numbers are quite large:

• Redis:

1. 15 thousand hits/second
2. 0.16 billion keys

• MySQL:

1. Over 400 GB of data
2. 0.3 billion records

We use Redis as the permanent storage (the most used resource), and Redis as the cache layer on MySQL. Compared with the typical cache, Redis stores a high rate of data-we store more than 0.155 billion permanent keys and only 5 million buffer keys. In fact, we can use Redis as the primary data storage.

Redis is configured with master-slave settings. In this way, we obtain HA-if the operation is interrupted, we can quickly switch the master node to a slave node. These configurations are also required for some management tasks such as upgrading. When upgrading a node, we can select a new master node, then upgrade the previous master node, and finally switch the two nodes.

We are still waiting for the production of ready Redis clusters, which can provide functions similar to automatic fault recovery (even manual fault recovery is more convenient when upgrading nodes. However, there is no message about the official release date.

MySQL is usually used as the layer-3 cache layer for non-depletion resources (Varnish> Redis> MySQL ). All tables are InnoDB and the most common queries are simple.

SELECT ... WHERE 'id'={ID}

This query returns a single result. We have not found any performance problems with this setting.

Different from Redis settings, MySQL runs on the master configuration. In addition to high availability, this also provides better write performance (this is not a problem in Redis, because we will not exhaust performance features .)

Application's Architecture

Symfony2 Function

Symfony has some great features that make the development process easier. Below we will introduce some of the developers' favorite features:

Note

We use the standard Symfony2 distribution with annotations:

• Route Selection-routing defines the application URL-we also tested the silly routing rules of Apache, but it does not have any major optimization.
• Service container -- we use JMSDiExtraBundle's service annotation to define our DI container-this accelerates development and allows us to process service definitions with PHP code. We found that PHP code is more readable.

Because the application is used as a rest api, we mainly do not use templates (such as Twig ). We reserve templates mainly for some internal dashboard panels.

We have not found the performance impact of different configuration types (YAML/XML. Because all comments are well stored, there is nothing confusing-and finally everything is pure PHP code.

The following is a sample service configuration obtained using JMSDiExtraBundle:

/** * Constructor uses JMSDiExtraBundle for dependencies injection. *  * @InjectParams({ *      "em"         = @Inject("doctrine.orm.entity_manager"), *      "security"   = @Inject("security.context") * }) */function __construct(EntityManager $em, SecurityContext $security) {    $this->em = $em;    $this->security = $security;}

In this way, you only need to change the code to change the class dependency.

Symfony2 monitoring-Monolog and Stopwatch

The application uses Monolog to record unexpected behaviors and capture error messages. We use multiple channels to obtain the separation logs of different application modules.

Because FingersCrossed handler uses a large amount of memory (which may cause memory leakage), we will not use it any more. We select the appropriate StreamHandler. When using this method, we need to add redundancy and additional content in the single line log information.

We also use the Stopwatch component in many places to control some typical application methods. In this way, we can find some major weaknesses in the customized logic.

For example, we track the number of requests for some external network services:

if (null !== $this->stopwatch) {    $this->stopwatch->start('my_webservice', 'request');}// Makes a CURL request to some my_webservice$response = $this->request($args);if (null !== $this->stopwatch) {    $this->stopwatch->stop('my_webservice');}

Console Components

During development and maintenance, we especially like the Symfony Console component, which provides a good object-oriented interface for creating CLI tools. The application adds about 50% of new features. These new features are based on CLI commands and are mainly used to manage or analyze internal components of the application.

Console components properly process command statements or options-you can set the default value, optional value or required value. Good practices always record these appropriate records as code-you can set a major description for commands and options. Commands are usually self-documented, because the formatted instruction description can be generated by adding the -- help option.

$ php app/console octivi:test-command --helpUsage: octivi:test-command [-l|--limit[="..."]] [-o|--offset[="..."]] tableArguments: table                 Database table to processOptions: --limit (-l)          Limit per SQL query. (default: 10) --offset (-o)         Offset for the first statement(default: 0)

We must remember to run commands in an accurate environment. The default dev may cause some problems, such as memory leakage (because more lengthy logs are stored and debugging information is saved ).

$ php app/console octivi:test-command --env=prod

To display better information, add the-v option.

$ php app/console octivi:test-command --env=prod -vvv

The progress bar is a good helper. The progress bar even considers the details of information display. When the degree is relatively low, only basic information is displayed. When the degree is relatively high, information such as running time and memory consumption can also be displayed.

In addition, there are some migration procedures that take about two days-0 memory leaks-No progress bars, monitoring them will be a disaster.

Data Layer

For Redis, the data layer uses PredisBundle.

We reject Doctrine ORM because it will add additional charges and we do not need any advanced object-oriented operations. We use Doctrine DBAL instead. The Doctrine DBAL features are as follows:

• Query Builder
• Preprocessing statement

The use of PredisBundle and Doctrine Bundle also allows us to monitor weak queries when using a large number of analysis tools.

Summary

Thanks to Symfony2, this setting maintains high performance and high availability while maintaining a friendly development environment-can be maintained and stable. In fact, this is the key business requirement of the key subsystem used as an e-commerce website.

Therefore, we can correct some incorrect ideas at the end of this article:

• Redis cannot be used as the primary storage-as we mentioned earlier, of course, yes! Redis is a very stable technology with some persistence mechanisms and you will not lose key data.
• Symfony2 features so much that it is so slow-when you don't use some time-consuming/memory tools such as ORM, you can get and Silex (yes, we tested it) the microframework has similar performance.

Handling 1 Billion requests a week with Symfony2