PHP APM Comparison Evaluation: OneAPM, New Relic, listen to the cloud

Thanks to the recommendation of @penguinz, but also found a provider of application performance management services of domestic manufacturers: "Listen to the Cloud", read the people-Wu Yu write the trial notes, only to understand the application of foreign properties management company New Relic is the real APM Daniel, product line coverage is very comprehensive, the function is very powerful, But indeed, as the man said, the visit was too slow. Look up, found from the product design to the interface, the products of the three companies are too similar, it is clear that the domestic two products are in the "learning" New relic products, hope that two domestic manufacturers are not just a simple copy of foreign products, but can make the products in line with the needs of domestic users.

Last time I wrote a ONEAPM evaluation, about listening to the cloud product testing I will not write more, the People's blog has provided a very detailed trial report, you can go to see. http://www.imsiren.com/archives/1192. Just after the spring festival a little time, put 3 products are loaded once, respectively carefully used for a period of time, to say a few products of the contrast feeling.

Comparison of response time graphs

Looking at the test report of the people, and found that listening to cloud products can monitor the performance of NoSQL access, so this testing in the original WordPress application based on the addition of several PHP scripts, the application of the MySQL database, but also introduced to MongoDB, Access to Redis and memcached. From the response time comparison, listen to the cloud support performance indicators are the most, see the following table:

Response Performance Metrics	OneAPM	Listen to the Cloud	New Relic
PHP code	Support	Support	Support
RDMS Database	Support	Support	Support
Memcache	Not supported	Support	Support
External services	Not supported	Support	Support
Redis	Not supported	Support	Not supported
Mongodb	Not supported	Support	Not supported
Blocking time	Not supported	Support	Not supported

Also, later on, listening to the cloud also provides deeper analysis of these commonly used NoSQL databases, while the other two products only perform deep analysis of the RDMS relational database.

ONEAPM response time graph, showing only the performance decomposition of web transactions and databases

Listen to the cloud's response time graph, showing performance decomposition for multiple components including applications, databases, non-relational databases, and more

New Relic Response Time graph showing php,database, Memcache, Web external 4 performance decomposition

Topology Logic diagram comparison

From the topology diagram can also be seen from the various applications of the different backend service support, both the listening cloud and new relic support the display of NoSQL database, and ONEAPM only the database service display. The topology diagram of ONEAPM can drill down into the decomposition reports of web transactions and databases directly on the graph, while the listening cloud and new relic do not provide drillthrough capabilities, only the response graph of the corresponding service is presented.

ONEAPM topology diagram, can be dragged and drilled

Listen to cloud topology diagram, identify the most services, not drag and drill

New Relic Map, recognize part of NoSQL, not drag and drill

Transactional performance Analysis vs.

ONEAPM List of transactions

Transaction list for New Relic

Listen to the cloud's list of transactions

As can be seen from the list of transactions, the New relic support for WordPress is better than the other two, can be based on the different parameters that WordPress received to identify different transaction names to summarize statistics, while the other two can only be in the form of URIs to identify and statistics transactions.

Transaction Trace Comparison

Three products in the transaction performance of the summary analysis of the function is not small, the main difference is expressed on the trace of slow transactions. The trace feature retains detailed diagnostic data for very slow transactional access, including information such as the time-consuming situation of the code snippet, the detailed steps of the code snippet execution, and the call stack, related SQL statements, and so on. The default sort of list of tracking records, listening to the cloud using the response time of the inverted sort, and new relic and ONEAPM use is the acquisition time stamp sort, compare down, listen to the cloud sort way more reasonable, I certainly most priority is the slowest request.

OneAPM Trace Overview

Listen to the cloud application process tracking Summary

New Relic Transaction Trace Summary

In the trace's profile, the New Relic presentation Performance component is relatively concise, and has a clear meaning, and is very easy to read and approximate to locate the problem. Listening to the cloud components show the best decomposition, but because of the decomposition is too thin, but it is not easy to read, but also not enough introduction. and ONEAPM Although the component display is also relatively small, but decomposition is chaotic, completely unintelligible.

The second part of the trace details of the transaction trace shows the complete execution of the code in the slow transaction of the record, including nested calls to the code, the code stack, and so on. Both the listening cloud and the new relic provide more accurate and readable code invocation details and code stacks, and the code snippet in the ONEAPM details is problematic, sometimes with non-PHP C code, and no display of the code stack.

Listen to cloud tracking details

The trace Detail of New Relic

ONEAPM's Trace details

ONEAPM's trace information is more than the other two. The user-defined parameters section should refer to the form parameters submitted in the request. The other two have only some parameter information in the HTTP header.

Slow SQL Log Comparison

The slow SQL log analysis is similar to the slow query log in MySQL (MySQL slow query log), which can record SQL statements with slow query times. From a functional comparison, ONEAPM records only detailed SQL statements and query times, and new relic and listen clouds record the execution plan of the SQL statement and the application code call stack that invokes the SQL statement in addition to the query time and SQL statement. In addition, the listening cloud also shows a scatter plot of the corresponding SQL statement query time distribution, which is more intuitive to use for viewing slow SQL records.

Listen to the cloud's slow SQL trace data in the most detailed, including scatter plots, SQL statements, query times, execution plans, and code call stacks

New Relic's slow query trace, which includes query time, SQL statements, and code call stacks.

ONEAPM slow SQL records, only query time and SQL statements.

NoSQL Performance Comparison

Currently in three of the products, only listen to the cloud to provide the performance of the NoSQL service analysis, listening to the cloud provides three NoSQL services including MongoDB, Redis and memcached analysis, can see the response time and throughput of various operations, For MongoDB You can also press collection to see the performance of different operations. Although New Relic has memcached performance data in the previous response time, it does not provide more granular analysis data for this NoSQL service alone. ONEAPM does not currently support any kind of NoSQL database performance analysis.

Listen to the Cloud's NoSQL performance analysis module

MongoDB Analysis of Listening cloud

Redis Analysis of Listening cloud

Memcached analysis of the listening cloud

External service comparison

All three products support performance analysis of external services (that is, external APIs that are invoked through Web service invocation). The products of New Relic and ONEAPM show the average response performance of each host, but there seems to be a bug in the ONEAPM that causes the same host in the list to recur and the performance values to be inconsistent. Listening to the external service performance analysis of the cloud in addition to the host level of data, you can also down by the host under each different URI to summarize performance data, you can understand the different API interface performance differences, more practical value.

ONEAPM external Services, displayed on the host level, there are bugs that cause the same host to recur

New Relic External Services, presented at the host level

Listen to the external services of the cloud, display to host and specific URI level

Background task (CLI mode PHP script) performance comparison

For PHP scripts that are not accessed through the web, that is, the PHP program that is run by the command-line mode (CLI), three products are presented in the form of a background task. Currently ONEAPM's products cannot provide CLI-mode PHP application monitoring, this part of the data is empty. Both the New relic and the listening cloud can monitor the PHP run by the CLI and provide performance decomposition capabilities to see how the background task's performance is consumed in the code snippet. But new relic performance decomposition has a bug, I run the script is obviously access to the Redis database, it is broken out is memcache access, if so, the previous several charts memcache performance data estimation is also wrong ...

ONEAPM background task data is empty, unable to monitor CLI-mode PHP application performance

New Relic Background task data with "Non-web" type to demonstrate PHP application performance running in CLI mode

New Relic Background Task performance decomposition, you can see the code time and the NoSQL service operation time, but the Redis recognized as memcached

Background task analysis of listening cloud

Listen to cloud background task performance decomposition to correctly identify code execution times and the performance of Redis's various operations.

Error analysis Comparison

Error analysis records the exception information thrown in the application and the PHP error code to calculate the error rate for the entire application. Judging from this test results, listen to the cloud and the other two of the difference is relatively large, New relic and ONEAPM have recorded a large number of error messages, about 10% of the error rate, and listen to the cloud but an error message is not recorded.

After looking closely at the data, it was found that the errors recorded by New Relic and ONEAPM were actually not serious errors in the warning level (e_user_warning), and in fact my test application was normally accessed without errors. While listening to the cloud only records the error basic error, so a warning level of error messages are not logged. From a practical point of view, listen to the cloud more reasonable, because these warning level of error is really I do not need to care, or my application error rate is so high, users early complaints.

However, if possible, you may want to provide an error level setting option that allows you to choose which level of error logs to collect when needed.

Error analysis of ONEAPM

Error analysis of New Relic

Report comparison

Both New Relic and ONEAPM provide reporting functionality, which is a summary table showing a comparison of web transactions and SQL performance over time. From the test results, the New Relic can provide the report data, the ONEAPM report function seems to be unable to use this time, the tabular data is always empty, the last time the test was good. And listening to the cloud does not have this function module.

The ONEAPM report, which seems to be unusable recently, is always empty.

New Relic report showing comparison of performance data over time

Alarm setting comparison

Three products can be used to monitor the PHP application performance and error rate of the alarm settings, when the application performance problems and error rate is too high to send alerts to notify the user.

The comparison test found that ONEAPM and new relic can be pre-set different alarm strategies, such as alarm thresholds and trigger time and so on, and then assign the policy to the application need to alarm, through the policy can be set more flexible alarm rules and easy to copy to multiple applications, It is more convenient to use.

And listen to the cloud alarm settings can only set the alarm threshold for each application, unable to set the trigger time and other parameters, and because there is no policy allocation, can not replicate the same alarm settings on multiple applications, poor ease of use.

ONEAPM Alarm policy settings can be set for conditions such as thresholds and trigger times

The New Relic alarm policy setting also allows for threshold and trigger time settings

Listen to the cloud alarm settings can only be set threshold, and there is no concept of alert policy

Apply Settings comparison

There are many application settings, such as the trace threshold and the Apdext value settings, which have a large impact on the monitoring result data, so it is best to provide users with customized settings. In particular, the Apdext value directly affects the evaluation of the APDEX score and the result of the alert, which is very much needed to be dynamically set up at any time. From the test results, the application settings of the listening cloud are most convenient and can be modified online and take effect in real time without restarting the application server. ONEAPM and new relic are not so complete in their application settings.

ONEAPM App Settings page, there are actually no items to set up, only a few options listed, may be configured in the configuration file, but there is no corresponding explanation and explanation. Modifying the settings of a configuration file requires restarting the application server to be effective and less practical.

New Relic application Settings, you can modify the application name and Apdext value, the other options can only be modified in the configuration file, the configuration file is described in more detail, but the same problem is modified to restart the service, the usefulness is slightly insufficient.

Listen to the application settings of the cloud, you can modify the parameters and thresholds of the most complete, while providing the configuration file and the function of the line settings, the settings are detailed description and explanation, the online settings do not need to restart the service to take effect, can be adjusted at any time, very good ease of use.

Original articles, reproduced please indicate the source!

