Redmine Performance Optimization Scheme

Recently the company Redmine server performance is very bad, in the 16-core, 64GRAM machine, the results of the test is only 5~7 per second request, part of the page does not come out.

Here is my Redmine performance optimization scenario:

Redmine server performance Troubleshooting and optimization recommendations: The following recommended scenarios are render time-consuming, activerecord-time-consuming in log files based on the Redmine runtime, sampling analysis of Linux system performance metrics and MySQL performance metrics, and redmine in different web Benchmark under Server:   I. Troubleshooting and Quantitative AnalysisBy analyzing the log of the Redmine runtime, compare the request volume of the MySQL peak time with the system CPU, mem, disk, iowait and so on, as well as the performance test results of the fake concurrency in the same environment. You can see that the peak hours in the server are not using CPU and memory efficiently. The main 4 reasons are: 1. The rails framework is cumbersome, inefficient, and 2. A ruby process can only take advantage of a single core CPU, its multithreading is controlled by the Gil, multithreaded inefficient, unable to effectively utilize multi-core CPUs, so during peak hours only the passenger process occupied a few CPUs full load, the remaining utilization is basically 0. 3. The current Redmine release method is Apache+passenger, passenger has not been optimized, only 6 workers are opened, and only process-level support, which results in a very low iredmine server throughput. 4. mysql operation mode has not been optimized configuration, basic bare run. 1.1 Description of the performance indicators:The performance metrics data has been stored and can be referenced as follows for analysis: 1.     Linux system sampling script: in ~/performancelog/collectlog.sh; The sample results are ~/performancelog in a file named after the date.  Note: If you later want to use this script to collect data, if the MySQL address or directory is replaced, this script Dstat MySQL related data collection needs to rewrite its MySQL connection part of the code. 2. MySQL metrics data can be run by running the following command: Mycheckpoint--user=root--password=xxxx--socket=/redmine/mysql/tmp/mysql.sock--database= Mycheckpoint HTTP then opens the browser: http://172.26.1xx.xx:8080/mycheckpoint/can view MySQL metrics.   1.2 Benchmark Quantitative Analysis method Description:Forge concurrent HTTP requests to see the performance of different Web servers when 10 requests per second are concurrent. Load testing methods for different Web servers are as follows: In quantifying performance, the performance of this scenario is improved, in order to facilitate the AB test, it is recommended to close the CSRF protection in the Redmine code, and use the session, automatic login, to ensure that the test address has SQL operations, Make sure that the test process contains the SQL request by doing the following: To cancel the Csrf protection method:Locate the Applicationcontroller.rb file in the controller directory in the code published by Redmine, and csrf the protection function Protect_from_forgery Cookie.delete (auto_ Cookie_name) line commented out. (Production mode remember to transfer back)   AB analog concurrency test steps:0../ctlscript.sh stop Apache1.  Use puma-w 16-t 16-p 8080/redmine/apps/redmine/htdocs/config.ru to start Redmine2.  Chrome browser, open URL, view inspect tools, copy Cookies3. Using AB tool: Ab-n 800-c 16-h "cookie:key1=value1; Key2=value2 "http://localhost:8080/issues/48575 Two: Optimization program Description:
1. Slow response time when concurrency is too large #ok, use multi-process * multi-thread web Server2. Database Read and write performance is too low #ok, open the MySQL query cache and so on.      3. Use asynchronous IO library em-synchrony to improve IO throughput. #failed, tested, unstable, interface versions sometimes do not match, requires more than 3.1 version of rails. 4. Use Rubinus or JRuby to improve performance #failed, tested JRuby memory loss is too large, rubinus have compatibility issues. 5. Remove the non-required middleware #rails默认加载20多个中间件 rails is loaded by default, some of which are unnecessary. In the above several steps of the optimization adjustment, in the AB test found that throughput can be increased by 2 to 3 times times, load capacity is also improved, but not to test the limit data. Three: Web Server Optimization Recommendations:It is recommended to use multi-process multi-threaded version of Puma instead of multi-process single-threaded passenger, or use passenger, but configured to turn on 14 workers by default, allowing two CPUs to be used for MySQL during peak hours. In addition, because Puma can use multithreading to serve HTTP requests to a certain extent, in order to ensure thread safety, the Iredmine code needs to be modified: add config.threadsafe! to config/    PRODUCTION.RB file in the enviroment directory # enable multithread, and keep thread safe. Config.threadsafe!1.    Use of Puma: Start 14 worker Processes * 16 threads, increase throughput rate. Puma-w 14-t 16-p 8080/redmine/apps/redmine/htdocs/config.ru  2. Configure the passenger method: Config file under:/redmine/apache2/conf/bitnami directory, add the following: Passengermaxpoolsize 14 four: MySQL Server Tuning recommendations:1. Open the settings in the MySQL configuration file (you need to restart MySQL after the following two variable settings): threads_cached-on thread_cache_size = 64; Query_cache_size (>= 8M) join_buffer_size (> 128.0K) tmp_table_size (> 16M) max_heap_table_size (> 16 M) thread_cache_size (start at 4) Table_open_cache (>) innodb_buffer_pool_size (>= 371M) There are also: Thread_co  Ncurrency = 32; (cpu*2) for user request pressure on MySQL, see http://172.26.1xx.xx:8080/mycheckpoint/ V: Rails Optimization Recommendations: Delete unwanted middlewarePlease join the current Redmine plugin developer to filter and delete based on the actual middleware usage of the current rails. Method: You can add the following code to the application configuration file to remove it. # config/application.rbconfig.middleware.delete "Rack::lock" Config.middleware.delete ' Rack::cache '                          # full page caching appears to be unused in redmine. Config.middleware.delete ' Rack::runtime '                      # record X -runtime (convenient for clients to view execution time) Config.middleware.delete ' Actiondispatch::requestid '     # Record X-request-id (easy to see which of the requests are executing in the cluster) Config.middleware.delete ' Actiondispatch::remoteip '      # IP SpoofAttackconfig.middleware.delete ' actiondispatch::callbacks '       # Set Callbackconfig.middleware.delete ' Actiondispatch::head '             in front of the request, if it is a Head request, Executes according to the GET request, but does not return bodyconfig.middleware.delete ' actiondispatch::beststandardssupport ' # settings x-ua-compatible, Set on Nginx, etc...    Six: Remove the log and some exception corrections that rails does not need in production mode. 1. Because of the discovery of a large section of the debug mode in the Production.log log all output to the log file, a long time, it is recommended to delete. 2. Some URLs appear runexception. Suggested corrections or de-listing in middleware is called:Use Actiondispatch::showexceptionsuse actiondispatch::D ebugexceptions

Seven: part of the Redmine plugin is poorly written, render time up to 3000ms

Related information:

1. Linux system performance location and troubleshooting: http://www.cnblogs.com/ToDoToTry/p/4423577.html

2. AB Pressure test method:

3. mysql performance recording and analysis:

4. mysql Optimization tool:

5. Linux IO problem analysis and troubleshooting:

6. mysql Performance testing tools and methods:

Redmine Performance Optimization Scheme