Exploring some approaches to optimizing Ruby on Rails performance _ruby topics

1. The following two reasons are causing your Rails application to slow down:

1. Ruby and rails should not be used as the preferred place for Ruby and rails. (with Ruby and Rails doing a job that's not good for doing)
2. Excessive memory consumption leads to a lot of time for garbage collection.

Rails is a delightful framework, and Ruby is a simple and elegant language. But if it is abused, it can have a considerable impact on performance. There's a lot of work that doesn't apply to Ruby and Rails, you'd better use other tools, for example, the database has a significant advantage in large data processing, and R language is especially good for statistics-related work.

Memory problems are the primary reason why many Ruby applications are slowing down. The 80-20 rule of Rails performance tuning is this: 80% of the acceleration is derived from the optimization of memory, the remaining 20% of other factors. Why is memory consumption so important? Because the more memory you allocate, the more work the Ruby GC (Ruby's garbage collection mechanism) needs to do. Rails is already taking up a lot of memory, and the average application is just starting to consume nearly 100M of RAM. If you are not aware of the memory control, your program memory growth over 1G is very possible. So much memory needs to be recycled that it's no wonder that most of the time the program executes is consumed by the GC.

2 How do we make a Rails application run faster?

There are three ways to make your application faster: scalability, caching, and code optimization.

Expansion is now easy to achieve. Heroku is basically doing this for you, and Hirefire makes the process more automated. Other managed environments offer similar solutions. Anyway, you can use it if you like. But keep in mind that expansion is not a silver bullet that improves performance. If your application only needs to respond to a request within 5 minutes, the expansion will be of little use. There is also the use of Heroku + Hirefire can almost easily lead to your bank account overdraft. I've seen Hirefire expand my application to 36 entities and let me pay for $3100. I'm going to do it manually. The instance reduced to 2, and the code was optimized.

The Rails cache is also easy to implement. The block cache in Rails 4 is pretty good. The Rails documentation is an excellent source of information about caching. However, caching does not become the ultimate solution to performance problems compared with capacity expansion. If your code doesn't work as expected, you'll find yourself spending more and more of your resources on caching until the cache can no longer bring speed.

The only reliable way to get your Rails to apply faster is to code optimization. In the Rails scenario, this is memory optimization. And, of course, if you accept my advice and avoid using Rails for its design capabilities, you will have fewer code to optimize.

2.1 Avoid memory-intensive rails features

Some of the Rails features cost a lot of memory to cause additional garbage collection. The list is as follows.

2.1.1 Serialization Program

A serializer is a practical way of reading a string from a database that behaves as a Ruby data type.

Class Smth < activerecord::base
 Serialize:d ata, JSON
end
smth.find (...). Data
smth.find (...). Data = {...}

It consumes more memory to efficiently serialize, and you see for yourself:

Class Smth < activerecord::base
 def data
 json.parse (Read_attribute (:d ata))
 end
 def data= (value)
 Write_attribute (:d ata, Value.to_json)
 end

This will take as much as twice times the memory overhead. Some people, including myself, see the Rails JSON serializer memory leak, about 10% of the amount of data per request. I don't understand the reason behind this. I do not know whether there is a replicable situation. If you have experience, or know how to reduce memory, please tell me.

2.1.2 Activity Record

It is easy to manipulate data with ActiveRecord. But the essence of ActiveRecord is to wrap up your data. If you have 1g of table data, ActiveRecord says it will cost 2g, in some cases more. Yes, 90% of the situation, you get the extra convenience. But sometimes you don't need to, for example, batch updates can reduce ActiveRecord overhead. The following code, which does not instantiate any models, does not run validation and callbacks.

Book.where (' title like? ', '%rails% '). Update_all (Author: ' David ')
The scene behind it just executes the SQL UPDATE statement.

Update books
 Set author = ' David '
 where title like '%rails% '
Another example are iteration over a large dataset . Sometimes you need only the data. No typecasting, no updates. This snippet just runs the query and avoids ActiveRecord altogether: result
= Activerecord::base.execute ' SELECT * Fro M books '
Result.each do |row|
 # do something and Row.values_at (' col1 ', ' col2 ') end

2.1.3 String Callback

Rails callbacks like before/after save, before/after action, and a lot of use. But the way you write can affect your performance. Here are 3 ways you can write, for example: callback before saving:

Before_save:update_status
before_save do |model|
Model.update_status
End
before_save "Self.update_status"

The first two ways can work well, but the third is not. Why, then? Because a Rails callback requires that the execution context (variables, constants, global instances, and so on) be stored in the callback. If your application is large, you end up copying a lot of data in memory. Because the callback can be executed at any time, memory cannot be recycled until the end of your program.

There is a token that the callback saves me 0.6 seconds per request.

2.2 Write less Ruby

This is my favorite step. My university computer science professor likes to say that the best code doesn't exist. Sometimes it takes other tools to do the task at hand. The most common is the database. Why, then? Because Ruby is not good at processing large datasets. Very, very bad. Remember, Ruby takes up very large memory. So for example, you might need 3G or more memory to process 1G of data. It will take dozens of seconds to recycle this 3g. A good database can process the data in one second. Let me give you some examples.

2.2.1 Property preload

Sometimes the properties of the anti-normalization model are obtained from another database. Imagine, for example, that we are building a TODO list, including tasks. Each task can have one or several tag tags. The canonical data model is this:

• Tasks
• Id
• Name
• Tags
• Id
• Name
• Tasks_tags
• tag_id
• task_id

Load tasks and their Rails tags, you would do this:

This code has a problem, it creates an object for each tag, and it costs a lot of memory. Alternative solution to preload the label in the database.

tasks = Task.select <<-end
  *,
  Array (
  select Tags.name from tags inner join tasks_tags on (tags.id = Task s_tags.tag_id)
  where Tasks_tags.task_id=tasks.id
  ) as Tag_names
 End
 > 0.018 sec

This only requires memory to store an extra column, with an array tag. No wonder it's 3 times times faster.

2.2.2 Data collection

I say the data collection any code to summarize or analyze the data. These operations can be summarized simply, or some more complex. Take group rankings for example. Suppose we have an employee, Department, payroll data set, we want to calculate the employee's salary in one department rank.

SELECT * from Empsalary;

 Depname | Empno | Salary
-----------+-------+-------
 Develop |  6 | 6000
 Develop |  7 | 4500
 Develop |  5 | 4200
 Personnel |  2 | 3900
 Personnel |  4 | 3500
 Sales  |  1 | 5000
 Sales  |  3 | 4800

You can calculate rankings in Ruby:

Salaries = Empsalary.all
salaries.sort_by! {|s| [S.depname, S.salary]}
Key, counter = nil, nil
Salaries.each do |s|
 If S.depname!= key
 key, counter = s.depname, 0
 End
 counter + = 1
 s.rank = Counter
end

The data program for the Empsalary table 100K is completed in 4.02 seconds. Instead of Postgres queries, use the window function to do the same work over 4 times times in 1.1 seconds.

SELECT Depname, empno, salary, rank () over (PARTITION by depname order by salary
DESC) from
empsalary;

 Depname | Empno | Salary | Rank 
-----------+-------+--------+------
 Develop |  6 | 6000 | 1
 Develop |  7 | 4500 | 2
 Develop |  5 | 4200 | 3
 Personnel |  2 | 3900 | 1
 Personnel |  4 | 3500 | 2
 Sales  |  1 | 5000 | 1
 Sales  |  3 | 4800 | 2

4 times times the acceleration has been impressive, and sometimes you get more, to 20 times times. Give me an example from my own experience. I have a three-dimensional OLAP cube with 600k data rows. My program has been sliced and aggregated. In Ruby, it takes 1G of memory to complete in about 90 seconds. The equivalent SQL query is completed within 5.

2.3 Optimization Unicorn

If you are using unicorn, the following optimization techniques will apply. Unicorn is the fastest Web server in the Rails framework. But you can still make it run a little faster.

2.3.1 Pre-loading app application

Unicorn can preload the Rails application before creating a new worker process. This has two advantages. First, the main thread can share the memory data by copying the friendly GC mechanism (more than Ruby 2.0) when written. The operating system transparently replicates this data in case the worker modifies it. Second, preload reduces the time that the worker process starts. The Rails worker process reboot is common (will be elaborated later), so the faster the worker restarts, the better performance we can get.

If you need to turn on preload for the application, simply add a row to the Unicorn configuration file:

Preload_app true
2.3.2 GC between request requests

Please keep in mind that GC processing time is the most general meeting for 50% of the application time. This is not the only question. The GC is usually unpredictable and triggers the run when you don't want it to run. So, what do you do with it?

First, we'll think about what happens if the GC is completely disabled. This seems to be a bad idea. Your application is likely to fill up 1G of memory quickly, and you haven't been able to find it in time. If your server is running several worker at the same time, then your application will soon be out of memory, even if your application is in a hosted server. Not to mention the Heroku with only 512M memory limits.

In fact, we have a better way. So if we can't avoid GC, we can try to make the GC run time point as much as possible and run at leisure. For example, between two request, the GC is run. This is easily implemented through configuration unicorn.

For the previous version of Ruby 2.1, there is a unicorn module called OOBGC:

Require ' unicorn/oob_gc ' use
 (UNICORN::OOBGC, 1) # "1" means "Force GC to run after 1 request"

For Ruby 2.1 and later versions, it's best to use Gctools (https://github.com/tmm1/gctools):

Require ' GCTOOLS/OOBGC ' use
(gc::oob::unicornmiddleware)

However, there are some caveats to running GC between request. Most importantly, this optimization technique is perceptible. In other words, the user will obviously feel the improvement in performance. But the server needs to do more work. Unlike when a GC is run when it is needed, the technology requires the server to run the GC frequently. So you want to make sure that your server has enough resources to run the GC, and that there are enough worker to process the user's request while other worker is running the GC.

2.4 Limited growth

I've shown you some examples of applications that can consume 1G of memory. If your memory is sufficient, it is not a big problem to occupy such a large chunk of memory. But Ruby may not return this memory to the operating system. Now let me explain why.

Ruby allocates memory through two heaps. All Ruby objects are stored in Ruby's own heap. Each object occupies 40 bytes (in 64-bit operating system). When an object needs more memory, it allocates memory in the operating system's heap. When an object is garbage collected and released, the memory of the heap in the operating system is returned to the operating system, but the memory consumed by Ruby's own heap is simply marked free and not returned to the operating system.

This means that Ruby's heap will only increase and not decrease. Imagine if you read 1 million rows of records from a database, 10 columns per row. Then you need to allocate at least 10 million objects to store the data. Ruby worker typically consumes 100M of RAM after startup. To accommodate so much data, the worker needs an extra 400M of memory (10 million objects, 40 bytes per object). Even if these objects are finally retracted, the worker still uses 500M of RAM.

Here you need to declare that Ruby GC can reduce the size of this heap. But I have not found this function in the actual combat. Because in a production environment, the conditions in which the trigger heap is reduced are rarely present.

If your worker can only grow, the most obvious solution is to restart the worker whenever it takes up too much memory. Some managed services do this, such as Heroku. Let's look at other ways to implement this functionality.

2.4.1 Internal Memory control

Trusts in God, but lock your cars believe in God, but don't forget to lock the car. (Moral: Most foreigners have religious beliefs and believe that God is omnipotent, but in daily life, who can expect God to help himself?) Faith is faith, but in times of difficulty it is on one's own. ）。 There are two ways to allow your application to achieve a self memory limit. I take care of them, Kind (friendly) and hard (mandatory).

The Kind friendly memory limit is to force the memory size after each request. If the worker consumes too much memory, the worker will end, and Unicorn will create a new worker. That's why I call it "kind". It will not cause your application to break.

Gets the memory size of the process, using RSS metrics on Linux and MacOS or OS gems on Windows. Let me show you how to implement this limitation in the Unicorn configuration file:

Class Unicorn::httpserver
 Kind_memory_limit_rss = #MB
 alias Process_client_orig process_client
 undef _method:p rocess_client
 def process_client (client)
 process_client_orig (client)
 RSS = ' ps-o rss=-P #{ Process.pid} '. chomp.to_i/1024
 exit if RSS > Kind_memory_limit_rss
 End

The hard disk memory limit is by asking the operating system to kill your work process if it grows a lot. On Unix you can call Setrlimit to set the RSSX limit. As far as I know, this is only valid on Linux. The MacOS implementation was broken. I will be grateful for any new information.

This fragment comes from the Unicorn hard disk limit profile:

After_fork do |server, worker|
 Worker.set_memory_limits
End
class unicorn::worker
 Hard_memory_limit_rss = #MB
 def set_ Memory_limits
 process.setrlimit (process::rlimit_as, Hard_memory_limit * 1024 * 1024)
 End

2.4.2 External Memory control

Automatic control does not save you from occasional omm (insufficient memory). Usually you should set up some external tools. On the Heroku, there is no need for them to have their own monitoring. But if you are self hosted, using Monit,god is a good idea, or other monitoring solution.

2.5 Optimizing Ruby GC

In some cases, you can adjust the Ruby GC to improve its performance. I want to say that these GC tuning is becoming less important, and Ruby 2.1 's default setting has been beneficial to most people later.

My advice is that it's best not to change the GC settings unless you know exactly what you want to do and have enough theoretical knowledge to know how to improve performance. This is especially important for users who use the Ruby 2.1 or later version.

I know there's only one occasion. GC optimization does provide a performance boost. That is, when you want to overload a lot of data at once. You can reduce the frequency of GC operations by changing the following environment variables: RUBY_GC_HEAP_GROWTH_FACTOR,RUBY_GC_MALLOC_LIMIT,RUBY_GC_MALLOC_LIMIT_MAX,RUBY_GC_ Oldmalloc_limit, and Ruby_gc_oldmalloc_limit.

Note that these variables apply only to Ruby 2.1 and later versions. For versions prior to 2.1, a variable might be missing, or the variable would not use that name.

Ruby_gc_heap_growth_factor default value of 1.8, which is used when Ruby's heap does not have enough space to allocate memory, how much each time should increase. When you need to use a lot of objects, you want the memory space of the heap to grow faster. On this occasion, you need to increase the size of the factor.

Memory limits are used to define how often a GC is triggered when you need to apply space to the operating system's heap. Ruby 2.1 and later versions, the default limit is:

New generation malloc limit ruby_gc_malloc_limit 16M
Maximum New Generation malloc limit Ruby_gc_malloc_limit_max 32M Old
generation malloc limit ruby_gc_oldmalloc_limit 16M
Maximum old generation malloc limit Ruby_gc_oldmalloc_limit_max 128M

Let me briefly explain the meaning of these values. By setting the value above, each time a new object is assigned between 16M and 32M, and when the old object is occupied between 16M to 128M ("old object" means that the object was at least once called by garbage collection), Ruby runs the GC. Ruby adjusts the current quota value dynamically according to your memory mode.

So, when you have only a few objects that occupy a lot of memory (such as reading a large file into a string object), you can increase the limit to reduce the frequency at which the GC is triggered. Remember to add 4 limit values at the same time, preferably a multiple of the default value.

My suggestion is that it may be different from other people's suggestions. It may be right for me, but not for you. These articles will describe what applies to Twitter and which applies to discourse.

2.6 Profile

Sometimes, these suggestions may not be universal. You need to figure out your problem. At this point, you need to use Profiler. Ruby-prof is a tool that every Ruby user will use.

To learn more about profiling, read Chris Heald ' s and my article about using Ruby-prof in Rails. There are some perhaps outdated suggestions about memory profiling.

2.7 Writing performance Test cases

Finally, the technique of improving Rails performance, though not the most important, is to verify that the performance of the application does not degrade again because you have modified the code.

3 Summary Testimonials

For an article, it's really impossible to do everything you can to improve the performance of Ruby and Rails. So, after that, I'll summarize my experience by writing a book. If you think my suggestion is useful, please register mailinglist, when I have prepared the book preview version, will be the first time to inform you. Now, let's do it together and let the Rails app run faster!