On the path of code-level performance Optimization Change (II.)

This article is "on the Code level performance optimization Change Road One" of the second chapter.

In the previous article we mainly introduced the five points of the problem encountered, then today to discuss the remaining issues, let us review the previous discussion of the issue:

1, a single 40TPS, add to 4 servers to 60TPS, the scalability of almost no.
2, in the actual production environment, the frequent occurrence of database deadlock causes the entire service interruption is not available.
3, the database transaction chaos, resulting in a transaction time consuming too long.
4, in the actual production environment, the server often occurs memory overflow and CPU time is full.
5, the process of program development, the consideration is not comprehensive, fault tolerance is poor, often because of a small bug and caused the service is not available.
6, the program does not print the key log, or print the log, information is useless information does not have any reference value.
7, configuration information and small change of information will still be frequently read from the database, resulting in a large database IO.
8, the project is not completely split, a tomcat will be the deployment of multiple project War package.
9. Due to bugs in the underlying platform, or functional defects, program availability is reduced.
10, the program interface has no current limit policy, resulting in many VIP merchants directly to our production environment for pressure measurement, directly affect the real service availability.
11, there is no failure to downgrade the policy, the project after the problem of long-time resolution, or directly rough rollback project, but not necessarily solve the problem.
12, there is no suitable monitoring system, can not be quasi-real-time or early detection of project bottlenecks.

Iv. Optimization of solutions

5. Cache optimization Scheme
Information about the configuration and small changes can be placed in the cache, improve concurrency can also reduce the IO cache, specific cache optimization policy can be referred to my previous write:
http://www.jianshu.com/p/d96906140199

6, program fault-tolerant optimization scheme
In this piece I want to give an example of a program to explain what is fault-tolerant, first look at the program:

//service layer:   Public  void  insertorderinfo  (OrderInfo        OrderInfo) {try  {orderdao.insertorderinfo (orderInfo); } catch  (Exception e) {logger.error ( "order information inserted into the database failed!)        OrderId: " +orderinfo.getorderid (), E); }}//dao layer  public   void  insertorderinfo  (OrderInfo OrderInfo) {try< /span> {this . Sqlmapclient.insert (, OrderInfo)} catch  (Exception e) {}}  

Note:
So if the service layer method calls the DAO layer method, once the data insert fails, then is this exception handling a fault-tolerant approach?
The exception is eaten, when the service layer is called, although there is no error message printed, but this can be fault-tolerant?
The so-called fault tolerance refers to the characteristic that the computer system does not fail and still works correctly in the case of fault.

We use the cache as a case study, first look at a diagram:

This is one of the simplest diagrams, and app service gets configuration information from Redis on a regular basis, and some friends might think it's stable, but what if there's a problem with Redis? There may be friends who say that Redis will be clustered, fragmented, or master-slave to ensure that there are no problems. In fact, I think that, although the application service program as far as possible to keep the lightweight is good, but not so that the hope is all pinned to the middle component above, in other words, if the Redis is a single point, then the consequences will be what, then with a large number of concurrent requests come, The program will report a large number of errors, while the normal process can not go on the business may also be interrupted.

So in this scenario, my solution is to put the use of the cache level, some cache synchronization requirements are very high, such as payment limit configuration, in the background after the completion of changes in the foreground immediately can be perceived, and can successfully switch, this situation can only be real-time from Redis to get the latest data, However, each time you get the latest data, you can update the local cache synchronously, and when a single point of Redis is hung up, the application can at least read the information locally without the service being instantly hung up. Some caches are not sensitive to timeliness, allowing for a certain delay, then in this case I am using a scheme that leverages the local cache and the remote cache in a way that is combined, as shown in:
programme One:

This way the Redis cache server is updated more synchronously with the local cache through the Ehcache timer of the application server, and the disadvantage is that each server has a different timing ehcache time, so the time for each server to refresh the latest cache is not the same, resulting in inconsistent data. The consistency requirement is not high and can be used.

Scenario Two:

By introducing the MQ queue, the Ehcache of each application server listens to MQ messages synchronously, so that the quasi-synchronous update data can be achieved to some extent, via MQ push or pull, but because of the network speed between different servers, Therefore, it is not possible to fully achieve strong consistency. The same is true with distributed coordination notification components such as zookeeper, based on this principle.

7, part of the project is not completely split

• before splitting
  
  Note:
  A tomcat in the deployment of multiple application war packages, each other to each other in the case of a very large number of concurrent performance degradation is very obvious.

• after splitting
  

Note:
This is actually quite common before the split, and I've always thought that there would be no such situation in the project but in fact it does. The solution is simple, each application war is only in one tomcat, so there will be no competition between the resources and the number of connections between applications, and the performance and concurrency capability commits are more obvious.

8. Performance degradation due to imperfect base platform component function
Look at the code first:

 Public void Purchase(Purchaseparam Purchaseparam,LongTimeoutsencond) {future<string> future = Threadpool.submit (NewTestrunnable (Purchaseparam, Testservice)); Logger.info ("Time out ="+timeoutsencond);if(Timeoutsencond >0){Try{future.Get(Timeoutsencond, Timeunit.seconds); Logger.info ("Time-out returned, timeout ="+timeoutsencond); }Catch(Interruptedexception e) {Logger.info ("", e); }Catch(Executionexception e) {Logger.info ("", e); }Catch(TimeoutException e) {Logger.info ("", e); }        }    }

Note:
First of all we do not say how the format of this code, first look at the implementation of the function, using the future to do time-out control, this is why? The reason is that we call the Dubbo interface above, because it is Dubbo has passed two times the package, the result of the timeout to the flooding, the programmer can only control the timeout in this way, you can see that this usage is very poor, the performance of the program has a certain impact.

9. How to quickly locate program performance bottleneck

I believe that in the positioning program performance problems, we have a lot of ways, such as the use of JDK commands, such as Jcmd,jstack,jmap,jhat,jstat,iostat,vmstat and so on, but also with Visualvm,mat, JRockit and other visual tools, what I want to say today is to use one of the simplest commands to locate which program may have performance problems, see the following:

In general we will look at the CPU and memory consumption of each process through the top command, get our process ID, and then we will see through the pstack command the individual thread IDs inside and what the corresponding thread is doing now, By analyzing multiple sets of data, you can get a solution for which threads have slow operations that affect the performance of the server. Examples are as follows:

Input command: Pstack30222Shown below:Thread 9(Thread 0x7f729adc1700(LWP30251)):#0  0x00007f72a429b720 inchSem_wait () From/lib64/libpthread.So. 0#1  0x0000000000ac5eb6 inchSemaphore::d own() ()#2  0X0000000000AC5CAC inch Queue:: Get() ()#3  0x00000000009a583f inchDBManager::p rocessupdate(Queue*) ()#4  0X00000000009A4BFB inchDbupdatethread (void*) ()#5  0x00007f72a4295851 inchStart_thread () From/lib64/libpthread.So. 0#6  0x00007f72a459267d inchClone () FROM/LIB64/LIBC.So. 6Thread 1(Thread 0x7f72a60ae7e0(LWP30222)):#0  0x00007f72a4584c95 inch_xstat () from/lib64/libc.So. 6#1  0x00007f72a45483e0 inch__tzfile_read () from/lib64/libc.So. 6#2  0x00007f72a4547864 inchTzset_internal () from/lib64/libc.So. 6#3  0x00007f72a4547b20 inchTzset () from/lib64/libc.So. 6#4  0x00007f72a4546699 inchTimelocal () from/lib64/libc.So. 6#5  0x0000000000b0b08d inchAchieve:: Getremaintime(achievetemplate*) ()#6  0X0000000000B115CA inchAchieve:: Update() ()#7  0x0000000000a197ce inchPlayer:: Update() ()#8  0x0000000000b1b272 inchPlayermng:: Tick() ()#9  0x0000000000a73105 inchGameserver:: Frametick(unsigned int) ()#10 0x0000000000a6ff80 inchGameserver:: Run() ()#11 0X0000000000A773A1 inchMain () Input command: PS-eloPid,lwp,pcpu|Grep30222Shown below:30222 30222 31.430222 30251  0.030222 30252  0.030222 30253  0.0

This can be judged out in the LWP 30222 that the thread produced a performance problem, the execution time of up to 31.4 milliseconds, and then observe nothing more than the following several statements appear problems, only need to simple troubleshooting to know the problem bottleneck.

10, about the optimization of the index
* The principle of combination index is the left principle, so you need to pay more attention when using

• The number of indexes does not require too many additions, and the performance of both clustered and secondary indexes is different when added

• The index does not contain a column with null values
  This column is not valid for this composite index as long as the column contains null values that will not be included in the index, as long as there is a column in the composite index that contains null values. So we don't want the default value of the field to be null when the database is designed.

• MySQL Index sort
  The MySQL query uses only one index, so if an index is already used in the WHERE clause, the column in order by is not indexed. So do not use sort operations where the default sorting of the database is acceptable, and try not to include multiple columns, if you need to create a composite index for those columns.

• Considerations for using Indexes
  The following operators can apply an index:
  Greater than or equal
  Between
  Inch
  Like does not start with%

  The following operators cannot apply an index:
  Not in
  Like%_ start

• Indexing Tips
  Also 1234567890, numeric types store far more storage space than strings.
  Saving storage is saving Io, reducing IO is improving performance
  It is generally more efficient to index and retrieve numbers than to index and retrieve strings.

* 11. Some points to be aware of using Redis *

• Try to set the expiration time when you add the key, otherwise the memory usage of Redis server will reach
  Maximum system physical memory, resulting in redis using VMS to reduce system performance

• Redis key should be designed as short as possible, and value should not use complex objects as much as possible.

• Converts an object to a JSON object (using a ready-made JSON library) and then into Redis,

• Convert objects into Google Open Binary Protocol objects (Google Protobuf, and JSON data
  The format is similar, but because it is binary performance, performance efficiency and space consumption are smaller than JSON;
  The disadvantage is that Protobuf's learning curve is much larger than JSON.)

• Be sure to release the connection when Redis is finished, as an example:
  
  Whether it's going back to the connection pool or just releasing it, it's all about returning the connection.

* 12. Splitting of long time-consuming methods *
Our general technique for splitting long time-consuming methods is:
* Find redundant points of business, there are many repetitive code in the code, can be properly simplified.
* Check that the Library table index is properly joined.
* Algorithm-level optimization using unit testing or stress testing for long time-consuming operations, such as reading data in large batches from a library, or long-cycle operations, or dead-loop operations, etc.
* Find the business split point, split the synchronization operation according to business requirements asynchronous, such as can use Message Queuing or multithreading asynchronous.

After the above analysis, if the method execution time is still very long, this may be the business requirements, such as:

Then whether we can consider a long time-consuming method of splitting, split into a number of short time-consuming methods are called by the initiator , so that in the case of high concurrency does not cause a long time block of a method, to some extent can improve the concurrency capability, such as:

In the following third article we will introduce the system downgrade, current limit, and some of the monitoring options. Thank you

On the path of code-level performance Optimization Change (II.)