Some Linux System Service optimization/problem locating skills

Document directory

• Optimization-Server Memory "eat" swap Problems
• Optimization-system comparison
• Service Problems (Slow, slow, and down)
• Network problem (slow, card) Locating
• Miscellaneous-Firefox browser connection timeout
• Miscellaneous-IRQ affinity

Optimization-Server Memory "eat" swap Problems

Swap memory will certainly be slower, so we must try our best to avoid using it. Even if the system parameter VM is set. swappiness = 0, the system will also use swap, and the test finds that the use of swap is not necessarily related to the Java service process, even if it cannot afford any Java Process, swap will also be used.

Therefore, the value of the VM. swappniess parameter only tells the system to "actively" use swap (set to 100), or "Try not to" use swap (set to 0 ).
For general database servers, the VM. swappniess parameter must be set to 0 for smooth data reading and writing.

If the system does not use any swap, run the following command to disable the system's swap memory:

Swapoff-

Of course, you need to use the following command repeatedly to "make sure" swap really does not need to be used:

Free-MT

The principle is: Mem: total is always greater than total: used. Make sure that this condition is always true. Writing a script repeatedly on the server for a period of time is a way to verify this principle.

Caution: disabling swap will have serious consequences-If memory consumption is completed, the system may crash.

Optimization-system comparison

If the server is configured and the running service is the same and the effect is poor, you can compare the processes and parameters on the system:
Execute PS-EF/sysctl-A respectively and compare the process/system configuration on the two servers to see if they are different?
In addition, PS auxf can display the tree relationship between parent and child processes, and PS-elf can display it to threads.

Service Problems (Slow, slow, and down)

Check System: TCP connection, memory, CPU load, and Io information.

Check the access logs (for example, access. Log) of the front-end proxy service to analyze the problems of centralized access attacks and request distribution. For example, if there is a problem with some pages, users may frantically click the URL of the page, resulting in a burst of requests for this URI within a short period of time.

Check the number of connections in the standard output log (for example, stdout. Log) of the application server. The number of concurrent connections on the application server is limited, for example, resin Enterprise Edition:

......
[05:08:48. 687] [05:08:48. 687] webapp [http: // localhost: 8080]
Stopping
[05:09:03. 375] [05:09:03. 375] webapp [http: // localhost: 8080]
Closing with 512 active requests.
......

Check the standard error output log (for example, stderr. Log) of the application server. The problems found here should be obvious.

Check application logs (including business logs). Whether the problem can be found here depends on the log architecture design of the application. If the programmer "Eats" all exceptions, the fairy cannot find the problem-unless the code is checked.

If no problem is found above, go to the tool. For Java EE applications, JDK's built-in jmap can check for memory leaks, and jstack can check for thread blocking (threads may be blocked by slow database queries, Io busy due to frequent logging, thread object deadlocks, and other problems) jstat can be used to check GC conditions (this can guide you to adjust JVM startup parameters ).

If you still cannot find the problem, you can check your code ......

Network problem (slow, card) Locating

View network quality
. Run the ping command, specify the package size with the parameter-L, specify the number of packets with the value-N, and specify the number of packets with the value-F flood to check packet loss and understand the network quality.

View route information
. For example: MTR-r -- C www.baidu.com or Traceroute
-N www.baidu.com
.

View socket statistics
. SS-S:
Total: 215 (kernel 216)
TCP: 159 (estab 115, closed 21, orphaned 0, synrecv 0, timewait 9/0), ports 72

Transport total IP IPv6
* 216 --
Raw 0 0 0
UDP 11 11 0
TCP 138 138 0
Inet 149 149 0
Frag 0 0 0

Check the enabled network port
. SS-L.

View the network port enabled by the Process
. SS-Pl.

View captured packets
. Tcpdump, analyzing network packet capture is a fine job ......

View HTTP request playback on the client
. Create the server: NC-l 8888, and then request the client to access: http: // (serverhost): 8888
. You can receive the HTTP request from the client on the server side, analyze the request header information, and you can find problems with the client-the server is often wronged.

Miscellaneous-Firefox browser connection timeout

There is always a problem after the first connection or idle for a period of time.

Problems only occur in the company's internal network that centrally accesses this site.

Location: when many users access the local area network at the same time, the first connection to the server times out.

This problem occurs because the tcp_tw_recycle parameter of the system is set to 1. Run sysctl net. ipv4.tcp _ tw_recycle = 0.

Miscellaneous-IRQ affinity

Run the TOP command on a multi-CPU server and enter 1. if not all the CPUs are carrying the load, the Service may be affected due to high load.

Location: it is possible that the server is executing IRQ affinity-binding specific hardware interruptions to a specific CPU, disperses and balances each interruption to a different CPU to obtain greater performance processing capabilities.

Restart system services irqbalance and application services. Irqbalance is a system service used to automatically bind and balance IRQ.