Java Online service issues troubleshooting __java

1, business log related

If the system is abnormal or the business is abnormal, the first thought is to view the business log

To View the log tool:

Less or more

Grep

tail-f filename to see what's new in real time

PS: Avoid vim directly open large log file, because will be loaded directly into the memory of the <H1 id= "id-Online Troubleshooting-2, database-related" >2, database-related

Java application of many bottlenecks in the database, a SQL is not written to cause slow query, may bring the application of fatal harm.

If the could not get JDBC Connection appears, the interface response is slow, the thread is full, and so on,

Need to log on to the online library,

To view database connections: Show processlist, view the current database connection, indeed due to slow queries, requires manual kill 3, JVM-related

Java Virtual machine-related problems are generally the following issues: GC time is too long, OOM, deadlock, thread block, the number of threads soared and so on. Generally through the following several tools can locate the problem.

JPS command

Jinfo command

Jstat command

Jstack command

Jmap Command 3.1 oom problem

Occurrence Oom problem General Service will crash, business log will have outofmemoryerror. Oom is generally a memory leak, you need to see Oom when the JVM heap snapshots, if configured-xx:+heapdumponoutofmemoryerror, in the occurrence of oom will be in the-xx:heapdumppath heap of dump files, Combined with mat, the dump file can be analyzed to find out the cause of oom. About mat use is not detailed, Google on a heap (http://inter12.iteye.com/blog/1407492).

Ps.

1, the server's memory is generally large, so to ensure that the server's disk space is greater than the memory size

2, another manual dump heap snapshot, you can use the command jmap-dump:format=b,file=file_name pid or kill-3 pid 3.2 deadlock

Deadlock reason is two or more threads waiting for each other resources, the phenomenon is generally the emergence of thread hung live, more serious will appear in the number of threads, the system appears API alive alarm. The best way to view deadlocks is to analyze the line stacks.

Specific case can refer to the example in the Jstack command

Commands to use:

Jps-v

Jstack-l PID 3.3 thread block, number of threads exploding

Jstack-l PID |wc-l

Jstack-l pid |grep "BLOCKED" |wc-l

Jstack-l pid |grep "Waiting On condition" |wc-l

The thread block problem is generally waiting for IO, waiting for the network, waiting for the monitor lock and so on, may cause the request to time out, causes the thread number to soar causes the system 502 and so on.

If this problem occurs, it is primarily concerned with state information such as Jstack blocked, waiting on condition, and waiting on monitor entry.

If a large number of threads are in "Waiting for monitor entry":

It is possible that a global lock blocks a large number of threads.

If the thread dump file printed in a short time reflects that, over time, the waiting for monitor entry threads are growing, and there is no reduction in the trend, which may mean that some threads have been stuck in the critical section for too long, and that more and more new threads have been slow to enter the critical section.
If a large number of threads are in "waiting on condition": it may be that they run off to get third-party resources and delay getting response, causing a large number of threads to enter the waiting state. So if you find that a large number of threads are in wait on condition, looking from the thread stack and waiting for the network to read and write, this may be a symptom of a network bottleneck, because the network blocking causes the thread to fail. 3.3 gc Time is too long

Todo

Put one article first to occupy the pit: http://www.oracle.com/technetwork/cn/articles/java/g1gc-1984535-zhs.html 4, server problem 4.1 CPU

Top Command (Refer to https://linux.cn/article-2352-1.html)

Focus on CPU load and CPU-consuming processes

Since the server is now a virtual machine, but also to pay attention to St (ST's full name is steal time, is assigned to the other virtual machine to run the actual CPU times on the task)

Common Interactive Commands:

h Help, very useful

R: Reverse Sort

X: Sort Degauliang Display (vertical column)

Y will run the process highlighting (rampage)

Shift+> or shift+<: Toggle Sort Field

D or S: Set the refresh interval for display

F: Field management settings display fields

K:kill process 4.2 memory

Free command:

Free-m-C10-S1

-M: shown in MB, others have-k-g-B

-S: Continue to observe memory usage for how many seconds interval

-C: How many times to observe

vmstat Command: (http://man.linuxde.net/vmstat)

Vmstat 1

1 means every 1s output, 10 for output 10 times

Two parameters need attention

R: Running the number of processes in the queue, this value can also determine whether the need to increase the CPU. (longer than 1)

B: Number of processes waiting for IO.
4.3 IO

iostat Command (http://www.orczhou.com/index.php/2010/03/iostat-detail/)

Iostat-m 1 10-m: Some columns with block are used to force MB to 1 10: The data display is refreshed every 1 seconds, displaying a total of 10 times