Java problem targeting

Here are some of the things we have encountered, such as error or need to supplement the content, please modify directly.   Memory Exception:? java.lang.outofmemoryerror: permgen space     "Resin hot deployment, Reload the jar package, which is prone to this problem      the persistent generation is too small to cause this problem, you need to modify the JVM parameters:-xx:permsize=32m -xx:maxpermsize= 64m     "This is a common test environment, the problem is basically not on the line, our perm memory is sufficient, and the PERM&NBSP;GC is turned on.?  java.lang.outofmemoryerror: java heap space     " The general situation is to load a large amount of data from the database or cache, or the user uploads a large number of files, there is our code problem, collection class use inappropriate, scope of the problem.      "In general, this exception is difficult to find on the wire as long as there is no memory leak problem with the code because it triggers the GC.      Solution: Restart, repair code hidden trouble?  java.lang.outofmemoryerror: gc overhead limit  exceeded     this is due to GC using the Concurrency collection algorithm, and the-XX:-USEGCOVERHEADLIMIT option is added to the JVM startup parameters.      is currently only encountered in hive applications, and online applications are generally CMS algorithms, and this does not happen.      Solution: Add heap size or disable the option above.?  a lot of times memory anomalies do not appear to be abnormal, and you have not reached this point, and your system is unusable. This time you need to take the initiative to check the memory usage:     "Query the GC status to see if the JVM is doing GC operation:jstat -gcutil <pid> <time_span>[@tc -152-92 ~]$ jstat -gcutil 16590 3000  s0      S1     E      O       P     YGC     YGCT     fgc    fgct     gct     0.00    0.00  85.96  90.60  54.20   3336     0.781 38188 13952.038 13952.819  0.00   0.00   91.42  90.60  54.20   3336    0.781 38189  13952.565 13953.346  0.00   0.00  97.43  90.60   54.20   3336    0.781 38190 13952.960  13953.741 like the above situation is problematic: FULL&NBSP;GC times is much larger than YOUNG&NBsp GC frequency, very unreasonable. YOUNG&NBSP;GC are generally very fast, and FULL&NBSP;GC is at least 100ms+ (online), so FULL&NBSP;GC more then we need to increase the JVM's heap parameters.          query Java Object Memory usage to see if the Java object in memory is reasonable. [@zjm -110-88 ~]$ jmap -histo 2234 | head -10 num       #instances           #bytes   class  Name----------------------------------------------   1:        3373503     2209452824  [C   2:        3334031      133361240  java.lang.String    3:           260       101301344  [Lcom.caucho.util.LruCache$CacheItem;   4:         326846       63127704  [ljava.lang.object;   5:         151274       50828064  com.wap.sohu.mobilepaper.model.newscontent    6:         19812        45474976  [i   7:        110209        40197776  [B   8:         145988       30902344  [ ljava.util.hashmap$entry;   9:       1846859        29549744  java.lang.Object  10:         270121       19448712   Com.wap.sohu.mobilepaper.model.xml.image     ?  Common JVM Parameters:         -xx:maxpermsize=512m -xx: permsize=512m -xss128k        -xmx4096m -xms4096m - XMN1024M&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;-XX:+USECONCMARKSWEEPGC&NBSP;-XX: cmsinitiatingoccupancyfraction=85 -xx:+printgcdetails          -xx:maxtenuringthreshold=30       cpu exception:?  run a lot of threads, There are a number of threads in our application that perform tasks asynchronously, and a point or event triggers a large number of threads to perform operations at the same time, causing CPU resources to be strained.?  programs run slowly, such as a large number of computation operations, frequent loop traversal.  io operation Many, For example, frequent print logs, frequent network access (Mysql,memcache).?  too many synchronization operations. For example synchronize?  in general, we are observing the JVM stack information to identify program anomalies, mainly see java.lang.Thread.State this value, general blocked and runnable need to focus attention. The blocked state must be locked, such as frequent IO operations that cause the resource block or explicit locking in our code. The runnable state is theoretically normal, but it is likely that logical processing is too slow (such as network IO or computation) or that calls frequently result in a lengthy code execution time, which also needs to be optimized. [@yd -80-133 ~]$ jstack 1344 2013-06-08 16:15:42full thread dump java  hotspoT (TM)  64-Bit Server VM  (20.8-b03 mixed mode):  "Pool-40-thread-5"  prio= 10 tid=0x000000005cea6800 nid=0x639c runnable [0x00000000493c5000]    java.lang.thread.state: runnable        at  Java.net.SocketInputStream.socketRead0 (Native method)         at  java.net.socketinputstream.read (socketinputstream.java:129)          at com.mysql.jdbc.util.readaheadinputstream.fill (readaheadinputstream.java:114)          at  Com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary (readaheadinputstream.java:161)         at com.mysql.jdbc.util.readaheadinputstream.read ( readaheadinputstream.java:189)         - locked < 0x000000074c115a08>  (A com.mysql.jdbc.util.readaheadinputstream)         at  com.mysql.jdbc.mysqlio.readfully (mysqlio.java:3014)         at  com.mysql.jdbc.mysqlio.reuseandreadpacket (mysqlio.java:3467)          at com.mysql.jdbc.mysqlio.reuseandreadpacket (mysqlio.java:3456)          at com.mysql.jdbc.mysqlio.checkerrorpacket (mysqlio.java:3997)          at com.mysql.jdbc.mysqlio.sendcommand (mysqlio.java:2468)          at com.mysql.jdbc.mysqlio.sqlquerydirect (mysqlio.java:2629)          at com.mysql.jdbc.connectionimpl.execsql (connectionimpl.java:2719)          - locked <0x0000000770fb3380>  (a  Com.mysql.jdbc.JDBC4Connection) &NBSP;&NBSP;&NBsp;     at com.mysql.jdbc.preparedstatement.executeinternal ( preparedstatement.java:2155)         - locked < 0x0000000770fb3380>  (a com.mysql.jdbc.jdbc4connection)          at com.mysql.jdbc.preparedstatement.executeupdate (preparedstatement.java:2450)          - locked <0x0000000770fb3380>  (a  Com.mysql.jdbc.JDBC4Connection)         at  Com.mysql.jdbc.PreparedStatement.executeBatchSerially (preparedstatement.java:2006)          - locked <0x0000000770fb3380>  (a  Com.mysql.jdbc.JDBC4Connection)         at  Com.mysql.jdbc.PreparedStatement.executeBatch (preparedstatement.java:1467)          - locked <0x0000000770fb3380>  (a com.mysql.jdbc.jdbc4connection)          at com.mchange.v2.c3p0.impl.newproxypreparedstatement.executebatch ( newproxypreparedstatement.java:1723)         at  Org.springframework.jdbc.core.jdbctemplate$4.doinpreparedstatement (jdbctemplate.java:873)          at org.springframework.jdbc.core.jdbctemplate$4.doinpreparedstatement ( JDBCTEMPLATE.JAVA:1)         at  Org.springframework.jdbc.core.JdbcTemplate.execute (jdbctemplate.java:586)          at org.springframework.jdbc.core.jdbctemplate.execute (jdbctemplate.java:614)          at org.springframework.jdbc.core.jdbctemplate.batchupdate ( jdbctemplate.java:858)         at  Com.wap.sohu.mobilepaper.dao.statistic.Statisticdao$batchupdatetask2.run (statisticdao.java:285)         at  java.util.concurrent.threadpoolexecutor$worker.runtask (threadpoolexecutor.java:886)          at java.util.concurrent.threadpoolexecutor$worker.run ( threadpoolexecutor.java:908)         at java.lang.thread.run ( thread.java:662)     ?  provides a way to view the CPU-consuming threads:1.  Find the Java Process ID number, from the top command you can see the process that consumes the most CPU. The following is the process information: [@tc -152-92 ~]$ ps -ef | grep 18950smc       18950     1 90 Dec19 ?        &NBSP;21:22:46&NBSP;JAVA&NBSP;-SERVER&NBSP;-XMX768M&NBSP;-XMS768M&NBSP;-XSS128K&NBSP;-XMN300M&NBSP;-XX: maxpermsize=128m -xx:permsize=128m -xx:+useconcmarksweepgc -xx:cmsinitiatingoccupancyfraction= 85 -xx:+printgcdetails -xx:+usemembar -dHost_home=/opt/smc -dserver_log_home=/opt/smc/log/server -xloggc:/opt/smc/log/server/check_instance_ gc.log -dserver_ip=10.11.152.92 -dserver_resources=/opt/smc/apps/server/server_apps/check_instance/ resources/ -dserver_name=check_instance com.wap.sohu.server.smcapiserver 8010 2.  View the threads that consumed the most CPU at that time: (-p  Specifies the Java process number, -h  Show all threads) [@tc -152-92 ~]$ top -p 18950 - h pid user pr ni virt res shr s %cpu %mem time+  command 18999 smc       16   0 1676m  1.0g  14m s 32.3  6.5 268:50.16 java                                                                                                                 18997 smc       16    0 1676m 1.0g  14m s 30.4  6.5 267:47.97 java                                                                                                                 18998 smc        17   0 1676m 1.0g  14m s 30.4  6.5  268:15.28 java                                                                                                                 19000&nbSp;smc       16   0 1676m 1.0g  14m  S 30.4  6.5 268:06.23 java                                                                                                                 19001 smc        15   0 1676m 1.0g  14m s  5.7  6.5   81:34.02 java  3.&nKeep stack information at the time: [@tc -152-92 ~]$ jstack 18950 > 18950.txt 4.  Turn the above CPU-consuming thread ID into 16 binary: [@tc -152-92 ~]$ pythonpython 2.4.3  (#1,  jun 11 2009,  14:09:37)  [GCC 4.1.2 20080704  (red hat 4.1.2-44)] on linux2type   "Help",  "copyright",  "credits"  or  "license"  for more information.> >> hex (18999) ' 0x4a37 ' >>> 5.  find this hexadecimal thread number in the stack file: [@tc -152-92 ~]$ vim  18950.txt "Concurrent mark-sweep gc thread"  prio=10 tid=0x000000004183b800  Nid=0x4a39 runnable "gang worker#0  (parallel cms threads)"  prio=10 tid= 0x0000000041834000 nid=0x4a35 runnable "gang worker#1  (parallel cms threads)"  prio=10 tid=0x0000000041836000 nid=0x4a36 runnable "gang worker#2  (Parallel  cms threads) " prio=10 tiD=0x0000000041837800 nid=0x4a37 runnable "gang worker#3  (parallel cms threads)"  prio=10 tid=0x0000000041839800 nid=0x4a38 runnable  found here are all GC threads consuming CPU, From the top of the process information can see this Java process opened only 1G heap, the top information shows that the memory consumption has reached 1G, just can be right. Common exceptions in the      code:?  java.lang.outofmemoryerror: unable to create new There are too many threads in the  native thread program. One scenario is that the user (SMC) that starts the program has a limit (ulimit -a view) for the maximum running thread, and the other is that many threads are in the code. When this exception occurs, we log on to the current user on the server and a resource temporarily unavailable error occurs. (If it is the root user-initiated process, it will only restart the machine)?  broken pipe  // todo ...?  too many open files   // todo ...? &NBSP;C3P0 Connection Pool Exception:attempted to use a closed or broken resource pool< Property name= "Breakafteracquirefailure"  value= "true" ></property> changed to: <property name= "Breakafteracquirefailure"  value= "false" ></property> if the parameter is true, as long as there is a failure to get a database connection once, the entire dataThe source is declared as disconnected and permanently shut down, and the service is unavailable. If the parameter is False, the program throws an exception after the database connection fails, but the data source is still valid, and can be used normally after the next attempt to obtain a successful connection . ?  Class Initialization Exception java.lang.noclassdeffounderror: could not initialize class  com.wap.sohu.mobilepaper.util.clientusercenterhttputils    >  may be your classpath path is wrong, or class load order problem (jar conflict),jinfo <pid>   View Java.class.path Properties     >  class file does not exist The     >  class throws an uncaught exception when initializing, such as a static block or static variable initialization problem.     Online Debug mode installation:10.10.76.79:/root/btrace-bin.tar.gz  This file to the target machine, create a new btrace directory, unzip it: tar  -zxvf btrace-bin.tar.gz -c btrace/Modify Permissions:cd bin/;    chmod  744 * write Btrace scripts, such as we want to monitor the input parameters and output content of a method:import com.sun.btrace.annotations.*;import static  com.sun.btrace.btraceutils.*;import java.util.*; @BTracepublic  class Response {   @OnMethod (clazz= "Com.sohu.smc.reply.core.LocalCache",  method= "Getbulk", &NBSP;[EMAIL&NBSp;protected] (Kind.return))   public static void ongetbulk (String[] keys, @ Return map<string, object> result)  {    println ("=============== ======================== ");     printarray (keys);     println (Strcat (" Params keys length: ",  str (keys.length)))     println (strcat (" Result  Length: ",  str (Size (result)));  }//     @OnMethod (clazz=" Com.sohu.smc.reply.core.CommentCursorList ",  method=" Getidlist ",  [email protected] (Kind.return))     public static void ongetidlist (@Return  List<Long>  Result)  {//      println (strcat ("Id list:",  str (Size (result))); /    }}     Run this script (samples directory provides a lot of examples, basic can be directly used): [[email protected ] bin]$ ./bin/btrace -cp build/ 15631 response.java     ==================================== ===[]params keys length:0result length:0=======================================[rp_482712136,  rp_482665158, rp_482662804, ]Params keys length:3Result length:3     btrace after running will have a certain effect on the online JVM, we'd better restart the JVM instance to restore the normal state.           btrace Common scenarios are: Monitoring method response time, monitoring thread startup, class loading, Exception information (sometimes our code eats the exception and doesn't see exactly what line of code it produces).

Java problem targeting