WebLogic 10.3.0 suspends solution under aix6.1 and jdk1.6

 

Last week, weblogic10.3.0 was installed under aix6.1 and the hacmp cluster environment was configured. However, we encountered a suspension problem in the next few days and had to work overtime for a day.

Symptom description:

After Weblogic is started, it will be hang in 10 to 30 minutes, and the application and management console will not be accessible. After you force the kill-9 PID, the port cannot be released. Run the rmsock command to view the port display: Wait for exiting processes to be cleaned up before removing the socket.

Analysis and Handling Process

1. Use PS-Ef | grep Java to locate the Weblogic process and run kill-3 PID every three minutes to generate the javacore file under the domain directory.

2. Analyze WebLogic logs and find the following content:

<Aug 21,200 9 4:33:37 am CDT> <error> <weblogicserver> <BEA-000337> <[stuck] executethread: '1' for queue: 'weblogic. kernel. default (Self-tuning )'Has been busy for "620" seconds working on the request

"Weblogic. Work. selftuningworkmanagerimpl $ workadapterimpl @ 20de20de", which is more than the configured time (stuckthreadmaxtime) of "600" seconds. stack trace:

Java.net. socketoutputstream.Socketwrite0(Native method)

Java.net. socketoutputstream.Socketwrite(Socketoutputstream. Java: 103)

......

<Aug 21,200 9 4:34:37 am CDT> <error> <weblogicserver> <BEA-000337> <[stuck] executethread: '1' for queue: 'weblogic. kernel. default (Self-tuning )'Has been busy for "680" seconds working on the request

"Weblogic. Work. selftuningworkmanagerimpl $ workadapterimpl @ 20de20de", which is more than the configured time (stuckthreadmaxtime) of "600" seconds. stack trace:

Java.net. socketoutputstream.Socketwrite0(Native method)

Java.net. socketoutputstream.Socketwrite(Socketoutputstream. Java: 103)

......

3. Use IBM thread and monitor dump analyzer for Java to analyze the generated thread dump and find the following two thread information:

3 xmthreadinfo "[active] executethread: '5' for queue: 'weblogic. kernel. default (Self-tuning) '"TID: 0 × 39cbed00, j9thread_t: 0 × 3751c83c, state: R, PRIO = 5

3xmthreadinfo1 (native thread ID: 0xce1db, native priority: 0 × 5, native policy: Unknown)

4 xestacktraceJava/NET/plainsocketimpl. socketclose0(Native method)

4 xestacktrace at Java/NET/plainsocketimpl. socketpreclose (plainsocketimpl. Java: 706)

4 xestacktrace at Java/NET/plainsocketimpl. Close (plainsocketimpl. Java: 540)

4 xestacktrace at Java/NET/sockssocketimpl. Close (sockssocketimpl. Java: 1041)

4 xestacktrace at Java/NET/socket. Close (socket. Java: 1343)

4 xestacktrace at WebLogic/socket/socketmuxer. closesocket (socketmuxer. Java: 475)

4 xestacktrace at WebLogic/socket/socketmuxer. cancelio (socketmuxer. Java: 813)

4 xestacktrace at WebLogic/socket/socketmuxer $ timerlistenerimpl. timerexpired (socketmuxer. Java: 1021 (compiled code ))

4 xestacktrace at WebLogic/Timers/Internal/timerimpl. Run (timerimpl. Java: 273 (compiled code ))

4 xestacktrace at WebLogic/work/selftuningworkmanagerimpl $ workadapterimpl. Run (selftuningworkmanagerimpl. Java: 516 (compiled code ))

4 xestacktrace at WebLogic/work/executethread.exe cute (executethread. Java: 201 (compiled code ))

4 xestacktrace at WebLogic/work/executethread. Run (executethread. Java: 173)

3 xmthreadinfo "executethread: '7' for queue: 'weblogic. Socket. muxer '" TID: 0 × 35381d00, j9thread_t: 0 × 35385864, state: R, PRIO = 5

3xmthreadinfo1 (native thread ID: 0xb916f, native priority: 0 × 5, native policy: Unknown)

4 xestacktraceWebLogic/socket/posixsocketmuxer. Poll(Native method)

4 xestacktrace at WebLogic/socket/posixsocketmuxer. processsockets (posixsocketmuxer. Java: 102 (compiled code ))

4 xestacktrace at WebLogic/socket/socketreaderrequest. Run (socketreaderrequest. Java: 29)

4 xestacktrace at WebLogic/socket/socketreaderrequest.exe cute (socketreaderrequest. Java: 42)

4 xestacktrace at WebLogic/kernel/executethread.exe cute (executethread. Java: 145)

4 xestacktrace at WebLogic/kernel/executethread. Run (executethread. Java: 117)

4. Only the two execution threads are in the running state. One is close () and the other is poll (). Everything else is in the blocked or wait status.

5. After Metalink query and confirmation with the 800 support staff, this is a long-standing bug of Weblogic on the aix jvm. It has been found in different versions since 8.1.4. The reason is that the underlying JVM socket Implementation of IBM is not compatible with WebLogic, So patch is required.Cr370915_1030ga. Jar solution.

Procedure

1. In the Weblogic STARTUP script, find the classpath line

2. In the classpath variableFirstAdd a patch jar package
Eg: classpath = "$ {classpath }$ {classpathsep }$ {medrec_weblogic_classpath }"
->
Classpath =/path/cr370915_1030ga.jar: "$ {classpath }$ {classpathsep }$ {medrec_weblogic_classpath }"

3. the above operations only work for this domain. To work for all domains, you can add the weblogic_classpath = prefix to the commenv. Sh file in the common/bin/directory.

Summary

This bug is common on platforms combined with weblgoic and ibm jvm. If there is related log information, we can conclude that the cr370915 patch is required.

Update: My patch here is only for WebLogic 10.3.0.0. For other versions, you can use smart update to download it.

Patches for WLS 8.x can be found in my Oracle Support. open the patches & updates tab. search for Patch id 8173442 for the patches for WLS 8.1mp3, 8.1mp4, and 8.1mp5. search for Patch id 8179792 for the patch for WLS 8.1mp6.

Patches for WLS 9.x and higher can be downloaded from SMART Update using these patch IDs and passcodes:

--------------
Patch Repository Information
--------------
WLS version | Patch id | passcode
---- + ------
9.2 | t4dv | 7c7pyv9b
9.2mp1 | hzhq | ptuyccsi
9.2mp2 | wjd2 | gu1cw2ab
9.2mp3 | gnlt | 8j9l6q4y
10.0 | pmaj | 9uq69llt
10.0mp1 | itvl | k8rbhqq2
10.3 | 9yt5 | i1db5qsv

If the production machine cannot be connected to the Internet, you can

1. using smartupdate in offline mode ======================== you can apply the patch using smartupdate the following steps:

1. Download the patch using smartupdate on another machine with Internet access.
2. Copy the files (for example e5w8. jar and wgqj. jar) and patch-catalog.xml from your machine with Internet access to the offline machine. for example, say you have a test environment running on a Windows box. your production environment is running on UNIX. you might copy the jar files from % bea_home %/utils/BSU/cache-Dir to $ bea_home/utils/BSU/cache-Dir.
3. When a machine connects to SMART Update, the catalog of patches is always updated automatically. Thus, when a patch is being copied to an offline machine, the patch-catalog.xml file must also be copied over.
4. Run smartupdate in offline mode and apply patches and patch sets. This can be done using the smartupdate command-line interface (see http://download.oracle.com/docs/cd/E14759_01/doc.32/e14143/commands.htm#i1074489 ).
5. This is the syntax for the command to install a patch :.

/BSU. sh-prod_dir = <weblogic_home>-patchlist = <patchid>-verbose-installfor example ,. /BSU. sh-prod_dir =/opt/BEA/weblogic92-patchlist = e5w8-verbose-install. /BSU. sh-prod_dir =/opt/BEA/weblogic92-patchlist = wgqj-verbose-install2. applying the patch to the classpath manually ======== ==================

1. You can apply the patch to the offline system manually by extracting the actual patch and adding it to the classpath on the offline system: extract the actual patch JAR file. if you downloaded the patch using smartupdate, it will be in the form <patch_id>. jar (for example: e5w8. jar ). inside this jar file is the actual patch JAR file, which will be of the form cr326566_92mp3.jar. extract the latter file for the following steps.
2. Add the extracted JAR file as the first element of the classpath of the Admin Server as well as the managed servers in the domain.
3. If you are starting servers using the Weblogic STARTUP script, update the classpath in the startup script like this: Set classpath = <patch_dir>/jars/cr326566_92mp3.jar; % classpath % (Windows) classpath = <patch_dir>/jars/cr326566_92mp3.jar: $ classpath (UNIX) Where patch_dir is the directory on your local machine where you extracted/saved the patch file.
4. Similarly, if you are starting servers using Node Manager, add the patch jar to the beginning of the class path argument in the server start tab for the server (s ).

I usually use the second method. It is quick and convenient for a single patch. smartupdate can be installed separately, but it will let you select the main directory of the BEA application. Different versions and patches can be different under the platform. Of course there is no Bea version of Aix on windows, but you only need to create a directory and copy register. xml.

 

 

Note:

This article Reprinted from: http://www.hashei.me/2009/08/cr370915_in_weblogic10-3_and_jdk1-6.html