EM 12c: OMS Failed to start After restarting the Repository Database or reboot of the server, omsrestarting
EM 12c: OMS Failed to start After restarting the Repository Database or reboot of the server
This problem was found when Oms was started today and cannot be started:
[Oracle @ ora11g64 bin] $ emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996,201 2 Oracle Corporation. All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server is Down
View logs:
Log Record 1:
/U01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1/sysman/log
2014-09-19 13:57:06, 329 [Thread-1] INFO wls. OMSController run.1744-<OUT> FAILED_NOT_RESTARTABLE
2014-09-19 13:57:06, 330 [Thread-1] INFO wls. OMSController run.1744-<OUT>
13:57:06, 331 [Thread-1] INFO wls. OMSController run.1744-<OUT> sleep for 10 seconds...
13:57:16, 334 [Thread-1] INFO wls. OMSController run.1744-<OUT> status of EMGC_OMS1:
2014-09-19 13:57:16, 336 [Thread-1] INFO wls. OMSController run.1744-<OUT>
2014-09-19 13:57:16, 337 [Thread-1] INFO wls. OMSController run.1744-<OUT> FAILED_NOT_RESTARTABLE
2014-09-19 13:57:16, 401 [Thread-1] INFO wls. OMSController run.1744-<OUT>
13:57:16, 401 [Thread-1] INFO wls. OMSController run.1744-<OUT> timed out waiting for EMGC_OMS1 to be RUNNING
2014-09-19 13:57:16, 402 [Thread-1] INFO wls. OMSController run.1744-<OUT> status of node manager:
13:57:16, 402 [Thread-1] INFO wls. OMSController run.1744-<OUT> Currently connected to Node Manager to monitor the domain GCDomain.
2014-09-19 13:57:16, 403 [Thread-1] INFO wls. OMSController run.1744-<OUT> 1
13:57:16, 413 [Thread-1] INFO wls. OMSController run.1744-<OUT> Successfully disconnected from Node Manager.
13:57:16, 416 [Thread-1] INFO wls. OMSController run.1744-<OUT> _ END _
13:57:16, 416 [Thread-2] INFO wls. OMSController run.1744-<ERR> _ END _
13:57:16, 866 [main] INFO wls. OMSController processCmd.1392-wlst process is finished
2014-09-19 13:57:16, 867 [main] INFO wls. OMSController processCmd.1394-wlst process exited with code 1
2014-09-19 13:57:16, 867 [main] INFO wls. OMSController processCmd.1396-wlst outstream joined
2014-09-19 13:57:16, 868 [main] INFO wls. OMSController processCmd.1398-wlst errstream joined
2014-09-19 13:57:16, 868 [main] INFO wls. OMSController processCmd.1406-process exited with code 1
13:57:16, 880 [main] DEBUG oms. StatusOMSCmd processStatusOMS.159-HTTP port in emgc property file: 7203
13:57:16, 881 [main] DEBUG oms. StatusOMSCmd processStatusOMS.160-HTTPS port in emgc property file: 7302
13:57:16, 881 [main] DEBUG oms. StatusOMSCmd processStatusOMS.166-Webtier HTTP port in emgc property file: 7789
13:57:16, 882 [main] DEBUG oms. StatusOMSCmd processStatusOMS.167-Webtier HTTPS port in emgc property file: 7801
13:57:16, 882 [main] DEBUG oms. StatusOMSCmd processStatusOMS.186-Using https port.
13:57:16, 882 [main] DEBUG oms. StatusOMSCmd processStatusOMS.203-Using webtier https port.
13:57:17, 074 [main] DEBUG oms. StatusOMSCmd pingOMSCheck.341-Attempting to connect to https: // ora11g64: 7302/em/faces/logon/core-uifwk-console-login
13:57:18, 263 [main] DEBUG oms. StatusOMSCmd pingOMSCheck.341-Attempting to connect to https: // ora11g64: 7801/empbs/check
13:57:18, 281 [main] DEBUG oms. StatusOMSCmd processStatusOMS.240-
Console page status code is 404
13:57:18, 330 [main] DEBUG oms. StatusOMSCmd processStatusOMS.242-
Pbs page status code is 200
13:57:18, 330 [main] DEBUG oms. StatusOMSCmd processStatusOMS.245-console header is null
13:57:18, 330 [main] DEBUG oms. StatusOMSCmd processStatusOMS.247-pbs header is null
13:57:18, 330 [main] DEBUG oms. StatusOMSCmd processStatusOMS.287-Encountered error while hitting page. Status code is 404
13:57:18, 330 [main] INFO oms. StatusOMSCmd processStatusOMS.294-Failed to connect to em login page
13:57:18, 330 [main] INFO oms. StatusOMSCmd processStatusOMS.295-Encountered error while hitting page. Status codes are 404 for console and 200 for PBS
13:57:18, 331 [main] INFO oms. StatusOMSCmd processStatusOMS.296-Connection exception when trying to hit OMS page. OMS is down
13:57:18, 331 [main] INFO wls. OMSController statusOMS.1354-statusOMS finished with result: 8
13:57:18, 331 [main] INFO wls. OMSController startOMS.940-startOMS finished with result: 1
13:57:18, 332 [main] DEBUG oms. StatusOMSCmd processStatusOMS.159-HTTP port in emgc property file: 7203
13:57:18, 332 [main] DEBUG oms. StatusOMSCmd processStatusOMS.160-HTTPS port in emgc property file: 7302
13:57:18, 332 [main] DEBUG oms. StatusOMSCmd processStatusOMS.166-Webtier HTTP port in emgc property file: 7789
13:57:18, 332 [main] DEBUG oms. StatusOMSCmd processStatusOMS.167-Webtier HTTPS port in emgc property file: 7801
13:57:18, 333 [main] DEBUG oms. StatusOMSCmd processStatusOMS.186-Using https port.
13:57:18, 333 [main] DEBUG oms. StatusOMSCmd processStatusOMS.203-Using webtier https port.
13:57:18, 333 [main] DEBUG oms. StatusOMSCmd pingOMSCheck.341-Attempting to connect to https: // ora11g64: 7302/em/faces/logon/core-uifwk-console-login
13:57:18, 458 [main] DEBUG oms. StatusOMSCmd pingOMSCheck.341-Attempting to connect to https: // ora11g64: 7801/empbs/check
13:57:18, 480 [main] DEBUG oms. StatusOMSCmd processStatusOMS.240-console page status code is 404
13:57:18, 481 [main] DEBUG oms. StatusOMSCmd processStatusOMS.242-pbs page status code is 200
13:57:18, 481 [main] DEBUG oms. StatusOMSCmd processStatusOMS.245-console header is null
13:57:18, 481 [main] DEBUG oms. StatusOMSCmd processStatusOMS.247-pbs header is null
13:57:18, 482 [main] DEBUG oms. StatusOMSCmd processStatusOMS.287-Encountered error while hitting page. Status code is 404
13:57:18, 482 [main] INFO oms. StatusOMSCmd processStatusOMS.294-Failed to connect to em login page
13:57:18, 482 [main] INFO oms. StatusOMSCmd processStatusOMS.295-Encountered error while hitting page. Status codes are 404 for console and 200 for PBS
13:57:18, 484 [main] INFO oms. StatusOMSCmd processStatusOMS.296-Connection exception when trying to hit OMS page. OMS is down
13:57:18, 499 [main] INFO wls. OMSController statusOMS.1354-statusOMS finished with result: 8
Log File: 2:
2014-09-16 13:18:38, 209 [Thread-1] INFO wls. OMSController run.1744-<OUT>
Successfully started server EMGC_OMS1...
2014-09-16 13:18:38, 229 [Thread-1] INFO wls. OMSController run.1744-<OUT>
New status of EMGC_OMS1:
2014-09-16 13:18:38, 229 [Thread-1] INFO wls. OMSController run.1744-<OUT>
13:18:38, 229 [Thread-1] INFO wls. OMSController run.1744-<OUT> RUNNING
Log 3:
[Oracle @ ora11g64 logs] $ pwd
/U01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs
Repos details fetched from credstore
Fetched repository credentials from Credential Store
Invalid Connection Pool. ERROR = User credentials doesn't match the existing ones
Failed to verify repository
Java. lang. NullPointerException
Metalink: File ID 1459936.1
CAUSE
OMS will not be able to make the connection to the database and restarting also fails, if the background processes are not stopped completely.
SOLUTION
As a workaround, follow the steps.
1. Stop the OMS using
/Bin>./emctl stop oms-all
2. Check the following processes
$ Ps-ef | grep java
$ Ps-ef | grep opmn
$ Ps-ef | grep http
Please kill all the processes related to OMS user.
3. Restart the Repository Database.
4. Now start the OMS
OMS_HOME/bin>./emctl start oms
To prevent the issue in future:
Apply the Patch 13466904 to the 12.1.0.1.1 (With BP1) OMS home. After applying the patch, we dont need to restart the OMS if the reposiotry database is restarted.
This issue is fixed in 12.1.0.2
My Environment
[Oracle @ ora11g64 bin] $./emctl stop oms-all
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996,201 2 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Already Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
[Oracle @ ora11g64 bin] $ ps-ef | grep java
Oracle
47534708 2? 00:02:40/u01/app/oracle/Middleware/jdk16/jdk/bin/java-server-Xms256m-Xmx1740m-XX: MaxPermSize = 768 M-XX:-DoEscapeAnalysis-XX: + UseCodeCacheFlushing-XX: + UseConcMarkSweepGC-XX: + UseParNewGC-XX: + CMSClassUnloadingEnabled-XX: CompileThreshold = 8000-XX: PermSize = 128 m-Dweblogic. name = EMGC_OMS1-Djava. security. policy =/u01/app/oracle/Middleware/wlserver_10.3/server/lib/weblogic. policy-Dweblogic. productionModeEnabled = true-Dweblogic. system. bootIdentityFile =/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/data/nodemanager/boot. properties-Dweblogic. nodemanager. serviceEnabled = true-Dweblogic. security. SSL. ignoreHostnameVerification = true-Dweblogic. reverseDNSAllowed = false-DINSTANCE_HOME =/u01/app/oracle/Middleware/gc_inst/em/EMGC_OMS1-DORACLE_HOME =/u01/app/oracle/Middleware/oms-Ddomain. home =/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain-Djava. awt. headless = true-Ddomain. name = GCDomain-Doracle. sysman. util. logging. mode = dual_mode -Djbo.doc onnectionpooling = true-Djbo. txn. disconnect_level = 1-Docm. repeater. home =/u01/app/oracle/Middleware/oms-Djbo. ampool. minavailablesize = 1-Djbo. ampool. timetolive =-1-Djbo. load. components. lazily = true-Djbo. recyclethreshold = 50-Djbo. ampool. maxavailablesize = 50-Djava. security. egd = file: // dev /. /urandom-Dweblogic. debug. debugSecurityAtn = true-Dweblogic. debug. debugWebAppSecurity = true-Dweblogic. SSL. loginTimeoutMillis = 300000-Djps. auth. debug = true-Djps. authz = ACC-Djps. combiner. optimize. lazyeval = true-Djps. combiner. optimize = true-Djps. policystore. hybrid. mode =-Djps. subject. cache. key = 5-Djps. subject. cache. ttl = 600000-Xbootclasspath/p: /u01/app/oracle/Middleware/oms/sysman/jlib/kernel-Djdkpatchlog =/u01/app/oracle/Middleware/oms/sysman/log/diagpatch_bug11725986.log-Doracle. apm. home =/u01/app/oracle/Middleware/oms/apm/-DAPM_HELP_FILENAME = oesohwconfig. xml-da-Dplatform. home =/u01/app/oracle/Middleware/wlserver_10.3-Dwls. home =/u01/app/oracle/Middleware/wlserver_10.3/server-Dweblogic. home =/u01/app/oracle/Middleware/wlserver_10.3/server-Dcommon. components. home =/u01/app/oracle/Middleware/oracle_common-Djrf. version = 11.1.1-Dorg. apache. commons. logging. log = org. apache. commons. logging. impl. jdk14Logger-Ddomain. home =/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain-Djrockit. optfile =/u01/app/oracle/Middleware/oracle_common/modules/oracle. jrf_11.1.1/jrocket_optfile.txt-Doracle. server. config. dir =/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/servers/EMGC_OMS1-Doracle. domain. config. dir =/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig-Digf. arisidbeans. carmlloc =/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/carml-Digf. arisidstack. home =/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/arisidprovider-Doracle. security. jps. config =/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/config/fmwconfig/jps-config.xml-Doracle. deployed. app. dir =/u01/app/oracle/Middleware/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_ WL_user-Doracle. deployed. app. ext =/--Dweblogic. alternateTypesDirectory =/u01/app/oracle/Middleware/oracle_common/modules/oracle. ossoiap_11.1.1,/u01/app/oracle/Middleware/oracle_common/modules/oracle. oamprovider_11.1.1-Djava. protocol. handler. pkgs = oracle.mds.net. protocol-Dweblogic. jdbc. remoteEnabled = false-Doracle. apm. home =/u01/app/oracle/Middleware/oms/apm/-DAPM_HELP_FILENAME = oesohwconfig. xml-Dweblogic. management. discover = false-Dweblogic. management. server = https: // ora11g64: 7102-Dwlw. iterativeDev = false-Dwlw. testConsole = false-Dwlw. logErrorsToConsole = false-Dweblogic. ext. dirs =/u01/app/oracle/Middleware/patch_wls1035/profiles/default/sysext_manifest_classpath weblogic. server
Oracle 10120 8047 0 00:00:00 pts/2 grep java
[Oracle @ ora11g64 bin] $ kill-9
4753
[Oracle @ ora11g64 bin] $ ps-ef | grep java
Oracle 10147 8047 0 00:00:00 pts/2 grep java
[Oracle @ ora11g64 bin] $ ps-ef | grep opmn
Oracle 10151 8047 0 00:00:00 pts/2 grep opmn
[Oracle @ ora11g64 bin] $ ps-ef | grep http
Oracle 10155 8047 0 00:00:00 pts/2 grep http
[Oracle @ ora11g64 bin] $ sqlplus/as sysdba
SQL * Plus: Release 11.2.0.1.0 Production on Fri Sep 19 14:54:56 2014
Copyright (c) 1982,200 9, Oracle. All rights reserved.
Connected:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0-64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SYS @ ORA11G> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SYS @ ORA11G> startup
ORACLE instance started.
Total System Global Area 2137886720 bytes
Fixed Size 2215064 bytes
Variable Size 822084456 bytes
Database Buffers 1308622848 bytes
Redo Buffers 4964352 bytes
Database mounted.
Database opened.
SYS @ ORA11G> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0-64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
[Oracle @ ora11g64 bin] $./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
Copyright (c) 1996,201 2 Oracle Corporation. All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Successfully Started
Oracle Management Server is Up
Another problem may occur:
This is because there is insufficient free space for the files in the archive path. After deletion, restart the file.
However, my experiment is in non-archive mode, so it won't work.
The result is correct. Thanks to metalink!
2014/9/19
Tyger beijing