Cobbler CHECK:HTTPD does not appear to is running and proxying Cobbler__cobbler

The operation and maintenance DBA reflects that the production Oracle database storage is offline and the Oracle database fails to start. Error: ORA-00600 [2131] ORA-07445 [kdxlin] and other errors
1. ORA-00600 [2131] error at startup
Fri Nov 06 14:50:59 2015
ALTER DATABASE MOUNT
This instance was first to mount
Fri Nov 06 14:50:59 2015
ALTER SYSTEM SET local_listener = '(ADDRESS = (PROTOCOL = TCP) (HOST = 192.168.4.4) (PORT = 1521))' SCOPE = MEMORY SID = 'oracleplus1';
NOTE: Loaded library: System
SUCCESS: diskgroup DATA was mounted
NOTE: dependency between database oracleplus and diskgroup resource ora.DATA.dg is established
Errors in file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13221.trc (incident = 191085):
ORA-00600: internal error code, arguments: [2131], [33], [32], [], [], [], [], [], [], [], [], []
Incident details in: /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/incident/incdir_191085/oracleplus1_ora_13221_i191085.trc
Fri Nov 06 14:51:10 2015
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ORA-600 signalled during: ALTER DATABASE MOUNT ...
The reason for this error is due to: We are attempting to write a controlfile checkpoint progress record, but find we do not have the progress record generating this exception. Due to the exception of the control file, such a situation occurs, we generally use a single control file once Try, if none can consider rebuilding the control file
2. Due to bad blocks (logical / physical), database instance recovery cannot be performed
Beginning crash recovery of 2 threads
Started redo scan
kcrfr_rnenq: use log nab 393216
kcrfr_rnenq: use log nab 2
Completed redo scan
 read 4427 KB redo, 500 data blocks need recovery
Started redo application at
 Thread 1: logseq 5731, block 391398
 Thread 2: logseq 4252, block 520815
Recovery of Online Redo Log: Thread 1 Group 2 Seq 5731 Reading mem 0
  Mem # 0: + DATA / oracleplus / onlinelog / group_2.266.835331047
Recovery of Online Redo Log: Thread 2 Group 8 Seq 4252 Reading mem 0
  Mem # 0: + DATA / oracleplus / onlinelog / group_8.331.835330421
Errors in file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc (incident = 197486):
ORA-00600: internal error code, arguments: [kdxlin: psno out of range], [], [], [], [], [], [], [], [], [], [], [ ]
Incident details in: /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/incident/incdir_197486/oracleplus1_ora_13770_i197486.trc
Fri Nov 06 15:03:09 2015
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc (incident = 197487):
ORA-01578: ORACLE data block corrupted (file # 2, block # 65207)
ORA-01110: data file 2: '+ DATA / oracleplus / datafile / sysaux.257.835324753'
ORA-10564: tablespace SYSAUX
ORA-01110: data file 2: '+ DATA / oracleplus / datafile / sysaux.257.835324753'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object # 81045
ORA-00600: internal error code, arguments: [kdxlin: psno out of range], [], [], [], [], [], [], [], [], [], [], [ ]
Incident details in: /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/incident/incdir_197487/oracleplus1_ora_13770_i197487.trc
Errors in file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc:
ORA-01578: ORACLE data block corrupted (file # 2, block # 65207)
ORA-01110: data file 2: '+ DATA / oracleplus / datafile / sysaux.257.835324753'
ORA-10564: tablespace SYSAUX
ORA-01110: data file 2: '+ DATA / oracleplus / datafile / sysaux.257.835324753'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object # 81045
ORA-00600: internal error code, arguments: [kdxlin: psno out of range], [], [], [], [], [], [], [], [], [], [], [ ]
Recovery of Online Redo Log: Thread 2 Group 3 Seq 4253 Reading mem 0
  Mem # 0: + DATA / oracleplus / onlinelog / group_3.332.835330505
Hex dump of (file 14, block 62536) in trace file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc
Reading datafile '+ DATA / oracleplus / datafile / ht01.dbf' for corruption at rdba: 0x0380f448 (file 14, block 62536)
Reread (file 14, block 62536) found same corrupt data (logically corrupt)
RECOVERY OF THREAD 1 STUCK AT BLOCK 62536 OF FILE 14
Fri Nov 06 15:03:13 2015
Abort recovery for domain 0
Aborting crash recovery due to error 1172
Errors in file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc:
ORA-01172: recovery of thread 1 stuck at block 62536 of file 14
ORA-01151: use media recovery to recover block, restore backup if needed
Abort recovery for domain 0
Errors in file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13770.trc:
ORA-01172: recovery of thread 1 stuck at block 62536 of file 14
ORA-01151: use media recovery to recover block, restore backup if needed
ORA-1172 signalled during: ALTER DATABASE OPEN ...
View the data found and Bug 14301592 – Several errors by corrupt blocks shifted by 2 bytes in buffer cache during recovery caused by INDEX redo apply, which can be temporarily solved by ALLOW 1 CORRUPTION
3. Use ALLOW 1 CORRUPTION for recovery, ORA-07445 [kdxlin] error appears
Specify log: {= suggested | filename | AUTO | CANCEL}
+ DATA / oracleplus / onlinelog / group_3.332.835330505
ORA-00279: change 700860458 generated at 11/05/2015 21:20:15 needed for thread
1
ORA-00289: suggestion: + ARCHIVE / oracleplus / oracleplus_1_5731_835324843.arc
ORA-00280: change 700860458 for thread 1 is in sequence # 5731


Specify log: {= suggested | filename | AUTO | CANCEL}
+ DATA / oracleplus / onlinelog / group_2.266.835331047
ORA-00283: recovery session canceled due to errors
ORA-10562: Error occurred while applying redo to data block (file # 2, block #
70104)
ORA-10564: tablespace SYSAUX
ORA-01110: data file 2: '+ DATA / oracleplus / datafile / sysaux.257.835324753'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object # 82289
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [kdxlin () + 4088] [SIGSEGV]
[ADDR: 0xC] [PC: 0x95FB572] [Address not mapped to object] []


ORA-01112: media recovery not started
ORA-07445 [kdxlin () + 4088] did not find similar instructions. At this step, simple recovery cannot be successful. You can only skip instance recovery by setting implicit parameters and try to resetlog library

Continue to recover by setting the _allow_resetlogs_corruption parameter
SQL> startup pfile = '/ tmp / pfile.ora' mount;
ORACLE instance started.

Total System Global Area 7315603456 bytes
Fixed Size 2267384 bytes
Variable Size 2566915848 bytes
Database Buffers 4731174912 bytes
Redo Buffers 15245312 bytes
Database mounted.
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [700869927],
[0], [700860464], [], [], [], [], [], [], []
Process ID: 13563
Session ID: 157 Serial number: 3
alert log error
Fri Nov 06 19:26:39 2015
SMON: enabling cache recovery
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Errors in file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13563.trc (incident = 319140):
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [700869927], [0], [700860464], [], [], [], [], [], [], []
Incident details in: /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/incident/incdir_319140/oracleplus1_ora_13563_i319140.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Redo thread 2 internally disabled at seq 1 (CKPT)
ARC1: Archiving disabled thread 2 sequence 1
Archived Log entry 9956 added for thread 2 sequence 1 ID 0x0 dest 1:
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Errors in file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13563.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [700869927], [0], [700860464], [], [], [], [], [], [], []
Errors in file /home/app/oracle/diag/rdbms/oracleplus/oracleplus1/trace/oracleplus1_ora_13563.trc:
ORA-00600: internal error code, arguments: [kclchkblk_4], [0], [700869927], [0], [700860464], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 13563): terminating the instance due to error 600
Fri Nov 06 19:26:42 2015
Instance terminated by USER, pid = 13563
ORA-1092 signalled during: alter database open resetlogs ...
opiodr aborting process unknown ospid (13563) as a result of ORA-1092
Fri Nov 06 19:26:42 2015
ORA-1092: opitsk aborting process
Here is the more familiar ora-600 [kclchkblk_4] error, which is similar to the ora-600 [2662] error. You need to adjust the scn. Since the database version is 11.2.0.4, you cannot use the conventional method to adjust the scn. Choice of methods
5. Use oradebug method to deal with
Because it is an asm environment, other methods are relatively cumbersome to handle
[oracle @ wisetop1 ~] $ sqlplus / as sysdba

SQL * Plus: Release 11.2.0.4.0 Production on Fri Nov 6 19:30:59 2015

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup pfile = '/ tmp / pfile.ora' mount;
ORACLE instance started.

Total System Global Area 7315603456 bytes
Fixed Size 2267384 bytes
Variable Size 2566915848 bytes
Database Buffers 4731174912 bytes
Redo Buffers 15245312 bytes
Database mounted.
SQL> oradebug setmypid
Statement processed.
SQL> oradebug poke 0x06001AE70 4 0x2FAF0800
BEFORE: [06001AE70, 06001AE74) = 00000000
AFTER: [06001AE70, 06001AE74) = 2FAF0800
SQL> alter database open;

Database altered.

At this point, the database is opened successfully, and the follow-up is to deal with some bad blocks, and it is recommended that the customer logically rebuild the library.