Performance problems caused by the implicit _ complex_view_merging parameter of oracle10g

Today, I encountered a strange problem at oracle10g. One SQL statement was very fast on database 1 and very slow on Database 2. Database 2's data was imported from database 1 and the data volume was similar.

Run 0.01 s on database 1.

SQL> SELECT .*,
2 B. INCREASE_ID,
3 B. TRANSACTION_ID,
4 B. LINK_CARD_ID,
5 B. VALIDATE_FLAG,
6 B. ASSET_VALUE_SHARING,
7 B. RELATED_DEVICE_ID,
8 B. PARENT_CARD_CODE,
9 B. PROJECT_VALUE,
10 B. DELETE_FLAG,
11 B. DEPRECIATION_ADJUST_VALUE,
12 T. TRANSACTION_MODE_CODE,
13 T. TRANSACTION_NO,
14 T. TRANSACTION_FROM,
15 T. FROM_MODEL,
16 (select t. FULL_PATH
17 FROM AM_TECH_OBJECT_NODE_0900 T
18 where t. TECH_OBJECT_ID = A. DEVICE_ID
19 and t. NODE_TYPE = 2
20 and rownum = 1) AS FULL_PATH,
21 AAC. FULL_NAME CLASSIFY_FULL_PATH
22 FROM V_ASSET_CARD_0900,
23 GG_ASSET_INCREASE_ITEM B,
24 GG_ASSET_TRANSACTION T,
25 AM_ASSET_CLASSIFY AAC
26 where a. CARD_ID = B. CARD_ID
27 and B. TRANSACTION_ID = T. TRANSACTION_ID
28 and a. CLASSIFY_ID = AAC. DEVICE_CLASSIFY_ID (+)
29 and B. TRANSACTION_ID = '20140901 ';
Used time: 00: 00: 00.01
Execution Plan
----------------------------------------------------------
Plan hash value: 3643758043
Bytes -------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (% CPU) | Time |
Bytes -------------------------------------------------------------------------------------------------------------------
| 0 | select statement | 21 | 24129 | 167 (0) | 00:00:03 |
| * 1 | count stopkey |
| 2 | table access by index rowid | AM_TECH_OBJECT_NODE_0900 | 1 | 73 | 4 (0) | 00:00:01 |
| * 3 | index range scan | IDX_TECH_NODE_ID_0900 | 1 | 3 (0) | 00:00:01 |
| 4 | nested loops outer | 21 | 24129 | 167 (0) | 00:00:03 |
| 5 | nested loops | 21 | 22533 | 146 (0) | 00:00:02 |
| 6 | nested loops | 20 | 12700 | 106 (0) | 00:00:02 |
| 7 | nested loops | 20 | 10900 | 46 (0) | 00:00:01 |
| 8 | nested loops | 20 | 2000 | 6 (0) | 00:00:01 |
| 9 | table access by index rowid | GG_ASSET_TRANSACTION | 1 | 42 | 2 (0) | 00:00:01 |
| * 10 | index unique scan | PK_GG_ASSET_TRANSACTION | 1 | 1 (0) | 00:00:01 |
| 11 | table access by index rowid | GG_ASSET_INCREASE_ITEM | 20 | 1160 | 4 (0) | 00:00:01 |
| * 12 | index range scan | TRANSACTION_DETAIL_REF_TRANSAC | 20 | 1 (0) | 00:00:01 |
| 13 | table access by index rowid | GG_ASSET_CARD_0900 | 1 | 445 | 2 (0) | 00:00:01 |
| * 14 | index unique scan | PK_GG_ASSET_CARD_0303 | 1 | 1 (0) | 00:00:01 |
| 15 | table access by index rowid | GG_ASSET_VALUE_0900 | 1 | 90 | 3 (0) | 00:00:01 |
| * 16 | index range scan | ID_FAV_CARD_VALIDITY_0303 | 1 | 2 (0) | 00:00:01 |
| 17 | table access by index rowid | AM_ASSET_0900 | 1 | 438 | 2 (0) | 00:00:01 |
| * 18 | index unique scan | PK_AM_ASSET_0900 | 1 | 1 (0) | 00:00:01 |
| 19 | table access by index rowid | AM_ASSET_CLASSIFY | 1 | 76 | 1 (0) | 00:00:01 |
| * 20 | index unique scan | PK_AM_ASSET_CLASSIFY | 1 | 0 (0) | 00:00:01 |
Bytes -------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id ):
---------------------------------------------------
1-filter (ROWNUM = 1)
3-access ("T". "TECH_OBJECT_ID" =: B1 AND "T". "NODE_TYPE" = 2)
10-access ("T". "TRANSACTION_ID" = '000000 ')
12-access ("B". "TRANSACTION_ID" = '000000 ')
14-access ("GG_ASSET_CARD". "CARD_ID" = "B". "CARD_ID ")
16-access ("GG_ASSET_VALUE". "CARD_ID" = "GG_ASSET_CARD". "CARD_ID" AND
"GG_ASSET_VALUE". "VALIDITY_DATE_END" = "GG_ASSET_CARD". "DECREASE_DATE ")
18-access ("AM_ASSET". "DEVICE_ID" = "GG_ASSET_CARD". "DEVICE_ID ")
20-access ("AM_ASSET". "CLASSIFY_ID" = "AAC". "DEVICE_CLASSIFY_ID" (+ ))
Statistics
----------------------------------------------------------
8 recursive cballs
0 db block gets
28 consistent gets
0 physical reads
0 redo size
12384 bytes sent via SQL * Net to client
338 bytes encoded ed via SQL * Net from client
2 SQL * Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
1 rows processed

Very slow in database 2, 27.48 s
SQL> SELECT .*,
2 B. INCREASE_ID,
3 B. TRANSACTION_ID,
4 B. LINK_CARD_ID,
5 B. VALIDATE_FLAG,
6 B. ASSET_VALUE_SHARING,
7 B. RELATED_DEVICE_ID,
8 B. PARENT_CARD_CODE,
9 B. PROJECT_VALUE,
10 B. DELETE_FLAG,
11 B. DEPRECIATION_ADJUST_VALUE,
12 T. TRANSACTION_MODE_CODE,
13 T. TRANSACTION_NO,
14 T. TRANSACTION_FROM,
15 T. FROM_MODEL,
16 (select t. FULL_PATH
17 FROM AM_TECH_OBJECT_NODE_0900 T
18 where t. TECH_OBJECT_ID = A. DEVICE_ID
19 and t. NODE_TYPE = 2
20 and rownum = 1) AS FULL_PATH,
21 AAC. FULL_NAME CLASSIFY_FULL_PATH
22 FROM V_ASSET_CARD_0900,
23 GG_ASSET_INCREASE_ITEM B,
24 GG_ASSET_TRANSACTION T,
25 AM_ASSET_CLASSIFY AAC
26 where a. CARD_ID = B. CARD_ID
27 and B. TRANSACTION_ID = T. TRANSACTION_ID
28 and a. CLASSIFY_ID = AAC. DEVICE_CLASSIFY_ID (+)
29 and B. TRANSACTION_ID = '20140901 ';
Row 200 has been selected.
Used time: 00: 00: 27.48
Execution Plan
----------------------------------------------------------
Plan hash value: 2944357796
Bytes -------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | TempSpc | Cost (% CPU) | Time |
Bytes -------------------------------------------------------------------------------------------------------------------------
| 0 | select statement | 25 | 255K | 45815 (1) | 00:09:10 |
| * 1 | count stopkey |
| 2 | table access by index rowid | AM_TECH_OBJECT_NODE_0900 | 1 | 75 | 4 (0) | 00:00:01 |
| * 3 | index range scan | IDX_TECH_NODE_ID_0900 | 1 | 3 (0) | 00:00:01 |
| 4 | nested loops outer | 25 | 255K | 45815 (1) | 00:09:10 |
| * 5 | hash join | 25 | 253K | 45790 (1) | 00:09:10 |
| 6 | nested loops | 25 | 2750 | 6 (0) | 00:00:01 |
| 7 | table access by index rowid | GG_ASSET_TRANSACTION | 1 | 53 | 2 (0) | 00:00:01 |
| * 8 | index unique scan | PK_GG_ASSET_TRANSACTION | 1 | 1 (0) | 00:00:01 |
| 9 | table access by index rowid | GG_ASSET_INCREASE_ITEM | 25 | 1425 | 4 (0) | 00:00:01 |
| * 10 | index range scan | TRANSACTION_DETAIL_REF_TRANSAC | 25 | 1 (0) | 00:00:01 |
| 11 | VIEW | V_ASSET_CARD_0900 | 280K | 2744M | 45781 (1) | 00:09:10 |
| * 12 | hash join | 280K | 257M | 141M | 45781 (1) | 00:09:10 |
| * 13 | hash join | 274K | 137M | 27M | 12222 (1) | 00:02:27 |
| 14 | table access full | GG_ASSET_VALUE_0900 | 292K | 24M | 910 (2) | 00:00:11 |
| 15 | table access full | GG_ASSET_CARD_0900 | 274K | 114M | 4073 (1) | 00:00:49 |
| 16 | table access full | AM_ASSET_0900 | 756K | 315M | 10464 (1) | 00:02:06 |
| 17 | table access by index rowid | AM_ASSET_CLASSIFY | 1 | 76 | 1 (0) | 00:00:01 |
| * 18 | index unique scan | PK_AM_ASSET_CLASSIFY | 1 | 0 (0) | 00:00:01 |
Bytes -------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id ):
---------------------------------------------------
1-filter (ROWNUM = 1)
3-access ("T". "TECH_OBJECT_ID" =: B1 AND "T". "NODE_TYPE" = 2)
5-access ("A". "CARD_ID" = "B". "CARD_ID ")
8-access ("T". "TRANSACTION_ID" = '000000 ')
10-access ("B". "TRANSACTION_ID" = '000000 ')
12-access ("AM_ASSET". "DEVICE_ID" = "GG_ASSET_CARD". "DEVICE_ID ")
13-access ("GG_ASSET_VALUE". "CARD_ID" = "GG_ASSET_CARD". "CARD_ID" AND
"GG_ASSET_VALUE". "VALIDITY_DATE_END" = "GG_ASSET_CARD". "DECREASE_DATE ")
18-access ("A". "CLASSIFY_ID" = "AAC". "DEVICE_CLASSIFY_ID" (+ ))
Statistics
----------------------------------------------------------
218 recursive cballs
0 db block gets
70112 consistent gets
26412 physical reads
348 redo size
38174 bytes sent via SQL * Net to client
480 bytes encoded ed via SQL * Net from client
15 SQL * Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
200 rows processed

First Diagnosis: At first, I thought it should be the Index Cluster factor. I checked it and it was similar.

The second diagnosis: The test starts because the environment and parameters are different.

Use views v $ SQL and v $ SQL _plan and dbms_xplan.display_cursor to find out the actual SQL Execution Plan.

Select distinct s. SQL _ID, s. HASH_VALUE, s. CHILD_NUMBER, s. SQL _TEXT
From v $ SQL s, v $ SQL _plan p
Where s. SQL _ID = p. SQL _ID
And p. PLAN_HASH_VALUE = '20140901 ';
Select * from table (dbms_xplan.display_cursor (150270666, 0, 'advanced '));

Databases with slow performance:
/* +
BEGIN_OUTLINE_DATA
IGNORE_OPTIM_EMBEDDED_HINTS
OPTIMIZER_FEATURES_ENABLE ('10. 2.0.4 ')
OPT_PARAM ('_ complex_view_merging' 'false ')
ALL_ROWS
............
*/

Databases with high performance:
/* +
BEGIN_OUTLINE_DATA
IGNORE_OPTIM_EMBEDDED_HINTS
OPTIMIZER_FEATURES_ENABLE ('10. 2.0.4 ')
ALL_ROWS
............
*/

The outline of the two executions is a little different, so the implicit parameter _ complex_view_merging is found in the two databases. The database with slow performance is set to false, and the database with fast performance is set to true.

There is a view in the SQL statement. If there is a problem with this implicit parameter, the literal meaning is the combination of complex views. I adjusted this parameter with a try:
Alter session set "_ complex_view_merging" = true;

The result is very fast, 0.6 s.

SQL> SELECT .*,
2 B. INCREASE_ID,
3 B. TRANSACTION_ID,
4 B. LINK_CARD_ID,
5 B. VALIDATE_FLAG,
6 B. ASSET_VALUE_SHARING,
7 B. RELATED_DEVICE_ID,
8 B. PARENT_CARD_CODE,
9 B. PROJECT_VALUE,
10 B. DELETE_FLAG,
11 B. DEPRECIATION_ADJUST_VALUE,
12 T. TRANSACTION_MODE_CODE,
13 T. TRANSACTION_NO,
14 T. TRANSACTION_FROM,
15 T. FROM_MODEL,
16 (select t. FULL_PATH
17 FROM sz_1230.AM_TECH_OBJECT_NODE_0900 T
18 where t. TECH_OBJECT_ID = A. DEVICE_ID
19 and t. NODE_TYPE = 2
20 and rownum = 1) AS FULL_PATH,
21 AAC. FULL_NAME CLASSIFY_FULL_PATH
22 FROM sz_1230.V_ASSET_CARD_0900,
23 sz_1230.FM_ASSET_INCREASE_ITEM B,
24 sz_1230.FM_ASSET_TRANSACTION T,
25 sz_1230.AM_ASSET_CLASSIFY AAC
26 where a. CARD_ID = B. CARD_ID
27 and B. TRANSACTION_ID = T. TRANSACTION_ID
28 and a. CLASSIFY_ID = AAC. DEVICE_CLASSIFY_ID (+)
29 and B. TRANSACTION_ID = '20140901 ';
Row 200 has been selected.
Used time: 00: 00: 00.06
Execution Plan
----------------------------------------------------------
Plan hash value: 801438153
Bytes -------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (% CPU) | Time |
Bytes -------------------------------------------------------------------------------------------------------------------
| 0 | select statement | 25 | 28475 | 206 (0) | 00:00:03 |
| * 1 | count stopkey |
| 2 | table access by index rowid | AM_TECH_OBJECT_NODE_0900 | 1 | 75 | 4 (0) | 00:00:01 |
| * 3 | index range scan | IDX_TECH_NODE_ID_0900 | 1 | 3 (0) | 00:00:01 |
| 4 | nested loops | 25 | 28475 | 206 (0) | 00:00:03 |
| 5 | nested loops outer | 25 | 26275 | 131 (0) | 00:00:02 |
| 6 | nested loops | 25 | 24375 | 106 (0) | 00:00:02 |
| 7 | nested loops | 25 | 13425 | 56 (0) | 00:00:01 |
| 8 | nested loops | 25 | 2475 | 6 (0) | 00:00:01 |
| 9 | table access by index rowid | FM_ASSET_TRANSACTION | 1 | 42 | 2 (0) | 00:00:01 |
| * 10 | index unique scan | PK_FM_ASSET_TRANSACTION | 1 | 1 (0) | 00:00:01 |
| 11 | table access by index rowid | FM_ASSET_INCREASE_ITEM | 25 | 1425 | 4 (0) | 00:00:01 |
| * 12 | index range scan | TRANSACTION_DETAIL_REF_TRANSAC | 25 | 1 (0) | 00:00:01 |
| 13 | table access by index rowid | FM_ASSET_CARD_0900 | 1 | 438 | 2 (0) | 00:00:01 |
| * 14 | index unique scan | PK_FM_ASSET_CARD_0303 | 1 | 1 (0) | 00:00:01 |
| 15 | table access by index rowid | AM_ASSET_0900 | 1 | 438 | 2 (0) | 00:00:01 |
| * 16 | index unique scan | PK_AM_ASSET_0900 | 1 | 1 (0) | 00:00:01 |
| 17 | table access by index rowid | AM_ASSET_CLASSIFY | 1 | 76 | 1 (0) | 00:00:01 |
| * 18 | index unique scan | PK_AM_ASSET_CLASSIFY | 1 | 0 (0) | 00:00:01 |
| 19 | table access by index rowid | FM_ASSET_VALUE_0900 | 1 | 88 | 3 (0) | 00:00:01 |
| * 20 | index range scan | ID_FAV_CARD_VALIDITY_0303 | 1 | 2 (0) | 00:00:01 |
Bytes -------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id ):
---------------------------------------------------
1-filter (ROWNUM = 1)
3-access ("T". "TECH_OBJECT_ID" =: B1 AND "T". "NODE_TYPE" = 2)
10-access ("T". "TRANSACTION_ID" = '000000 ')
12-access ("B". "TRANSACTION_ID" = '000000 ')
14-access ("FM_ASSET_CARD". "CARD_ID" = "B". "CARD_ID ")
16-access ("AM_ASSET". "DEVICE_ID" = "FM_ASSET_CARD". "DEVICE_ID ")
18-access ("AM_ASSET". "CLASSIFY_ID" = "AAC". "DEVICE_CLASSIFY_ID" (+ ))
20-access ("FM_ASSET_VALUE". "CARD_ID" = "FM_ASSET_CARD". "CARD_ID" AND
"FM_ASSET_VALUE". "VALIDITY_DATE_END" = "FM_ASSET_CARD". "DECREASE_DATE ")
Statistics
----------------------------------------------------------
0 recursive cballs
0 db block gets
3029 consistent gets
0 physical reads
0 redo size
38039 bytes sent via SQL * Net to client
480 bytes encoded ed via SQL * Net from client
15 SQL * Net roundtrips to/from client
0 sorts (memory)
0 sorts (disk)
200 rows processed

Therefore, set _ complex_view_merging to true globally, and alter system set "_ complex_view_merging" = true scope = both;

Again, the other 10 Gb oracle Database settings are true, which should be a problem of database installation.