Optimize hundreds of millions of data queries in 10 minutes

Source: Internet
Author: User

A user developed QQ to contact me a few days ago, as shown below:

Free dolphins. 16:12:01

Island Lord, I cannot find the result of one of my SQL statements. Can you check it for me?

Orchid island master 16:12:10

How long will it take?

Free dolphin 16:12:17

No results are returned for a long time.

Orchid island master 16:12:26

Haha, good.

Orchid island master 16:12:39

Send SQL statements and execution plans.

Free dolphin 16:12:55

Select N. C1, N. C2, N. C3, N. C4, N. C5
From (select count (T. C1), T. C1, T. C2, T. C3, T. C4, T. C5
From tab1 t
Where T. C2 not in ('val1', 'val2', 'val3', 'val4', 'val5 ')
Group by T. C1, T. C2, T. C3, T. C4, T. C5) N
Where not exists
(Select * from (
Select count (S. C2), S. C1, S. C2
From (select M. C1, M. C2, M. C3, M. C4, M. C5
From tab1 m
Where exists (select C1
From tab2 n
Where c2> sysdate-14
And M. C1 = n. C1)
And M. C1 is not null
And M. C2 not in ('val1', 'val2', 'val3', 'val4', 'val5') S
Group by S. C1, S. C2) T1 where t1.c2 = n. C2)
And N. C1 is not null;

Orchid island master 16:13:12

Are these two tables big?

Free dolphin 16:13:16

Tab1 is small, with more than tabps and tens of millions of data in two weeks.

Lanhua island master 16:13:22

OK.

Orchid island master 16:16:29

So change the SQL statement:

With T1 (
Select count (T. C1), T. C1, T. C2, T. C3, T. C4, T. C5
From tab1 t
Where T. C2 not in ('val1', 'val2', 'val3', 'val4', 'val5 ')
And C1 is not null
Group by T. C1, T. C2, T. C3, T. C4, T. C5)
Select t1.c1, t1.c2, t1.c3, t1.c4, t1.c5
From T1
Where not exists (
Select/* + use_hash (m, n) */M. C1, M. C2, M. C3, M. C4, M. C5
From T1 M, tab2 n
Where n. c2> sysdate-14
And M. C1 = n. C1
And t1.c2 = M. C2 );

Orchid island master 16:16:43

Remove the execution plan.

Free dolphin 16:16:57

Okay.

Free dolphin 16:17:25


Orchid island master 16:17:57

Okay. Try it.

Free dolphin 16:19:28

The result is 37 s.

Orchid island master 17:20:21

Well, good.

Orchid island master 17:20:34

Can this happen?

Free dolphin 17:20:47

Yes.

Orchid island master 17:21:11

Well, well, let's do it first and stop calling it.

Free dolphin 17:21:30

Thank you, island master.

Orchid island master 17:21:53

You're welcome. You have to contact me.

Free dolphin 17:22:18

Well, you are busy...

At this point, the optimization of the user's SQL has ended. In fact, there should be room for optimization of this statement, but the user can say it is okay, because the optimization is endless and, further Optimization may require further information and sometimes requires greater changes. In view of various factors, we have processed the statements and plans and recorded them here!











Optimize hundreds of millions of data queries in 10 minutes

Related Article

E-Commerce Solutions

Leverage the same tools powering the Alibaba Ecosystem

Learn more >

Apsara Conference 2019

The Rise of Data Intelligence, September 25th - 27th, Hangzhou, China

Learn more >

Alibaba Cloud Free Trial

Learn and experience the power of Alibaba Cloud with a free trial worth $300-1200 USD

Learn more >

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.