Atitit. Improve backup file copy speed (1)-----Analyze statistical problems and solutions

Atitit. Improve backup file copy speed (1)-----Analyze statistical problems and solutions

1. The present situation 1

2. Hard drive information approx. 50mb/s, 50iops 1

3. Statistical small file ratios 2

4. Replication Speed Estimation: 2

5. Small file type ratios by extension 2

6. Reasons for Slow::: ioPS too low ... 4

7. Solution::: 4

7.1. Upgrade to SSD Drive. 4

7.2. Clean up Garbage 4

7.3. Clean up Duplicate Files 5

7.4. Move the Pic/isho file: 5

7.5. Packaging Files: . 5

8. Post-cleanup Scenario 5

9. Hard drive detailed Speed test report 80g+160g 5

SELECT * from ' foldercount ' where file isn't in (select file from Foldercountclraftr)

1. The present situation

10.6g, 40,370 files. 2,894 folders ..... Two HDD intermediate copy uses 19min

Replication is good for a long time. From PC to mobile hard drive., hard drives are usually hard drives. USB cable is 2.0, can support 40mb/s, so, speed problem is not in this place ...

Should be insufficient iops ...

2. Hard disk information about 50mb/s, 50iops

To calculate the slowest drive ...

Author:: Old Wow's paw Attilax Ayron, email:1466519819@qq.com

Reprint please indicate source: Http://blog.csdn.net/attilax

3. Statistics of small file ratios

Walk Ten horses A tool, and had to write from the blue ...

Traverse folder, no file name, size, Java 17s go to generate SQL LAN ... Put a database black ... In statistics ...

Range::: 0-----500KB, 500kb--1.5m, 1.5 M ...

1M below according to Radom RW ..... 1Mabover in sequence io rw

Select COUNT (*) from Foldercount where size<500000 >>>38320

Select COUNT (*) from Foldercount where size<1500000 and size>500000 >> 694

-----Large File:

Select COUNT (*) from Foldercount where size>1500000 >>687

Select sum (size) from Foldercount where size>1500000 9605MB

Other EHR

Select COUNT (*) from Foldercount where size<20000 >>>25530 files.

4. Replication Speed Estimation:

Lit file less than 500kb copy 38320/50iops=13min

Lit File---1.5MB::: 694/30 iops=23s

Big file Copy 10g/50mps=3.2min

5. Small file type ratios by extension

Select COUNT (*) from Foldercount where size<20000 >>>25530 files.

Select COUNT (*) Sum,extname from Foldercount where size<20000 GROUP by Extname have sum>5 order by sum DESC

5732 gif OK

3150 jpg OK

2506 htm OK

2426 JS OK

1991 txt OK

1067 CS OK

1015 CSS OK

1010

888 PNG OK

589 URL OK

484 vcf OK.

PHP OK

342 aspx OK

Class OK

html

225 Java

184 old

153 SWF

131 lnk

A. resx

111 Extra

104 dat

103 doc

rar

About XLS

Bayi DLL

xml

Resources

ps

jsp

+ Bat

Ico

sql

Wu She

Ssk

53 9]

Ashx

50 7]

Jpeg

Scc

TLog

Cache

Ahk

+ Bak

asp

Csproj

Log

Config

Docx

csv

Json

27 5]

Refresh

Zip

Bmp

25 2]

25 10]

24 1]

% exe

Reg

S

Map

Jar

SLn

User

db

Thmx

Data

xsd

tab

H

vbs

Ini

Settings

Ten ascx

Ten TPL

Ten CFG

9 Manifest

9 suo 6. Slow Reason::: ioPS too low ... 7. Solution: :: 7.1. Upgrade to SSD drive.

SSD ioPS are often more than 20 times times the average HDD ... 7.2. Cleaning up rubbish

Junk csv,, SVN Trash folder ... Big size ... Delete...

Rules for junk pictures

Gif

Jpg::width/height is too small. Size is too small 180*180 below ... pc pictures (ma yuan data)

7.3. clean up duplicate files 7.4. Move the Pic/isho file. 7.5. packaging files ...

You can use the WinRAR tag to the per folder.

8. The situation after cleaning up

Ori 10.6g, 40,370 files. 2,894 of folders

and a 10.1g ... File 14378, folder 1459 9. Hard drive detailed Speed test report 80g+160g

HD Tune pro:st380815a File Benchmark 80g

Drive C:

Transfer Rate Test

File Size: MB

Sequential Read 55074kb/sec

Write Sequential 50811kb/sec

Random Read 137 IOPS

Random Write IOPS

Random Read (Queue depth = +) 154 IOPS

Random Write (Queue depth = +) 258 IOPS

HD Tune PRO:WDC wd1600aajs-00b8d File Benchmark 160g

Drive U:

Transfer Rate Test

File Size: MB

Sequential Read 99937kb/sec

Write Sequential 91575kb/sec

Random Read 104 IOPS

Random Write 340 IOPS

Random Read (Queue depth = +) IOPS

Random Write (Queue depth = +) 323 IOPS

HD Tune pro:st380815a Random Access

Test Capacity:full

Read Test

Transfer size operations/sec avg. access time Max. Access time avg. speed

512 bytes

4 KB 16.740 IOPS 72.026 Ms 0.233 MB/s

KB, IOPS 17.410 ms 42.229 ms 3.590 MB/s

1 MB IOPS 33.651 ms 60.259 ms 29.716 MB/s

Random $ IOPS 27.175 ms 89.132 ms 18.857 MB/s

HD Tune PRO:WDC wd1600aajs-00b8d Random Access

Test Capacity:full

Read Test

Transfer size operations/sec avg. access time Max. Access time avg. speed

512 bytes

4 KB, IOPS 17.303 ms 653.923 ms 0.226 MB/s

KB 18.937 Ms 297.883 ms 3.300 MB/s

1 MB 29.347 IOPS 308.293 Ms 34.075 MB/s

Random, IOPS 23.852 ms 308.929 ms 21.484 MB/s