original 2016-07-11 Yang Shanggang Docker
Flash existence in the past few years the storage domain has developed very fast, the application is also more and more widespread, how to use the flash memory better, this share tells some flash related optimization and the application.
Flash application Scenarios
Database
Nosql
Distributed storage
Cdn
Public cloud storage
Combined with the above scenarios, flash memory is primarily suitable for scenarios with high random IO requirements and bandwidth requirements. Scene selection, but also to play the advantages of Flash. At present, the above business in the next few years will be more rapid development in the public cloud storage this part. is a manufacturer of cloud disk comparison, you can see that the price of flash memory is very close to the mechanical hard disk, and single from the cost of each IOPS look, cost-effective will be higher.
Flash Overview
Solid-state drives, but can be understood from a broad sense, starting from 2010 in the Internet industry, large-scale application, performance and stability has been large-scale cluster online verification, application scenarios are very extensive. Of course, the IOPS of flash memory is several orders of magnitude higher than traditional mechanical drives, but the more core is the lower latency, the more advantageous.
It is good to note that the Flash has an elevation of access delay.
Mentioned Flash, had to mention the very basic component NAND in flash memory. NAND classification is now also very much.
Test
Why do we have to do the test?
Learn about Products
Know Yourself
Optimize yourself
Optimize cost Models
So it is a very important problem to be able to test more efficiently in the face of so many vendors and products. While everyone is now turning to cloud services, there is not much direct access to hardware, but cloud-vendor testing remains an important part.
is the test very low?
The test is simple?
No technical content?
Is the test boring?
is the storage technology stack we need to know.
Test criteria:
Clear goals
Efficient
Completeness of
Can be quantified
can be compared
Output
Test process:
Test tools:
IO level: Fio, Sysbench, Iometer, DD, etc.
Oltp:sysbench,tpc-c
Auxiliary tools: Tcpcopy, Tcprstat, Pt-log-player
Sysbench:
Open-source multithreaded performance testing tools
Support for tests such as CPU IO Mutex OLTP
Test cases can be customized with LUA scripting
Common Insert Select and OLTP three scenarios
Test Pain points:
A lot of repetitive work
Standards are not uniform
Very long test cycles
High labor costs
Exception handling during testing
Test data processing and test reports
To solve the pain point is the first normalization, mainly the following aspects:
Standardized test objectives
Standardized testing tools
Standardized testing processes
Standardized test reports
Automated Test Flow:
Automated Test framework
Based on Python
Contains the overall standard test process
Coverage of mainstream test tools
Processing and generating reports at the data
Custom test Plans
is the test flowchart
The benefits of automation are also obvious:
A few things to keep in mind when testing flash:
Some problems with MySQL testing:
Test data set size, at least over billion
and the memory buffer ratio, to see the performance under the small cache
Physical Reading
Complexity of transactions
Multi-table concurrency
Some points of attention at the system level:
Test optimization combined
InnoDB Compression test:
InnoDB built-in compression
Based on Zlib library
The theory can reach about 50% compression ratios
But there's a loss of performance
CPU Time Swap storage space
Benefits for SSD Life
How to use it well?
Based on our previous testing process, we can get the conclusion that the InnoDB compression ratio is about 50%, the write performance loss is large, the loss ratio is about 70%. Based on this conclusion, we can choose whether to use InnoDB compression for our online business.
TOKUDB:
A storage engine for MySQL that supports transaction ACID features
Support for multiple versioning (MVCC)
Based on fractal Tree Index, ideal for writing dense scenes
High compression ratio
Native support for online DDL
Mainstream branches are supported
Fee to open source
This is our test result, we can see tokudb better compression ratio and better write stability, of course, the cost is higher CPU consumption.
Summarize
Now is no longer the era of performance for Kings
It's more important to really understand your needs.
Discover flash performance, hardware and software integration
Expand Flash Application Space
To do something truly worthwhile
How to do a better combination of hardware and software (in fact, now the hardware is ahead of some software)
At the end of the picture, don't just live in the present, brave to accept the new technology, the courage to try and wrong, of course, the cost of trial and error and the benefits to be assessed and controllable. In fact, a lot of technical understanding, and may not be said to be "evil."
Recommended activities
Flash optimization testing and application in the Internet scenario