how data is written1. Update-in-place in-situ update2. Append-only btree/copy on Write tree sequence file end Append data is placed in a specific way to improve read performance, but write performance degrades, and random reads and writes are required for B + tree and hash updates:1. Two points to find, the file data stored in an orderly manner, using binary search to complete the search for the specified key2. Hashing, splitting the data into different buckets with a hash3. B + Tree to reduce the read of external files4. External file, save data as log, and create a hash or find tree map corresponding file
Storage structure (disk because of seek, etc., sequential reading than random read block n magnitude):
the whole disk is looking at something. A log that stores persistent data and its index in the log, each time it is added to the end of the log;by converting a lot of small files to a continuous mass transfer, the majority of access to the file system is sequential, which improves disk bandwidth utilization and fast failure recovery. simply divided into a part of the resident memory, can be any convenient key value lookup data structure, another resident hard disk, similar to B-tree, this part of the frequently accessed nodes will also be cached in memoryThe log file is first written to the insert operation log. Then write to the memory section. When the memory close threshold is rolled into the hard disk. adding data to a file, because it is in full order, is good for write operations, but reading some data from a log file will consume more time than the write operation, requiring a reverse scan and knowing what is needed. scenarios where logs are applicable:
- Data is accessed by the whole, WAL (Write-ahead-log)
- Know the definite Offset,kafka
log-structured Merge-tree,lsm-treetransforms a large lookup structure that was previously used to save the write sequence to a few similar ordered files (sstable). Each file contains some changes in a short period of time, because the files are ordered and subsequent lookups are quick. The file is not modifiable, never updated, the new operation will only be written to the new file, read and write check all files, through periodic merging to reduce the number of files. Keep the log file write performance, let the operation sequence, continue to append, not modify, delay the update, bulk write to the hard disk, suitable for a large number of insertion environmentwrite operations are processed in batches, written only to sequential blocks, and periodic merges affect IO, and operations can potentially access a large number of files (scattered reads)
- Update operation-"Memory Cache" (memtable) uses tree structure to keep key in order-"Wal write disk anti-loss/recovery/-" to a certain size brush to disk in a new file, here simple to generate a new file is not edited, so it is sequential write, fast
The more data to the storage system, there will be more non-modifiable order sstable files are created, they represent small, chronologically modified, system periodically initiated compaction, merging files to remove redundant redundancy, reduce the number of files, to ensure the performance of operations, Because sstable is an orderly structure, merging is very efficient
- Read operation-"first check memory data (memtable)-" No this key-"reverse check sstable until found.
because of the need to traverse all sstable, when the number of excessive performance will be reduced, on the one hand the system periodically merge sstable, with the cache technology, on the other hand, using bloom to avoid a large number of read file operations. cycle Merge (by layer/file size): To maintain and reduce the number of sstable files in order to ensure the LSM read speed
Why NoSQL fast--disk sequence write
