[Turn] on the idea and realization method of using SQLite to store discrete tiles

Article copyright by the author Xiaohui Li is and the blog park, if reproduced please clearly indicate the source: http://www.cnblogs.com/naaoveGIS/

1. Background

In a number of projects involving the Internet map of the intranet display, through the self-made tools to complete the Internet map tile download. However, there are several problems with this approach:

A. Tiles are discrete images, and remote deployment is time-consuming.

B. In the tile download, it involves downloading the internet tiles to memory, then creating the corresponding folders, and then saving to the local process, the efficiency is not high.

In addition to the above two problems, there is a lot of storage consumption and so on shortcomings. Do you have a bundle-type tile organization format similar to ArcGIS to solve existing problems with storage footprint, remote deployment, and more?

2. Custom bundle format 2.1 Advantages of using SQLite to store 2.1.1Sqlite

A. lightweight

Unlike the database software for SQLite and C/s mode, it is a database engine within the process, so there is no client and server for the database. With SQLite, you can use only one of its dynamic libraries to enjoy its full functionality.

This makes it very easy for us to look at it as a document.

B. Single File

The so-called "single file" is that all the information in the database (such as tables, views, triggers, etc.) is contained within a single file. And this file can be copied to other directories or other machines, also can be used.

With this single file, we can manage the files that need to be stored discretely.

c. Cross-platform / portability

In addition to the main operating system, SQLite also supports many unpopular operating systems, such as Android, Windows Mobile, Symbin, Palm, VxWorks and other support.

One-time storage, multi-point use.

d. Memory Database ( in-memory Database )
Now that memory is getting cheaper, SQLite's memory database features are becoming more usable. The SQLite API does not differentiate whether the current operating database is in memory or in a file (for storage media is transparent). So if you think that disk I/O can be a bottleneck, consider switching to memory mode. When switching, the operation of the code of SQLite basically do not change, as long as the beginning of the file load to memory, the end of the memory of the database dump back to the file.

When storing tiles, if the tile data is not too much, memory is enough, can switch to memory mode, further improve tile reading efficiency.

2.1.2 Disadvantages

A. locking mechanism for concurrent access

SQLite's performance in concurrent (including multi-process and multi-threaded) reading and writing has been suboptimal. The database may be exclusive to write operations, causing other read and write operations to be blocked or faulted.

B. SQL Standard support not complete
On its official website, there are specific examples of what SQL92 standards are not supported.

2.2 Tiles are organized in the Mbtiles specification

Mbtiles is a map tile storage data specification, can greatly improve the reading speed of massive map tiles, than through the tile file way read much faster, for Android, iphone and other smartphones offline map storage.

Mbtiles through the view, redundant tile data can be reused, thus reducing the space occupied by tiles, rather than a single, text table. For example: The map covers a large area of pure blue like sea or empty land, resulting in thousands of duplicate, redundant tile data, for example, 4/2/8 tiles in the middle of the Pacific, may appear to be a blue picture although it may be some in the 3rd level, but at level 16 may exist millions of blue pictures, They are all exactly the same.

The Mbtiles general approach is to divide the tile table into two pieces: one for storing the original image and one for storing the tile coordinates for those pictures. The specific design is as follows:

A. Design Map Table

Store the tile row number and the corresponding tile ID.

B. Design Images Table

Storage of tiles.

c. Design View Tiles

Based on map and images generation.

3. Development of tile download and packaging storage Tools 3.1 Tile Download tool

Tile download tool based on the development of the tile addressing algorithm, for different internet maps, tiles, such as the row number algorithm is slightly different. Especially for Baidu map, its tile algorithm will be based on Baidu's tile grading shift rule conversion. There is no exhaustion here.

3.2 Storage based on the Mbtiles specification

The design idea is:

A. Multi-threaded tile download, in memory open container pool.

B. When the memory container pool is full, go to SQLite for the whole storage. Lock in storage, avoid sqlite to multi-transaction support is not ideal problem.

4. Retrofit back end for testing 4.1 retrofit Backend

The backend follows the idea of the connection pool, which supports reading the tiles in SQLite through the row and column numbers.

4.2 Front-end testing

5. Summary of advantages of the program

A. Improve tile download storage speed. Tested, the efficiency of the local image storage is increased by at least one times. Mainly because, one is binary storage, no conversion process. The second is to reduce the level of folder creation time.

B. Reduce storage space. The Mbtiles specification can reduce redundant tiles.

C. facilitates data transfer. All tiles are stored in a single folder for easy data transfer deployment.

D. Improve the reading speed of the massive map tiles, much faster than through the tile file way reading. The reason is that a single file no longer involves directory addressing, and you do not need to read tiles for binary re-egress.

6. Pending test

Because SQLite itself is not very good for multi-transaction support, large concurrent access to tile scenarios need to be tested. Next, you are ready to test using JMeter.

7. Extended Use

At present, the vector cutting tool, the storage is also a discrete type of PBF. Storage with Mbtiles can also be used for easy deployment purposes. Also, a layer uses a view that is more similar to the layers in ArcGIS.

[Turn] on the idea and realization of using SQLite to store discrete tiles