- Solutions for massive data
-
- Caching and Page statics
- The cache can be stored directly in memory using map (CONCURRENTHASHMAP), or using the cache framework Ehcache,memcache,redis. The most important thing about caching is the creation time and invalidation mechanism of the cache. The cache should define a type with null values, prevent the discovery of a database lookup value frequently after an empty cache, and cache data that is not frequently required for real-time changes
- Page static, save the last page generated by the program, you can use the template technology freemaker,velocity to generate static pages, you can also use the cache server squid and nginx and so on.
- Database optimization
- Table Structure Optimization
- SQL statement Optimization
- Partitioning: The data in a table is stored in a different area according to certain rules, and when querying data, if the data is in the same area, then you can operate on only one area of data, so that the amount of data is less and faster.
- Sub-table
- Index optimization
- Using stored procedures instead of direct operations
- Separate active data (inactive data into another data table, query the active table first to query the data can not find the table of inactive data)
- Bulk Read and deferred modification (improves efficiency by reducing the number of operations)
- Bulk reading is the merging of multiple queries into a single time. Can be a batch read of data in the same request or it can be multiple requests that are merged into a single query.
- Deferred modification, for high concurrency and frequently modified and new data, you can temporarily save the data that needs to be modified in the cache, and then periodically save the data in the cache in the database, the program can read the data in the database and the data in the cache.
- Read/write separation
Massive data and high concurrency solutions