Need to import from the file once a day, now think of using MySQL table storage, do not know there is no better way.
Reply content:
For this small problem, if the key is small, you can put all key-pos memory, POS is the offset of value in the file, each access only need one disk seek.
Use PROTOCOL-BUF or thrift to network services, you can add a layer of proxy, by key hash distributed to different servers to reduce the pressure.
Well, like these, beansdb are already done. There is no strong correlation between data, in which case it is more appropriate to use Mongo+redis, where hotspot data is kept in Redis, and other data md5 dispersed to several mongodb. Then the request is scattered over different mongodb.
Extension is also convenient 1. The total metadata amount is less than 100G, plus the primary key index, the expected 100g-120g
2. Need to import from the file once a day, now think of using MySQL table storage, do not know if there is a better way? It's not clear that there are 100 million records in the present, or maybe 100 million records after each day.
This is very important, if each day accumulated together is 100 million records, and is based on the PK query query, there is no need to divide the table
3. It is critical that only the query or other operations, such as Update,insert, must be confirmed clearly
4. Whether the query is in the form of PK single, or a large number of readings, the key points need group by statistics, ORDER by paging
5. Whether the primary key is simple such as 1 or 2 integer field mode
6. Using MySQL, if very simple query, and according to PK, you can consider the use of Handlersocket mode, for complex recommendations to go SQL protocol ...
Recommended Handlersocket technical article information:
Handlersocket principles and other series of chapters
http://www. MYSQLOPS.COM/2011/10/19 /handlersocket-principle.html
1. Mongodb+redis
Cool, and for your scenario, no risk (even if you lose 1 days of data, you can re-import), mature and reliable.
MongoDB stores data (must use master or replication set), Redis does the cache.
2. mysql+memcached.
There is no doubt that you can achieve your needs. Pour after read only? Use Java to write a random one. Index is placed in memory and the data is placed on the hard disk. If the access is redundant, add a cache. Use Redis+mysql Bar, MySQL as a tape, Redis to do the front desk query, more appropriate. Stable, fast. Redis is recommended, it caches k-v data in memory, high performance, stable technology, support master-Slave synchronization http://www. thuir.org/thuirdb/
What you've just seen on Weibo is very much in line with your needs.
Of course, the use of immature technology is not recommended. Mark MySQL is used to store data, Redis queries, pure memory operation, the speed is absolutely fast!
The disadvantage is that the hardware input is large, 100 million data can be stored in the Redis server's memory, Redis is not used in the function, all do not use.
