Real-time traffic flow data storage system based on HBase

Real-time traffic flow data storage system based on HBase

Lu Ting Fang Jun Choyank

Traffic flow data has many characteristics, such as multiple sources, high speed and large volume, and traditional data storage methods and systems expose the problem of weak expansibility and low real-time storage. Aiming at the above problems, a real-time storage system based on HBase traffic flow data is designed and implemented. The system adopts distributed storage architecture, The data are normalized by preprocessing of the front-end, the different types of stream data are divided by the multi source buffer structure, and the data are stored in parallel to HBase Cluster Server by means of the consistent hashing algorithm, multithreading technology and the optimization design of the key line. The experimental results show that: Compared with the real-time storage system based on Oracle, the storage performance of the system improves 3~5 times, and its storage performance is 2~3 times compared with the native HBase method, and it has good extended performance.

Real-time traffic flow data storage system based on HBase