Overview of Big Data Processing

When processing large-scale data, such as full-text search, approximate document search, and data mining, RDBMS often cannot meet our needs. So what should we do? One solution is to use batch processing to retrieve data from RDBMS, create an Index Server, and then make the web application query through RPC (Remote Procedure Call) or web API (HTTP + JSON.

The above method is called "Special Purpose Index ". After creating a special index, you can handle the hard-to-handle tasks that RDBMS can do. RDBMS is a general system that can sort data, process statistics, and join data for a variety of purposes. If it is only for a specific purpose, it is dedicated to optimizing the data structure, it is much faster than RDBMS. The inverted rank index used for search is the most typical example. Before the index, you can perform natural language processing, without having to query all the data in RDBMS.

Special Purpose Index:

Regularly retrieve data

Create a Data Structure Based on the Retrieved Data

Reverse index used for search

Trie used for keyword links

Use C ++ to develop a server that stores structured data and access it through RPC or Web APIs.

RDBMS-> Information Search

• Obtain data in a relational database using batch processing

• Create inverted indexes and use search algorithms