Design and implementation of data mining and data migration system under Hadoop architecture

Design and implementation of data mining and data migration system under Hadoop architecture

Lu Ming of Shanghai Jiaotong University

Usually the enterprise's information system will contain multiple business systems, each business system contains its own set of online business system, backup system and filing system, the system management is complex, easy to cause the waste of storage space, and the system scalability is poor. In view of the above shortcoming, this paper designs and realizes a tiered storage system, uses a large data platform to carry on the unified management to the multiple business system data, unifies each business system backup system and the filing system. This tiered storage system provides a scenario for data mining and data migration systems based on the Hadoop architecture. The specific contents are as follows: (1) The key technologies of Hadoop, such as MapReduce distributed architecture, HBase database, HDFs Distributed File system, are analyzed. (2) A layered storage system based on Hadoop architecture is designed and implemented, and the architecture design of the system and large data platform is described in detail. (3) Design and implement the data mining module based on MapReduce mode, apply the traditional relational database analysis method to the HBase database, and effectively excavate and classify the HBase database data. (4) The Data Migration module is designed and implemented, which realizes the function of migrating the structured data and unstructured data of the online business platform to the large data platform. The process of structured data migration uses MapReduce data migration to design a scheduling algorithm based on Io load, which takes resource usage into account and avoids assigning tasks to IO load-heavy nodes. The design realizes the unstructured data migration tool, skillfully uses the FTP, the more concurrent migration data and so on on-line business platform log file to the HDFs designated directory. (5) The function and performance of the system are tested completely. The test results show that all the functional modules of the system meet the design requirements, and the IO Scheduler presented in this paper has better migration performance than the default scheduler. The data Mining and data migration system designed by the thesis satisfies the specific requirements, has good concurrent data migration performance, and realizes data classification mining under large data scale.

Design and implementation of data mining and data migration system under Hadoop architecture