Research on resource discovery and collection method of OA periodicals based on Hadoop

Research on resource discovery and collection method of OA periodicals based on Hadoop

Du Baorie of Yanshan University

A large number of OA periodical paper resources on the internet belong to the Deep Web (deepweb) resources, the traditional search engine can not set up the index effectively, and it is difficult for the users to get the expected OA periodical paper resources in the retrieval. An effective way to solve this problem is to realize the integrated integration of OA periodical paper Resources on the Internet, and to provide users with a unified and transparent search service interface, while the discovery and collection of OA journal paper Resources is an important link. Distributed storage and parallel mechanism can greatly improve the efficiency of the large amount of OA periodical resource processing. The Distributed File System (HDFS) and the parallel mechanism (MapReduce) based on Hadoop are a feasible platform to deal with the mass information, and this paper realizes the discovery and collection of the OA periodical paper resources based on Hadoop. This article mainly did the following research. Firstly, aiming at the problem that the traditional search engine can not effectively index the resources of OA periodicals, this paper designs a collection system of OA periodicals ' resources based on Hadoop. Firstly, the overall frame, overall module and overall flow of the acquisition system are designed, and then the function and workflow of each main module are introduced in detail. Secondly, through the research on the access mode of the paper resources in the periodical site, this paper proposes a method for discovering the resources of OA periodical site. This method first constructs the C4.5 decision tree by extracting the homepage feature of the OA periodical site, then uses the decision tree to classify the periodical site; Finally, aiming at these two kinds of OA periodical sites, we propose a new paper resource discovery algorithm, and save the thesis resource information to the paper information resource database file. Thirdly, aiming at the requirement of building OA periodical Knowledge Meta Data Warehouse, this paper proposes a method of thesis collection oriented to OA periodical site. Firstly, the paper obtains the downloading information and downloading parameters of the paper by analyzing the Document information resource database, and then downloads the paper files by using HTTP protocol, and finally, according to the threshold value, the paper is packaged into sequence large files and uploaded to HDFs. Finally, it realizes the prototype system of OA journal paper Resource acquisition based on Hadoop, and verifies the experiment by the prototype system.

Research on resource discovery and collection method of OA periodicals based on Hadoop