Actual Combat | Elasticsearch build Knowledge Base Retrieval System

Preface

Discussion issues from the "Dead Knock Elasticsearch" technology Group:

--I want to use ES to do something similar to the Knowledge base, so I need to index some PDF, Word and other files, this you have tried before. Can you give me a direction?

My thoughts are as follows: 1, PDF, Office class documents how to be indexed by ES.

More specifically, PDF, Office class documents (Word,ppt,excel, etc.) are imported into ES.
As shown in the figure:

The issue is shifted to: How to Index Office class documents, PDF documents into ES, and provide full-text indexing services. 2. Elasticsearch The maximum length of the field to be retrieved is supported.

ES5. After the x version, keyword supports a maximum length of 32,766 UTF-8 characters, and the text has no limit on the length of the characters.
After setting the Ignore_above, the data beyond the given length is not indexed and the results cannot be retrieved by term exact match.

Reference: HTTP://T.CN/RYLEMGN

Reference 6.0 official website interpretation: Http://t.cn/RYlELoV

Reference LUNCENE7.1API:HTTP://T.CN/RYWVUGL

3, Office&pdf document deposit Elastisearch attention to issues list

Less nonsense, directly on the map.
4, the analysis of the actual code

Only the core sample code is listed above.

5. Summary
From a functional and performance perspective, the recommendations for building a knowledge base are as follows:
1) The core of the knowledge Base is data import es, the core of the import ES is the parsing of various types of documents;
2) Set the mapping in advance, define the field word segmentation, non-segmentation strategy;
3) for storage greater than 1MB a field, it is recommended to use FVH highlighting method, set in mapping. Reference:

[1] Java Read Office document reference: Http://t.cn/RZsiHcY
[2] HTML2MD reference: Http://t.cn/RYlEEoC
[3] pdf2html reference: Http://t.cn/zWgVuFv
[4] OpenOffice Reference: Http://t.cn/8sGyKZd