How to parse PDF instances using mongominer in Python

This article mainly introduces the example of using mongominer to parse PDF code in Python. I think it is quite good. I will share it with you and give you a reference. Let's take a look at the small Editor. This article mainly introduces Python's example of using mongominer to parse PDF code. The small editor thinks it is quite good. now I will share it with you and give you a reference. Let's take a look at it with Xiaobian.

In the near future, crawlers sometimes encounter the situation where the website only provides pdf, so that scrapy cannot be used to directly crawl the page content, and it can only be processed by parsing PDF, currently, only pyPDF and mongominer are available. Because it is said that mongominer is more suitable for text parsing, and I need to parse the text, so I finally chose to use mongominer (which means I have no idea about pyPDF ).

The first thing to note is that PDF parsing is a tough task, even if the simplified miner is not well-formed, the PDF parsing effect is not very good, so even the developers of the simplified miner are talking about PDF is edevil. but these are not important.

1. installation:

1. first download the source file package pypi.python.org/pypi/mongominer/. then, run the following command to install Python setup. py install:

2. run the following command to test the installation: 20172txt. py samples/simple1.pdf. if the following content is displayed, the installation is successful:

Hello World H e l o W o r l d

3. if you want to use Chinese and Japanese characters, you need to compile them before installing them:

# make cmappython tools/conv_cmap.py pdfminer/cmap Adobe-CNS1 cmaprsrc/cid2code_Adobe_CNS1.txtreading 'cmaprsrc/cid2code_Adobe_CNS1.txt'...writing 'CNS1_H.py'......(this may take several minutes) # python setup.py install

II. use

Since PDF parsing is very time-consuming and memory-consuming, mongominer uses a policy called lazy parsing, which is parsed only when needed, to reduce time and memory usage. To parse a PDF file, there must be at least two types: PDFParser and PDFDocument. PDFParser extracts data from the file and the PDFDocument stores the data. In addition, we also need PDFPageInterpreter to process the page content, which should be converted into what we need by device. PDFResourceManager is used to save shared content such as fonts or images.

The official documentation provides several demos, but they are too simple. Although a detailed Demo is provided, the link address is old and has expired, but the new address denis.papathanasiou.org/posts/2010.08.04.post.html is found.

This Demo is more detailed. the source code is as follows:

#!/usr/bin/pythonimport sysimport osfrom binascii import b2a_hex###### pdf-miner requirements###from pdfminer.pdfparser import PDFParserfrom pdfminer.pdfdocument import PDFDocument, PDFNoOutlinesfrom pdfminer.pdfpage import PDFPagefrom pdfminer.pdfinterp import PDFResourceManager, PDFPageInterpreterfrom pdfminer.converter import PDFPageAggregatorfrom pdfminer.layout import LAParams, LTTextBox, LTTextLine, LTFigure, LTImage, LTChardef with_pdf (pdf_doc, fn, pdf_pwd, *args): """Open the pdf document, and apply the function, returning the results""" result = None try:  # open the pdf file  fp = open(pdf_doc, 'rb')  # create a parser object associated with the file object  parser = PDFParser(fp)  # create a PDFDocument object that stores the document structure  doc = PDFDocument(parser, pdf_pwd)  # connect the parser and document objects  parser.set_document(doc)  # supply the password for initialization  if doc.is_extractable:   # apply the function and return the result   result = fn(doc, *args)  # close the pdf file  fp.close() except IOError:  # the file doesn't exist or similar problem  pass return result### ### Table of Contents### def _parse_toc (doc): """With an open PDFDocument object, get the table of contents (toc) data [this is a higher-order function to be passed to with_pdf()]""" toc = [] try:  outlines = doc.get_outlines()  for (level,title,dest,a,se) in outlines:   toc.append( (level, title) ) except PDFNoOutlines:  pass return tocdef get_toc (pdf_doc, pdf_pwd=''): """Return the table of contents (toc), if any, for this pdf file""" return with_pdf(pdf_doc, _parse_toc, pdf_pwd)###### Extracting Images###def write_file (folder, filename, filedata, flags='w'): """Write the file data to the folder and filename combination (flags: 'w' for write text, 'wb' for write binary, use 'a' instead of 'w' for append)""" result = False if os.path.isdir(folder):  try:   file_obj = open(os.path.join(folder, filename), flags)   file_obj.write(filedata)   file_obj.close()   result = True  except IOError:   pass return resultdef determine_image_type (stream_first_4_bytes): """Find out the image file type based on the magic number comparison of the first 4 (or 2) bytes""" file_type = None bytes_as_hex = b2a_hex(stream_first_4_bytes) if bytes_as_hex.startswith('ffd8'):  file_type = '.jpeg' elif bytes_as_hex == '89504e47':  file_type = '.png' elif bytes_as_hex == '47494638':  file_type = '.gif' elif bytes_as_hex.startswith('424d'):  file_type = '.bmp' return file_typedef save_image (lt_image, page_number, images_folder): """Try to save the image data from this LTImage object, and return the file name, if successful""" result = None if lt_image.stream:  file_stream = lt_image.stream.get_rawdata()  if file_stream:   file_ext = determine_image_type(file_stream[0:4])   if file_ext:    file_name = ''.join([str(page_number), '_', lt_image.name, file_ext])    if write_file(images_folder, file_name, file_stream, flags='wb'):     result = file_name return result###### Extracting Text###def to_bytestring (s, enc='utf-8'): """Convert the given unicode string to a bytestring, using the standard encoding, unless it's already a bytestring""" if s:  if isinstance(s, str):   return s  else:   return s.encode(enc)def update_page_text_hash (h, lt_obj, pct=0.2): """Use the bbox x0,x1 values within pct% to produce lists of associated text within the hash""" x0 = lt_obj.bbox[0] x1 = lt_obj.bbox[2] key_found = False for k, v in h.items():  hash_x0 = k[0]  if x0 >= (hash_x0 * (1.0-pct)) and (hash_x0 * (1.0+pct)) >= x0:   hash_x1 = k[1]   if x1 >= (hash_x1 * (1.0-pct)) and (hash_x1 * (1.0+pct)) >= x1:    # the text inside this LT* object was positioned at the same    # width as a prior series of text, so it belongs together    key_found = True    v.append(to_bytestring(lt_obj.get_text()))    h[k] = v if not key_found:  # the text, based on width, is a new series,  # so it gets its own series (entry in the hash)  h[(x0,x1)] = [to_bytestring(lt_obj.get_text())] return hdef parse_lt_objs (lt_objs, page_number, images_folder, text=[]): """Iterate through the list of LT* objects and capture the text or image data contained in each""" text_content = []  page_text = {} # k=(x0, x1) of the bbox, v=list of text strings within that bbox width (physical column) for lt_obj in lt_objs:  if isinstance(lt_obj, LTTextBox) or isinstance(lt_obj, LTTextLine):   # text, so arrange is logically based on its column width   page_text = update_page_text_hash(page_text, lt_obj)  elif isinstance(lt_obj, LTImage):   # an image, so save it to the designated folder, and note its place in the text    saved_file = save_image(lt_obj, page_number, images_folder)   if saved_file:    # use html style  tag to mark the position of the image within the text    text_content.append('')   else:    print >> sys.stderr, "error saving image on page", page_number, lt_obj.__repr__  elif isinstance(lt_obj, LTFigure):   # LTFigure objects are containers for other LT* objects, so recurse through the children   text_content.append(parse_lt_objs(lt_obj, page_number, images_folder, text_content)) for k, v in sorted([(key,value) for (key,value) in page_text.items()]):  # sort the page_text hash by the keys (x0,x1 values of the bbox),  # which produces a top-down, left-to-right sequence of related columns  text_content.append(''.join(v)) return '\n'.join(text_content)###### Processing Pages###def _parse_pages (doc, images_folder): """With an open PDFDocument object, get the pages and parse each one [this is a higher-order function to be passed to with_pdf()]""" rsrcmgr = PDFResourceManager() laparams = LAParams() device = PDFPageAggregator(rsrcmgr, laparams=laparams) interpreter = PDFPageInterpreter(rsrcmgr, device) text_content = [] for i, page in enumerate(PDFPage.create_pages(doc)):  interpreter.process_page(page)  # receive the LTPage object for this page  layout = device.get_result()  # layout is an LTPage object which may contain child objects like LTTextBox, LTFigure, LTImage, etc.  text_content.append(parse_lt_objs(layout, (i+1), images_folder)) return text_contentdef get_pages (pdf_doc, pdf_pwd='', images_folder='/tmp'): """Process each of the pages in this pdf file and return a list of strings representing the text found in each page""" return with_pdf(pdf_doc, _parse_pages, pdf_pwd, *tuple([images_folder]))a = open('a.txt','a')for i in get_pages('/home/jamespei/nova.pdf'): a.write(i)a.close()

This code focuses on 128th lines. it can be seen that mongominer is a coordinate-based parsing framework. all the components that can be parsed in PDF include the coordinates of the upper, lower, and left edges, for example, x0 = lt_obj.bbox [0] is the coordinate of the left edge of the lt_obj element. Similarly, x1 is the right edge. The code above means to divide all elements with x0 and x1 coordinates less than 20% into a group, thus achieving targeted extraction of content from the PDF file.

---------------- Supplement --------------------

Note that when parsing PDF files, the following error occurs: invalid minerpolicdocument. PDFEncryptionError: Unknown algorithm: param = {'CF ': {'stdcf': {'length': 16, 'cfm':/AESV2, 'authevent':/DocOpen }}, 'O': '\ xe4 \ xe74 \ xb86/\ xa8) \ xa6x \ xe6 \ xa3/U \ xdf \ x0fWR \ x9cPh \ xac \ xae \ x88B \ x06 _ \ xb0 \ x93 @ \ x9f \ x8d ', 'filter ': /Standard, 'P':-1340, 'length': 128, 'r': 4, 'U ': '| UTX # f \ xc9V \ x18 \ x87z \ x10 \ xcb \ xf5 {\ xa7 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 \ x00 ', 'V': 4, 'stmf ':/StdCF, 'strf':/StdCF}

Literally, this PDF is an encrypted PDF, so it cannot be parsed. However, if you open the PDF directly, it is possible that you do not need to enter the password or something, the reason is that this PDF file is encrypted, but the password is empty, so this problem occurs.

To solve this problem, use the qpdf command to decrypt the file (to ensure that qpdf has been installed). to call this command in python, you only need to use call:

 from subprocess import callcall('qpdf --password=%s --decrypt %s %s' %('', file_path, new_file_path), shell=True)

The file_path parameter is the path of the PDF file to be decrypted, and new_file_path is the path of the decrypted PDF file. then, it is OK to parse the decrypted file.

The above is a detailed explanation of how Python uses mongominer to parse the details of PDF instances. For more information, see other related articles in the first PHP community!