Python reads PDF content

1, Introduction

At night read the "Python Network data Collection" This book, see the code reading the PDF content, think of the previous few days the collection has just released a crawl page PDF content Crawl rules, this rule can be the PDF content as HTML to do Web crawling. The magic is due to the ability of Firefox to parse PDFs, to convert the PDF format into HTML tags, such as a div tag, to use Gooseeker web crawler software like a common Web page to grab structured content.

This creates a problem: how much can you do with a python crawler? An experimental procedure and source code are described below.

2. Python source code to convert PDF to text

The following Python source code reads the contents of the PDF file (on the Internet or locally), converts it into text, and prints it out. This code uses a third-party library pdfminer3k to read the PDF into a string and then convert it to a file object using Stringio. (Source code download address see GitHub Source at the end of the article)

From urllib.request import urlopenfrom pdfminer.pdfinterp import Pdfresourcemanager, Process_pdffrom Pdfminer.converter Import textconverterfrom pdfminer.layout import laparamsfrom io import stringiofrom io import opendef r Eadpdf (pdffile):    rsrcmgr = Pdfresourcemanager ()    retstr = Stringio ()    laparams = laparams ()    device = Textconverter (Rsrcmgr, Retstr, Laparams=laparams)    process_pdf (rsrcmgr, Device, Pdffile)    device.close ()    content = Retstr.getvalue ()    retstr.close ()    return contentpdffile = Urlopen ("http://pythonscraping.com/ Pages/warandpeace/chapter1.pdf ") outputstring = Readpdf (pdffile) print (outputstring) pdffile.close ()

If the PDF file is on your computer, replace the Urlopen returned object pdffile with the normal open () file object.

3, Outlook

The experiment just converted the PDF into text, but there was no conversion from the beginning to the HTML tag, so the ability to do so in a python programming environment was left to be explored in the future.

