How to identify the code of Python web crawler

Source: Internet
Author: User
Tags python web crawler

Some websites are signed in by the way the verification code, such as today we want to test the site patent search and analysis.


The key to logging on to such a site is to identify the verification code. So how to identify the verification code. Let's first look at the source code of the Web page. In the Web page, the verification code is obtained by downloading an image. The picture is src=/sipopublicsearch/portal/login-showpic.shtml

We see from the actual fiddler capture, but also by requesting the above image source address to get a JPEG picture and display in the browser 

So in scrapy we'll first download the image locally and then identify it.
  def   parse (self ,response): 
    ret=response.xpath ( '//*[@id = " Codepic "]/@src ' ). Extract ()
    image_source=ret[ 0]
    image_url=response.urljoin (image_source)
    r=requests.get (image_url)
    with open ( "E://scrapy_project/image2. JPEG ' ) as code:
        code.write (r.content)
First extract the SRC value out, then use the requests method to download and save the picture. Open the file as follows. 

The next step is to start identifying the verification code in the image, which requires the Pytesser and pil libraries. 
The first is to install TESSERACT-OCR and install it after downloading it online. The default installation path is C:\Program FILES\TESSERACT-OCR. Add the path to the system property's path path. 
Then install pytesseract and PIL via Pip . Let's see how it's used. The code is as follows: (' E:\\scrapy_project\\image2. JPEG ')
Im.convert (' L ')
Ret=image_to_string (IM,config='-PSM 7 ')
The results are as follows: The verification code in the picture has been identified.
Image_to_string to configure PSM N, the parameters are explained as follows, generally we choose the first 7 
For N is:
0 = Orientation and script detection (OSD) only.
With OSD.
2 = Automatic page segmentation, but no OSD, or OCR.
3 = Fully Automatic page segmentation, but no OSD. (Default)
of variable sizes.
of vertically aligned text.
of text.
As a single text line.
As a single word.
In a circle.
As a single character.
E:\python2.7.11\python.exe e:/py_prj/

How to identify the code of Python web crawler

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.