Use the PHP blue Sky grabber to capture the content of today's headline Ajax articles

Today's headline data are AJAX loaded display, according to the normal URL is not fetching data, need to analyze the loading address, we Https://www.toutiao.com/search/?keyword=%E6%96%B0%E9%97%BB An example to capture a list of articles

Open the link with Google Chrome, right click on "review" to switch to network in the console and click Xhr, so you can filter images, files and so on unnecessary requests only see page content request

Because the page is Ajax loaded, so pull the page to the bottom, will automatically load more articles, this time the console grabbed the link is what we really need the list page link:

Https://www.toutiao.com/search_content/?offset=20&format=json&keyword=%E6%96%B0%E9%97%BB&autoload= True&count=20&cur_tab=1&from=search_tab

Create a task in blue Sky Collection

Click "Collect Settings" to fill in the "Start Page URL" to the above crawled link

Next match content page URL, headline article URL format is https://www.toutiao.com/group/number/

Click "Content page URL" to write the "Match content url" rule:

(? <content1>http://toutiao.com/group/\d+/)

This is a regular rule, meaning that the matching URL is loaded into the capture group Content1, and then fill in the following [content 1] that corresponds to the above content1 can get to the content page link

You can click Test to see if the link was successfully crawled

Grab a successful start to get the content

Click "Get Content" to the right of the field list to add the default fields, such as the title, body, etc. can be intelligently identified, if you want to accurately edit the field, support Regular, XPath, JSON and other matching content

We need to crawl the title and text of the article, because it is the Ajax display so to write rules to match the content, analysis of the source: https://www.toutiao.com/a6358823350874145025/, find the article location

Title rule: articleinfo\s: \s{\stitle:\s' [content 1] ',

Body Rules: content\s: \s' [content 1] ', \s*groupid

Rules must be unique, otherwise they will be matched to other content, add rules to the field, and get the way to select the rule to match:

After the rules are written, click Save, and then click "Test" to see how it works.

The rule is correct, crawl normal, crawl to the data can also be published to CMS system, direct database storage, save as Excel file, click the bottom navigation bar "Release settings" can, good today the headlines of the collection to the end of here, we might as well try!

Use the PHP blue Sky grabber to capture the content of today's headline Ajax articles