Data mining tools Analysis of Beijing house price (I) data crawling and Acquisition

I. Preface

House prices are always the biggest headache and excitement, especially in the imperial capital. I also want to use a topic of interest as a case to conduct a complete drill on the software I have developed. A complete process is implemented from data collection, cleaning, analysis, and presentation of final visualization and reports. On the one hand, we can share some useful information with you, and better understand the use process of this software. For more information about this tool, see Introduction to the data mining platform (Summary)-platform introduction.

Naturally, the article is divided into four parts. This section is the first part: Data crawling and collection.

2. Preparations

As the saying goes, it's hard to be a man without rice. Without data, analysis is meaningless. In order to find the right data, I moved to the official website of the chain. Http://beijing.homelink.com.cn/here, although the data is far from accurate, after all, the online things and real data will certainly have a gap, but the readability is very good, the data is also relatively complete. Let's see how to crawl the data!

Chain Home Website:

The total number of houses is 72964 units on sale. The main concerns are area, price, location, characteristics, geographical coordinates, and users (reflecting the popularity ). Let's see if we can capture the data!

The Web collector introduced in this article has the following features:

• No need to write regular expressions or code
• You only need to select and name the data you are interested in (the default name is also available)
• Set the batch execution method to Perform Batch collection.
• Suitable for crawling list pages
• Simple, convenient, and efficient
• Ability to crawl dynamic webpages and webpages that require Logon (this operation is slightly more complex)

 

 

First, drag and drop the "Web page Collector" module into the algorithm processor:

Is the webpage collector's setting menu:

First fill in the URL path, for the second-hand data path, the general default encoding for the UTF-8, the program to identify the encoding format. Click to access the webpage:

You can view the webpage in the original HTML and browser in the main window:

I usually use HTML source code to view data.

Copy the data you are interested in from HTML to the "Filter field" dialog box. The system automatically searches for the corresponding path.

 Name the attribute, such as "Introduction". Click "add field ".To add a property.

Similarly, you can add fields to the hyperlinks.

When selecting data, select representative data as much as possible. At the same time, data should be distributed among different items in the list.

You can view and modify the added attributes at any time. In advanced options, click attribute list to view all the attributes currently added and add, delete, modify, and query the attributes.

After selecting five data items of interest, click "start crawling ". At this point, the program will try to capture the web page. We can use this attempt to determine the accuracy of program execution:

You have obtained 12 items and click to view them at the same time:

Similarly, we can add others, including the number of floors and orientation, in a similar way. If some data programs cannot be automatically discovered, hand-written XPath can be used to solve the problem.

3. Batch Tasks

After adding up to 10 attributes and performing a Web page crawling test, we think that the program settings are okay, then we can perform batch web page crawling.

You can set the batch task attributes. We can see from the web page that there are a total of 6073 pages:

At the same time, the difference between different pages is only at the end of the URL:

Http://beijing.homelink.com.cn/ershoufang/pg2/

Http://beijing.homelink.com.cn/ershoufang/pg6074/

Therefore, the batch parameters can be written as follows:

Http://beijing.homelink.com.cn/ershoufang/4250#/, you can change the number to be replaced to the braces.

Added a latency of 10 ms.

Save the current crawler as a task and name it "chain crawler"

Next, on the menu bar,Run->Task> Start a batch task

Select the task to be executed as a "chain crawler ":

Set the module to be traversed as a Web page COLLECTOR:

Select the traversal parameters:

Set the parameter list:

You can enter the interval, maximum value and minimum value to generate parameters on the left side, or enter the parameters to be traversed with one parameter per line on the right side. The generated parameter is 1-6074.

Click Finish to execute the batch task:

In about two hours, you can crawl all the data ~~~ Hahaha

Check the results:

We export the data to a file for further analysis. (It can also be saved to the database)

Select the file format and enter the file name:

Select the field to be exported and whether to modify the field name:

Click OK to export the data.

Task completed!

Iii. Summary

As a result, 70 thousand pieces of data were collected within two hours, and the entire configuration took less than two minutes. However, we must find that there are many data problems. For example, if the area and some attributes are empty, it will affect data analysis in the future. Therefore, we will introduce data cleansing in the next section.

Data mining tools Analysis of Beijing house price (I) data crawling and Acquisition