Search cheap second-hand houses with Web Robots

1. Problem

I need a second room and one living room. The price is about 0.4 million. It is a second-hand house located in several districts in Shanghai (such as Xuhui District and Minhang District. Of course, my ideal house must meet many requirements. For example, it must be a house with two rooms facing south and its average price must be less than 7000 yuan per square meter.

I usually use the Shanghai hotline of second-hand housing network (http://secondhand.online.sh.cn/) to find the latest second-hand housing information.

However, it is too tiring to track the latest second-hand house information on the website every day. The second-hand house network adds a lot of latest information every day. To identify a house that meets my requirements, I must read all the information. The whole process is a waste of time.

The reason is:

1. The website will only display the ** summary ** of some newly added second-hand houses on the homepage. In order to make my own judgment, it is not enough to look at the summary information. I must look at the details. This means that I have to click the "details" hyperlink in the summary to download the details page.

2. Not all information is stored on a page. In fact, the information of the Shanghai hotline's second-hand house network is displayed on pages, and each page shows only 10 pieces of information. If I need to view all the information, I need to check about-pieces of information (the number of information depends on whether the search criteria are strict and the size of the area to be searched ).

The above means that even if I only need to view 500 pieces of information, I also need to view 550 pages:

500/10 (summary page) + 500*1 (details page of each piece of information) = 550

The loading time for each page is 10 seconds (the actual sampling result), and the time for reading each page is 25 seconds (my reading speed is very fast ).

550*(10 + 25) = 19250 seconds = 320 minutes = 5 hours.

That is to say, it takes me five hours to read information. In fact, I have not considered the time to analyze the information (for example, list the 10 houses with the lowest average price), which is far more time than reading.

2. Nature of the problem

Because many of the information is now stored on the internet, similar problems are common. Such as recruitment and securities. This problem is how to efficiently obtain and analyze data from the Internet.

3. A solution

I have read some related articles in China. The solutions of these articles are based on IE automation. The webbrowser control is usually added in VB.

The main drawback of this solution is that webbrowser encapsulates many data operations, which makes simple operations very troublesome (I guess the reason is that Microsoft designed webbrowser to automate ie operations ). For example, to obtain the current page, if webbrowser is used for development, the current page needs to be obtained separately. In an event, the page is successfully obtained by two processes. It takes a lot of extra work to make the two processes communicate.

My solution is to use python to develop a script, because the python Library supports a relatively complete set, this can also obtain the additional benefits of cross-platform. I use structured design because the use of web robots to obtain data is structured. I encapsulate operations related to a specific website in several functions, which means that other applications can be used if the script is modified as appropriate. That is to say, this script is not only used for the Shanghai hotline second-hand house network, but also used to search for second-hand houses.

Before writing this script, I have some experience writing Web Robots, including Vb, Java, and python. I think the Web robot I wrote is quite mature.

The main points of this solution are as follows: First, submit the conditions to the second-hand house network, then obtain the paging search results, and read the information terms () from the first page (), read the necessary information such as the total number of pages (read of such information also requires hard encoding of the start and end signs) and so on. Then, based on the information, simulate page turning and click the "details" link. These actions are nothing more than get and post actions in the HTTP standard.

Some of the above schemes require hard encoding. If necessary, analyze the browser client script (such as JavaScript ):

* Start and end signs of various information terms

* Start and end signs of the current page count

* Start and end signs of the total number of pages

* The hyperlink address in the page flip action

* Click the hyperlink address in the "details" hyperlink.

I printed all the obtained data on the screen. The data is separated by commas. If you want to import the data to a file, you can use the MPs queue mechanism.

For example, assume that the script is house.py. you want to import the data of house.pyto the file house.csv. You can enter the following command:

Python house. py> house.csv

You can analyze house.csv. I usually use editors that support regular expressions to drop some formatting characters in house.csv and use Excel for data analysis.

4. Specific implementation and code

Some minor problems are also encountered in the specific implementation. For example, to support Chinese characters, I must add the line "#-*-coding: MBCS-*-" at the beginning of the script. I need to use the specified proxy server to access the Internet, so I use the urllib2 library instead of the urllib library. These problems have been explained in the source code.

The following is the source code:

1 #-*-coding: MBCS-*-2 3 # ------------------ user configuration ------------ begin 4 5 # Do you want to use the HTTP proxy to be verified? (Limited to a very small number of companies, usually set to 0) 6 config_using_proxy = 0 7 8 # proxy address and port number, valid only when config_using_proxy is equal to 1 9 config_httpproxy = "" 10 11 # username used to log on to the proxy server, valid only when config_using_proxy is equal to 1, otherwise, you can ignore 12 config_username = "" 13 14 # The password used to log on to the proxy server, which is valid only when config_using_proxy is equal to 1, otherwise, you can ignore 15 config_password = "" 16 17 # select Changzhi district ("", "Minhang", "Xuhui", "Yangpu", "Hongkou", "Changning ", "Jing 'an", "Luwan", "Huangpu", "Zhabei", "Putuo", "Pudong", "Baoshan", "Jiading", "Qingpu", "Fengxian ", "Nanhui", "Jinshan", "Songjiang", "Chongming"), "" indicates all 18 config_szqx = "Hongkou" 19 20 # How many rooms are selected? ("", "1", "2", "3", "4", "5") 21 config_fx_room = "2" 22 23 # How many rooms do I select? ("", "0", "1", "2", "3", "4") 24 config_fx_hall = "1" 25 26 # select the lower limit of the total price (unit: 10 thousand), ("0", "10", "15", "20", "25", "30", "35", "40", "50 ", "60", "70", "80", "90", "100 "), "0" indicates not limited 27 config_jg_min = "20" 28 29 # select the upper limit of the total price (unit: 10000000), ("", "10", "15", "20 ", "25", "30", "35", "40", "50", "60", "70", "80", "90 ", "100"), "10000000" indicates no limitation 30 config_jg_max = "60" 31 32 # select the transaction method, ("0", "1"), 0 indicates the sale, 1 indicates replacement of 33 config_type = "0" 34 35 # registration date (unit, Day), ("15", "30", "180 ",""), "" 36 config_djrq = "15" 37 38 # search for the flag string starting from the first row of the table (the first row starts searching for the row marker after the string) 39 config_tbl_begin_str = "> districts and counties <" 40 41 # search for the flag string at the end of the last row of the table (the last row starts searching for the end flag of the row before the string) 42 config_tbl_end_str = "available eligible Listings" 43 # Your User Configuration --------------- end 44 45 46 # Your administrator configuration ------ begin 47 48 config_post_data = {"szqx": config_szqx, "fx_room": config_fx_room, "fx_hall": config_fx_hall, "jg_min": config_jg_min, "jg_max": config_jg_max, "type": config_type, "djrq": comment, "sortfield ": "djrq", "sorttype": "DESC", "whichpage ": "1"} 49 50 # ---------------------- administrator configuration ------ end 51 52 53 54 55 from string import * 56 import sys 57 58 # ----------------- print routines ------------- begin 59DefDump_row_end (): 60 SYS. stdout. Write ('/N') 61 62 63DefDump_table_begin (): 64 sys. stdout. write ("district/county" + ",") 65 sys. stdout. write ("Property Address" + ",") 66 sys. stdout. write ("room type" + ",") 67 sys. stdout. write ("property type" + ",") 68 sys. stdout. write ("building area" + ",") 69 sys. stdout. write ("total price" + ",") 70 sys. stdout. write ("registration time" + ",") 71 sys. stdout. write ("housing number" + ",") 72 sys. stdout. write ("property name" + ",") 73 sys. stdout. write ("room age" + ",") 74 sys. stdout. write ("property description" + ",") 75 sys. stdout. write ("Intermediary quote" + ",") 76 sys. stdout. write ("property details" + ",") 77 sys. stdout. write ("house orientation" + ",") 78 sys. stdout. write ("floor" + ",") 79 sys. stdout. write ("decoration degree" + ",") 80 sys. stdout. write ("indoor condition" + ",") 81 sys. stdout. write ("validity period" + ",") 82 sys. stdout. write ("standby note" + ",") 83 sys. stdout. write ("Contact" + ",") 84 sys. stdout. write ("contact number" + ",") 85 sys. stdout. write ("/N") 86 87 88DefDump_one_field (STR): 89 SYS. stdout. Write (STR + ",") 90 91 # ----------------- print routines ------------- end 92 93 94 # ------------------- house parser ------------- begin 95DefGet_last_page_number (s): 96 no_begin = find (S, "Last page") 97IfNo_begin =-1: 98Return0 99 100 no_begin = rfind (S, "javascript: form_submit (/'", 0, no_begin) 101 no_begin + = Len ("javascript: form_submit (/'") 102 no_end = find (S, "/'", no_begin) 103IfNo_end =-1:104Return0105 106IfNo_begin> no_end: 107Return0108ReturnAtoi (s [no_begin: no_end]) 109 110 111DefGet_data_in_one_tag (latency 4, tag): 112 tag_begin = find (latency 4, "<" + tag) 113IfTag_begin =-1:114ReturnLimit 4115 116 tag_begin = find (limit 4, ">") 117IfTag_begin =-1:118Return20174119 tag_begin + = 1120 121 tag_end = find (limit 4, "</" + tag + ">", tag_begin) 122IfTag_end =-1:123Return20174124 125ReturnLifecycle 4 [tag_begin: tag_end] 126 127DefFilter_rubbish_data (STR): 128ReturnStrip (replace (STR, ""), '/N', ''),'/t ',''), '/R', '') # Maybe we will output data in CSV format129 130DefGet_one_detailed_data (partition 3, keyword): 131 132 # print partition 3 # debugworkflow # print keyword # debug134 135 data_begin = find (partition 3, keyword) 136IfData_begin =-1: HangzhouReturn"" 138 # handle data 139 data_begin = find (versions 3, "<TD", data_begin) 140IfData_begin =-1:141Return"" 142 143 data_begin = find (orders 3, ">", data_begin) 144IfData_begin =-Return"" 146 data_begin = data_begin + 1147 148 data_end = find (products 3, "</TD>", data_begin) 149IfData_end =-1:150Return"" 151 152IfData_begin> data_end: 153Return"" 154 # Delete space, comma, tab and linefeed155 # Return Replace (partition 3 [data_begin: data_end], "") 156ReturnFilter_rubbish_data (rows 3 [data_begin: data_end]) 157 158 159DefGet_detailed_data (second 2): 160 dump_one_field (second 2, "house number") 161 dump_one_field (second 2, "property name") 162 dump_one_field (get_one_detailed_data (second 2, "Age") 163 dump_one_field (get_one_detailed_data (Region 2, "") 164 dump_one_field (get_one_detailed_data (Region 2, "")) 165 # Delete the href 166 tmpstr = get_one_detailed_data (Region 2, "Property Address") 167 tmppos = find (tmpstr, "<a") 168IfTmpstr <>-1: 169 tmpstr = strip (tmpstr [: tmppos]) 170 dump_one_field (tmpstr) 171 dump_one_field (get_one_detailed_data (orientation 2, "orientation ")) 172 dump_one_field (get_one_detailed_data (second 2, "floor") 173 dump_one_field (second 2, "decoration degree") 174 dump_one_field (get_one_detailed_data (second 2, "indoor conditions ")) 175 dump_one_field (get_one_detailed_data (expiration 2, "validity period") 176 dump_one_field (expiration 2, "standby note") 177 dump_one_field (partition (get_one_detailed_data (Limit 2, "Contact person"), "Div") 178 dump_one_field (get_data_in_one_tag (get_one_detailed_data (contact number 2, "contact number"), "Div") 179 180 181DefGet_data (instr, tbl_begin_str, tbl_end_str): 182 # Table begin 183 idx = find (instr, tbl_begin_str) 184IfIdx =-1: 185Return186 idx = find (instr, "<TR", idx) 187IfIdx =-1: 188Return189 table_begin = idx190 # print instr [table_begin: table_begin + 100] # debug191 192 # Table end 193 idx = find (instr, tbl_end_str, table_begin) 194 # print instr [table_begin: id x] 195IfIdx =-1: 196Return197 idx = rfind (instr, "</tr>", table_begin, idx) 198IfIdx =-1: 199Return200 table_end = idx + Len ("</tr>") 201 # print instr [table_begin: table_end] # debug202 203 # search rows204 tr_idx = table_begin205WhileTr_idx <table_end: 206 # tr begin207 tr_idx = find (instr, "<TR", tr_idx) 208IfTr_idx =-1:209Return210 tr_idx = find (instr, ">", tr_idx) 211IfTr_idx =-1:212Return213 tr_begin = tr_idx + 1214 215 # tr end216 tr_idx = find (instr, "</tr>", tr_begin) 217IfTr_idx =-1:218Return219 tr_end = tr_idx220 # print instr [tr_begin: tr_end] # debug221 222 223 # search cells in one row 224 td_idx = tr_begin225 is_really_a_row_dumped = 0226WhileTd_idx <tr_end: 227 # TD data begin228 td_idx = find (instr, "<TD", td_idx) 229 # print td_idx # debug230IfTd_idx =-1:231Return232 td_idx = find (instr, ">", td_idx) 233 # print td_idx # debug234IfTd_idx =-1:235Return236 tddata_begin = td_idx + 1237 238 # TD data end 239 td_idx = find (instr, "</TD>", td_idx) 240 # print td_idx # debug241IfTd_idx =-1:242Return243 tddata_end = td_idx244 245IfTddata_begin> tddata_end: 246Continue247 248IfTddata_end> tr_end: 249Continue250 251IfTddata_end> table_end: 252Continue253 254 tddata = filter_rubbish_data (instr [tddata_begin: tddata_end]) 255 256 # If the tddata is a href, let's get more data from the href 257 href_begin = find (tddata, "href =/" javascript: urll (/") 258IfHref_begin =-dump_one_field (tddata) 260Continue261 262 263 href_begin = href_begin + Len ("href =/" javascript: urll (/'") 264 265 href_end = find (tddata,"/' ", href_begin) 266IfHref_end =-1: 267Return268 269 view_url = "http://secondhand.online.sh.cn/" + tddata [href_begin: href_end] 270 # print view_url # debug271 # dump_one_field (view_url) 272 273 view_result = inline (view_url) 274 view_data = inline () 275 # print "view_data =" + view_data # debug276 get_detailed_data (view_data) 277 is_really_a_row_dumped = 1278 279IfIs_really_a_row_dumped: # Sometimes, no TD output280 dump_row_end () 281 # ----------------- house parser ----------- end282 283 284DefInstall_proxy (): 285 httpproxy = config_httpproxy286 username = config_username287 Password = config_password288 httpproxystring = 'HTTP: // '+ username + ': '+ password +' @ '+ httpproxy289 290 # build a new opener that uses a proxy requiring authorization291 proxy_support = urllib2.proxyhandler ({"HTTP": httpproxystring}) 292 293 authinfo = login () 294 opener = urllib2.build _ opener (proxy_support, authinfo, urllib2.httphandler) 295 296 # Install it297 urllib2.install _ opener (opener) 298 299 # ---------------- main pipeline begin300If_ Name __= = "_ main _": 301 # Get the page302 import urllib2303 import urllib304 305 # Using proxy 306IfConfig_using_proxy: 307 install_proxy () 308 309 F = urllib2.urlopen ("http://secondhand.online.sh.cn/selllist.php", urllib. urlencode (config_post_data) 310 # print F. headers # debug311 S = f. read () 312 # print s # debug313 314 # parse the HTML page 315 # S = "<Table> <tr> <TD> data11 </TD> <TD> data12 </TD> </tr> <TD> data21 </TD> <TD> data22 </TD> </tr> </table> "# debug316 # config_tbl_begin_str =" <Table> "# debug317 # config_tbl_end_str =" </table> "# debug318 319 # print out the table header 320 dump_table_begin () 321 # print out the first page 322 get_data (S, config_tbl_begin_str, config_tbl_end_str) 323 324 # Get the page size from the first page data 325 last_page = get_last_page_number (s) 326 # print out other pages (if exist) 327ForIInRange (2, last_page): 328 config_post_data ['whichpage'] = STR (I) 329 F = urllib2.urlopen ("http://secondhand.online.sh.cn/selllist.php", urllib. urlencode (config_post_data) 330 s = f. read () 331 get_data (S, config_tbl_begin_str, config_tbl_end_str) 332 333 # S = "<TD Height = 26> header1 </TD> <TD Height = 26> data1 </TD>" # debug334 # print get_one_detailed_data (S, "header1") # debug335 336 # print get_last_page_number ("<a href =/" javascript: form_submit (/'51/',/'djrq/',/'desc /') /"> last page </a>") # debug337 # ---------------- main --------------------- end