BeautifulSoup analysis of Python Development crawler Web page: Crawling home site on the Beijing housing data

Objective: To crawl the home site of the previous 10 pages of Beijing secondary housing data, including the secondary house source name, price, several rooms, size, construction year, contact, address, label and so on.

The website is: https://beijing.anjuke.com/sale/

BeautifulSoup Official website: https://www.crummy.com/software/BeautifulSoup/

Directly on the code:

ImportRequests fromBs4Importbeautifulsoupheaders={'user-agent':'mozilla/5.0 (Windows NT 6.3; Win64; x64) applewebkit/537.36 (khtml, like Gecko) chrome/63.0.3239.132 safari/537.36'}link='https://beijing.anjuke.com/sale/'defgethouseinfo (link): R=requests.get (link,headers=headers) Soup=beautifulsoup (R.text,'lxml') House_list=soup.find_all ('Li', class_='List-item')     forHouseinchHouse_list:name=house.find ('Div', class_='House-title'). A.text.strip () Price=house.find ('span', class_='Price-det'). Text.strip () Price_area=house.find ('span', class_='Unit-price'). Text.strip ()#Unit AreaNo_room=house.find ('Div', class_='Details-item'). Span.text#several rooms and several hallsArea=house.find ('Div', class_='Details-item'). contents[3].text Floor=house.find ('Div', class_='Details-item'). contents[5].text Year=house.find ('Div', class_='Details-item'). contents[7].text Broker=house.find ('span', class_='Brokername'). Text Broker=broker[1:] Address=house.find ('span', class_='comm-address'). Text.strip () address=address.replace ('\xa0\xa0\n',' ') Tag_list=house.find_all ('span', class_='Item-tags') Tags=[i.text forIinchTag_list]Print(name,price,price_area,no_room,area,floor,year,broker,address,tags) forIinchRange (1,11): Link=link+'/ P'+Str (i)Print('page'+str (i)) gethouseinfo (link)

Please refer to the previous blog and BeautifulSoup website for related code analysis.

Output (only part of this is truncated due to too much output):

Page1 Hong Star Real Estate rental Department NEW! 1700 South Garden, donated 5 million mahogany furniture 38,106,442 yuan/M2 5 Room 2 Hall 357m2 a total of 3 layers 2008 construction Summer Fall/winter Blue Water Manor changping-Shahe-Ding Si Road ['single-house Villa','owner's only home','owner Housing'] No, not like it! "Golden Spring Jeonnam house with terrace?" + River View Garden? Rare Ah 10,007,608 Yuan/m2 2 Room 2 Hall 92m2 High-level (30 stories) 2009 Construction of the spring Golden Spring home Chaoyang-village-Tai Tun Road ['Room type is','near Subway','type Quannan'] Five years low down payment, ample parking space, two-bedroom south, northwest Wang, Software Park phase two, lighting good 8,816,287 yuan/M2 3 Room 2 Hall 132m2 Low Level (total 9 stories) built in 2008 Sun Guopeng Mountain (Paulisi) Haidian-Wenquan Town-Black dragon Pool Road ['Matching Mature','Quality Community','living room facing south'] Golden 21 Roland Boulevard Classic type, annual room capacity is limited do not miss 7,361,864 Yuan/M2 3 Room 1 Hall 118m2 High-level (total 15 stories) 2009 Construction Xu Gong Vientiane New Day Home Chaoyang-Changying-Chang Ying North Road 10th ['near Subway','Matching Mature','View Room'] Ocean Lavie: French big alone, three facing the lake, garden 3500 square meters 158,092,941 yuan/M2 6 Room 3 Hall 1700m2 Underground (total 3 stories) built in 2013 Zhang Yunfeng Ocean Lavie Chaoyang-to Canton Camp-Kangying East Road, near the Airport expressway ['Matching Mature','Quality Community','View Room'] Cabbage price! Just to sell the house! The bedroom faces south towards the garden+ Send storage Room "Great!" 12,577,639 Yuan/m2 3 Room 2 Hall 161m2 low level (28 stories) built in 2009, the spring Golden Spring home Chaoyang-village-Tai Tun Road ['Matching Mature','View Room','Bag Check -In'] Tongzhou metro line Line 6 Material College Road+4.5 m Bungalow Apartment + Big property independent room 2.997 million Yuan/m2 2 Room 1 Hall 80m2 High level (total 7 layers) built in 2016 joining North 8080 Tongzhou-Beiguan-bin Yu Dong Lu ['near Subway','Matching Mature','Quality Community'] down payment 1 million regular one bedroom travel convenience 150,000 decoration country wind beauty tang 2,658,956 yuan/M2 1 Room 1 Hall 44m2 Low level (total 6 layers) 2003 Jian creating farmlands Qiang Tian Xin Homeland changping-huoying-Huilongguan East Street [] Beijing core trend just need a house, happy life start place, close to the subway convenient 5,420,973 yuan/M2 2 Room 1 Hall 75m2 a total of 4 storeys 2008 built Tangtian coastal South Chaoyang-Baiziwan-Guang Qu East Road 33rd ['near Subway','Room type is','Bag Check -In'] Cherish the entrusted Qinghe City, New city, a regular one-bedroom full five-year cycle of 4 months 4,170,689 Yuan/M2 1 Room 1 Hall 58m2 mid-level (total 27 stories) built in 2009 Baixiaowei Qinghe New Town Haidian-Qinghe-Beijing-Tibet Expressway, near Qinghe Zhong Jie ['near Subway','Matching Mature','Quality Community'] Kyushu Good listing two years new garden South 88 Ping Transparent large two home edge household hardcover 2,123,863 yuan/M2 2 Room 2 Hall 88m2 Low Level (total 9 stories) 2012 Construction pay Autumn Yan Yu New garden south of Daxing-Daxing District Other-yuk Jen Street 1th ['living room facing south','Small type','a ladder of two households'] Master to do the gate in depth to understand the housing estate store long Bo Tao carefully recommended good faith sell 23,150,326 yuan/M2 4 Room 2 Hall 153m2 High (20 stories) built in 2003 Liu Botao Sun Lai King West-de-Ville-Huangsi Street, 23rd ['near Subway','near School'] Greenland International Huadu Nan District New community environment good North and South permeability elevator room 2,426,086 yuan/M2 2 Room 2 Hall 92m2 Low level (total 15 layers) 2011 Construction Du Shouliang Greenland International Huadu Miyun-Miyun-City back Street 32nd ['Matching Mature','Beautiful Environment','Convenient Transportation'] Only 5w of tax+ South Row Building + front and rear quiet not street + North and South double permeability + with balcony 11,079,994 Yuan/m2 3 Room 2 Hall 137m2 Low level (total 24) 2007 Construction Yang Shaofeng Australia kang Chaoyang-Wangjing West-Wangjing North Road 39th ['Quality Tenants','Beautiful Environment','Quality Community'] on the West two flag Rong ze ka garden Two, Yu Xiang Primary school next to Ming Kitchen Ming Wei Three-bedroom commercial housing!! 5,653,333 USD/M2 3 Room 1 Hall 105m2 mid-level (total 34 stories) 2015 construction Feng Baocheng Rong Jia Yuan Sixth (west side) Changping-Huilongguan-long West Second Road, No. 6th ['Matching Mature','Beautiful Environment','Sub-new district'] West Five ring hardcover new house, Xishan a one, garden Bo Park next to the five period formally enter the market and so you see 8,057,142 yuan/M2 4 Room 2 Hall 140m2 mid-level (total 16 stories) built in 2018 Liu Jianjin Xishan a Fengtai-Marco Polo Bridge-Changshun two road ['Matching Mature','Beautiful Environment','Convenient Transportation'"Focus on the Milky Way Moon" two-bedroom facing south+ Self-living hardcover + all-day lighting + furniture 2,122,580 Yuan/m2 2 Room 1 Hall 93m2 low level (18 layers) 2010 Construction of the Logan Galaxy Moon Beijing surrounding-Yan Jiao-Shun Road, near Yanxing Street ['Matching Mature','Convenient Transportation','Quality Tenants'] Grand Garden East-west four-bedroom, small board floor with elevator hot spring into the home, full five years Ming kitchen Ming Wei 15,627,049 yuan/M2 4 Room 2 Hall 122m2 mid-level (total 7 stories) built in 2005 Tan Naval Grand Park West-taoranting-Fu Chang Street 68th ['near Subway','near School'] Special Room Special room Day spring and autumn hardcover house gift 80 flat loft+20 Terrace 19,470,937 Yuan/m2 4 Room 2 Hall 160m2 High-level (6-storey) 2018 construction Liu Suli Ocean days in the spring and autumn Shijingshan-apple orchard-Golden Summit Road ['North-South permeability','Convenient Transportation','Quality Tenants'] The south end of the row, Garden 200 Ping, has entrusted the key, see the room at any time 16.08 million Yuan/M2 5 Room 3 Hall 200m2 a total of 2 floors 2001 built Liu Fei Sword xiangjiang Garden Chaoyang-to Guang Ying-North Xiangjiang Road 1th ['near Subway','single-house Villa','with Garden'You can choose this set! Classic Practical 3 can also view! There's a key! Metro line Line 14 10,566,878 Yuan/M2 3 Room 2 Hall 157m2 High Level (total 28 stories) 2000 built Wang Yan Pigeon lishui ka yuen Chaoyang-Chaoyang Park West-Chaoyang Park South Road ['near Subway','near School','Convenient Transportation']K2 Group strongly build Tongzhou Garden-style residential community, north-south permeability pattern, 300,000 first pay 2,076,623 yuan/M2 2 Room 2 Hall 77m2 High Level (Total 20 stories) 2018 built Yu Hairi K2 10 Spring Breeze Tongzhou-Tongzhou District Other-Huoxian path ['North-South permeability','Convenient Transportation','Quality Tenants'] new out to see! 200 Flat garden 2015 new decoration big row! 64,100,039 USD/M2 6 Room 4 Hall 581m2 a total of 3 floors 2012 built Peng Wu new home Chaoyang-Asian Games Village-Xin Dian Road 1th # ['near Subway','Low Total Price']..... Fine decoration at any time to see the room north-South permeability Direct bag check in no tax real photos 2,370,434 USD/M2 3 Room 2 Hall 115m2 mid-level (total 6 stories) 2000 built Yang Xingmingzhu Garden (east) Miyun-Miyun-West Bridge Road ['Quality Community','a ladder of two households','away from the road']page10 A new two-bedroom, convenient, near the park, with self-built 4.896 million Yuan/M2 2 Room 1 Hall 50m2 low Level (total 6 stories) built in 1989 Li Bonghan-Hua-an district Chaoyang-Ann-Anding Gate Wai Street ['near Subway','Matching Mature','Quality Tenants'] Chaoyang 5-storey villa, wedding room standard, new not live, the same look at the original, Cofco Rui house 30,094,637 Yuan/M2 4 Room 3 Hall 317m2 a total of 3 floors 2016 construction Dong Weixia Thai wo Beijing Courtyard Chaoyang-to Canton Camp-Shun Huang Road [] Dachang Chaobai River Peacock City Burton Manor Villa owners anxious to sell a key at any time to see the room 7.584 million Yuan/M2 5 Room 2 Hall 315m2 a total of 3 floors 2014 built Tian Wei-min Chaobai River Peacock City Burlington Manor (Villa) Beijing around-Langfang-Houtan line ['Matching Mature','Quality Tenants','High Safety'] gifted mountain Beautiful ground double Garden 200 draw near Shunyi UK* See at any time 26,863,058 Yuan/m2 4 Room 2 Hall 425m2 total 4 stories built in 2008 Li Tootto Yosemite C Area Shunyi-central Villa area-Yu Yuan Road ['Matching Mature','Good lighting','Beautiful Environment'] ... 

Bibliography: Thalictrum, from Python web crawler: Getting Started to practice

BeautifulSoup analysis of Python Development crawler Web page: Crawling home site on the Beijing housing data