Are you hungry? There are so many unlicensed shops! Or big Python is more awesome!

First look at the crawl, there are so many shops do not have a business license.

Second, the operating environment

• Python3

• Pymongo

• Requests

Third, analysis

First visit the Hungry home page, enter the area you want to search, and the page returns to the nearby store.

The page browser address is as follows:

https://www.ele.me/place/wtw39y8614v4?latitude=31.237236&longitude=121.36636

Open the Browser developer tool, analyze the data that needs to be crawled, then fetch the page address with requests, and use the BeautifulSoup to parse the data and return the empty value data.

Review the page information again, scroll down the page and discover that the XHR asynchronous load type has returned JSON-formatted data. As shown in the following:

Wheat-Mouth hu

Links: https://www.jianshu.com/p/19c38bfa7388

Source: Pinterest

Copyright belongs to the author. Commercial reprint please contact the author for authorization, non-commercial reprint please specify the source.

Where parameters:

1, Longitude,latitude adopted the German coordinate system corresponding to the latitude and longitude of the search area

2, limit=24 each limit to return a maximum of 24 stores; set up more than 30 back

3, offset=24 each return of the number of stores, 24 orders of magnitude increase, such as the page continues to turn offset=48,72 ...

More store data can be obtained by increasing offset offsets from the loop.

This page we need to collect name, phone, address that is the merchant name, telephone, business addresses three data can be. A page can collect 24 business data, JSON data such as:

The code is as follows:

Import Requestsimport jsonurl= ' https://www.ele.me/restapi/shopping/restaurants?extras[]=activities&geohash= Wtw39y8614v4&latitude=31.237236&limit=24&longitude=121.36636&offset=24&terminal=web ' r= Requests.get (URL). Textd=json.loads (R) for V in D:print (v[' name '],v[' phone '],v[' address ')

The returned results are as follows:

Four, the code

V. Results show

Sorted by address in MongoDB, sort ({address:1}) sorted by address ascending

Have you learned it?

You are welcome to follow my blog: ttps://home.cnblogs.com/u/python1234

Welcome to join thousands of people to answer questions group: 125240963

Are you hungry? There are so many unlicensed shops! Or big Python is more awesome!