Urban public transit data download (continued) analysis process and source code (supporting 440 cities in China)

ArticleDirectory

• 1. Find the target data website
• 2. List of all bus in the city
• 3. Obtain the specific data of each bus
• 4. Other Instructions:
• 5. the problem to be solved,

I had nothing to do with the download of public transit data in a city before, and it was okay on weekends. I saw a message saying that there was no Kunming, and I wanted to download the data from across the country. So I wrote it out:

• 1. More cities are supported, including 440 cities in China. For more information, see
• 2. multi-thread download,
• 3. Use independent storage to save the serialized city list, so that the list can be downloaded only when it is opened for the first time.

The following is my implementation process. I will share with you: 1. Find the target data website.

Enter "national bus query" in Baidu, the first http://old.8684.cn/to check that all provinces and cities in China are here.

2. List of all bus in the city

Go to http://shenzhen.8684.cn/in Shenzhen ,:

Seeing this, I was disappointed not to list all the buses in Shenzhen as I did in the previous dibao.
Finally, I learned something about it:

In this tab, select a line type in firebug, You can see Ajax GET request http://shenzhen.8684.cn/city1.php? T = 16/PP7, the returned data is:
, 18 long-distance, 19 long-distance, 58 long-distance, 68 long-distance, 328 long-distance, 5 long, 7 long, 9 long, 10 long, chang16 Road, chang21 Road, chang28 Road, kang11 Road, Shenhui chengba line 1, Shenhui chengba line 2, Shenhui chengba line 1, shenyi chengba Line 2
It looks like all bus lines under this type. In this case, retrieve all types and then request the page loaded by ajax to retrieve all bus lists.
Then I found the category of the bus route in http://shenzhen.8684.cn/js/16.js, so that I can obtain the list of all bus routes in the city. 3. Obtain the specific data of each bus.

In the above figure, after selecting a bus, the line name will be placed in the text box above,
In addition, I found that the address of the submitted page is x_5cbfef9e Based on the line name (because 1 Road in Shenzhen and 1 Road in other cities, the following is the same
Here I come up with two methods to get the data of the bus thread: 1. simulate form post, 2. the specific URL of the line is encrypted Based on the name. You can just unlock it.
However, I am not satisfied with the two methods. If there are too many public buses in post, the efficiency may be relatively low, and I have no way to decrypt them,
There seems to be a deadlock here, but there are always some solutions. You can use firebug to view the "line search" button and find that it calls fsub (1)
Then the fsub (1) method is found in home. JS, followed by location. href, and the actual address is so. php? K = PP & Q = 1, followed by the name of the bus line
In addition to the list of all previously obtained routes, all bus data can be downloaded. 4. Other Instructions:

A. InProgramSo. php? When K = PP & Q = 1 URL, the following parameters must be encoded using gb2312
B. for example, in Beijing, and so on, fuzzy results may occur sometimes, so the program judges that if the P in UL is not matched, it will find another a tag again, directly find and construct a specific address to obtain
C. There are small cities like Cixi, Zhejiang province, which are different from other cities. 5. Special treatment is required. 5. problems to be solved,

Hope you can give me some advice:
A. The thread pool is used in the program. I do not know if the thread pool can be stopped.
B. Another 20 threads are used, but with 360 of the traffic, we found that there are only 2 to 3 actually connected threads.
C. Independent memory does not know how to determine the creation or access time, so that the city list can be updated after one week (or longer ).

Finally:
Program download
C # source code download(Download points are not required for csdn)
Interface: