Python and Perl are used to draw a running map of Beijing, China.

Python and Perl are used to draw a running map of Beijing, China.

When you have been running thousands of kilometers in a city, crossing the streets and alleys, the obvious idea is how fast I have been and how the traffic has changed, what would happen if we could draw all the routes in the city?

1. Data source: yidong GPS

There are a lot of code in the article. In order not to confuse people, let's first look at the final effect:

[/Code]

First, we need raw data. Many running software on the mobile phone provide detailed records, but the common problem is that they are not allowed to be imported or exported freely (probably for user stickiness ). Therefore, a smart sports watch should be the best choice. My name is Garmin Fenix3. We recommend that you:

Yidong GPS is an industry conscience. It can synchronize codoon, Garmin watches, and yueda circle data. Therefore, I use it as an entry to capture all GPS data.

For how to synchronize data, refer to the introduction on the website. The following is what I logged on to the website:

Http://edooon.com/user/5699607196/record/15414378

After clicking it, you can see the export route button:

Unfortunately, it does not provide batch export buttons, and hundreds of records are exhausted. So let's use the code to edit it.

2. obtain data on the yidong website

After logging on, we can see that it is dynamically loaded. When the scroll wheel is pushed to the bottom, the subsequent content is automatically loaded. It was supposed to sniff and analyze http requests, and then it was lazy. After the file is fully loaded, the current html file is saved.

The next step is to parse this Html, which is basically done through XPath. Experienced students can understand the following points:

The highlighted part is the actual address of the gpx file to be downloaded. We save it in urllist. At the same time, metadata is stored in a json file.

Folder = u 'd:/buptzym synchronization Disk/Baidu cloud/My Documents/Data Analysis/datasets/rungps/'; cookie = 'jsessionid = 69DF607B71B1F14AFEC090F520B14B55; logincookie = 5699607196 $ response; persistent_cookie = 5699607196 $ response; uname_forloginform = "buptzym@qq.com"; _ utma = response; _ utmb = 54733311.5.10.1456907433; _ utmc = 54733311; _ utmz = 54733311.145690 7433.7.3.utmcsr = baidu | utmccn = (organic) | utmcmd = organic; cookie_site = auto'userid = '000000'; f = codecs. open (folder + 'desert.htm', 'R', 'utf-8'); html = f. read (); f. close (); root = etree. HTML (html) tree = etree. elementTree (root); listnode = tree. xpath ('// * [@ id = "feedList"]'); numre = re. compile (u'ride | run | km |, | time consumed | grand cars'); urllists = [] records = []; for child in listnode [0]. iterchildren (): record ={}; temp = child. xpath ('d Iv [2]/div [1]/a [2] ') if len (temp) = 0: continue; source = temp [0]. attrib ['href ']; record ['id'] = source. split ('/') [-1]; info = temp [0]. text; numinfo = numre. split (info); if len (numinfo) <6: continue; record ['type'] = info [0: 2]; record ['distance '] = numinfo [1]; record ['hot '] = numinfo [6]; urllists. append ('HTTP: // edooon.com/user/%s/record/export? Type = gpx & id = % s' % (userid, record ['id']);

It is worth noting that the cookie is required for downloading. Therefore, the reader must replace the userid and the cookie used to log on to GPS.

The next step is the download process. Get the XPath of the URL of the export data button, construct a request with a cookie, and save the file, which is very easy.

opener = urllib.request.build_opener()opener.addheaders.append(('Cookie', cookie));path='//*[@id="exportList"]/li[1]/a';for everyURL in urllists:id = everyURL.split('=')[-1];print(id);url='http://edooon.com/user/%s/record/%s' % (userid, id);f = opener.open(url);html = f.read();f.close();root = etree.HTML(html)tree = etree.ElementTree(root);fs = str(tree.xpath(path)[0]);if fs is None:continue;furl = 'http://edooon.com/user/%s/record/%s' % (userid, fs);f = opener.open(furl);html = f.read();f.close();filename=folder+'id'+'.gpx';xmlfile = codecs.open(filename, 'wb');xmlfile.write(html);xmlfile.close(); 

Then, we saved more than 300 gpx files.

3. parse gpx data

The so-called gpx data is a general standard GPS data format, detailed information can be searched by themselves.

We need to use the python gpx parser. gpxpy is a good choice.

Pip3 install gpxpy.

Gpxpy provides rich interfaces. for statistics, we only need to extract part of the data:

Def readgpx (x): file = open (dir + x + '. gpx', 'R') txt = file. read () gpx = gpxpy. parse (txt) mv = gpx. get_moving_data () dat = {'moving time': mv. moving_time, 'static time': mv. stopped_time, 'moving distance ': mv. moving_distance, 'pause distance ': mv. stopped_distance, 'maximum speed': mv. max_speed}; dat ['total time'] = (gpx. get_duration () dat ['id'] = str (x) updown = gpx. get_uphill_downhill () dat ['upgrade'] = (updown. uphill); dat ['downhill '] = (updown. downhill) timebound = gpx. get_time_bounds (); dat ['start time'] = (timebound. start_time) dat ['end time'] = (timebound. end_time) p = gpx. get_points_data () [0] dat ['lat'] = p. point. latitudedat ['lng '] = p. point. longitudefile. close () return dat

The readgpx function reads the file name x and returns a dictionary. And get a table similar to the following:

Because we only need to plot the region of Beijing, we need a Coordinate Expression to screen out the region outside Beijing. The screening Code uses pandas and contains more detailed code in the attachment.

Required tids = detailed [(detailed. lng <116.1) | (detailed. lng> 116.7) | (detailed. lat <39.9) | (detailed. lat> 40.1)]. id

def filtercity(r):sp=r.split('/')[-1].split('.')if sp[1]!='gpx':return False;if sp[0] in exceptids.values:return False;return True; bjids= [r for r in gpxs if filtercity(r)] 

In this way, we will filter out all the sports data completed in Beijing.

4. Draw GPS Data

Making wheel repeatedly is not fun, drawing gpx has been relatively powerful library, address in http://avtanski.net/projects/gps/

Unfortunately, this library uses Perl as the development language and GD as the visual rendering library. I spent a lot of time installing GD.

By default, Perl is installed in Ubuntu. GD requires libgd, but libgd is very difficult to download on the official website. After downloading it, I find that the version is incorrect, which allows me to travel abroad on the Internet for several hours, they are all about to die... In the end, I discovered that the libgd library can be installed as long as the following steps are completed:

Apt-get install libgd-gd2-perl

I think this is where apt-get is stuck. apt get gd or libgd cannot be found at all. If you don't check it, who knows how to write it! As for Perl's CPan management tool, it's a tear.

Next, download gd 2.56, which is a very new version. I found various installation steps for the Chinese version and found all problems. The best way to do this is read README. MD!

After decompression, perl./Makefile. PL

Make later

Make install

Then you can ......

This gpx library introduces itself as follows:

This folder contains several Perl scripts for processing and ploying

GPS track data in. GPX format. its readme has many usage instructions. Of course we don't talk nonsense about it. Copy all gpx data to the sample_gpx folder, and then run the program with Lili. /runme. if there is no problem with sh, it should be as follows:

[/Code

I suppose you are familiar with bash. You can modify the runme. sh file to view more options. The final result is as follows:

I was shocked when I saw this result! This is the result of self-running around 2000 Kilometers. The main roads in the third ring road of Beijing (mainly concentrated in the north of Chang 'an Street) are all over. In particular, the North Third Ring Road and North Tucheng Road (North Section of Line 10) were abused by me. Every white line is a story, and every point is my footprint!

5. Summary

This article is clearly not detailed enough, far from hand by hand. And I didn't provide more data analysis (obviously I have done all these jobs), but I believe that the running programmers must be very good. I have the right to give a reference.

In fact, it can be made into a web service. When the runners upload their own running software IDs, they can automatically render various beautiful running paths and analysis diagrams. It should be very meaningful!

It took me seven or eight hours to get rid of blood and spent a lot of time installing GD instead of downloading data. The lesson tells me that you must read the instruction documents that come with the installation package. Because the versions between libraries are different, it may cause version hell, and the new version cannot be uninstalled at that time, when the old version is unavailable, don't say I didn't remind you!

It is worth mentioning that the gpx file downloaded by mobile gps does not contain line breaks, which causes the gpx_disualization library to fail to parse it (this regular expression is wrong). I am too lazy to change the perl regular expression, therefore, line breaks are added by replacing them.

The above is a small series of Python and Perl methods to draw a map of China's Beijing running, I hope to help you!

Articles you may be interested in:

• Using a Python Mysql database to operate a Mysql database using Perl
• Some similarities and differences between Perl and Python
• Use perl, python, php, shell, sed, awk, and c to flip strings
• Python and perl: Batch renaming of e-book files in the directory
• Use Python to simulate the spread of plague on a map.
• Accessing MySQL database code instances using Shell, Perl, Python, and PHP