Use Python and Perl to draw the Beijing run map

When you are in a city, across the streets, running for thousands of kilometers, an obvious idea is, how much faster I am than before, how the running volume changes, if you can all the routes in this city to draw out, what will be the scene?

1. Data Source: Ipsos GPS

Article code more, in order not to hang people's appetite, first look at the final effect:

First of all, there is the raw data information, many running software on the phone provide detailed records, but their common problem is not allow the free import export (probably for the user sticky bar). Therefore, a smart sports watch should be the perfect choice. My is Garmin Fenix3, recommend:

The plump GPS is the industry conscience, able to synchronize the data of the Garmin Watch and the lap, so I will use it as a portal to crawl all the GPS data.

As for how to synchronize, you can refer to the relevant information on the website, the following is my login to the site after:

http://edooon.com/user/5699607196/record/15414378

When you go in, you can see the button that exports the route:

The very pit daddy is, it does not provide the batch export button, hundreds of records, sequentially exports all exhausted. So consider using code to edit it.

2. Access to data on the Ipsos website

After logging in, you can see that it is dynamically loaded, and when the scroll wheel rolls to the bottom, it automatically loads the contents of the back. It was supposed to sniff and parse HTTP requests, and then lazy. When it is all loaded, the current HTML file is saved.

The next step is to parse the HTML, which is basically done by XPath. Experienced students read it all understand:

The highlighted part of the figure is to download the actual address of the GPX file. We keep it in the urllist. At the same time, the metadata is stored in the JSON file.

folder = U'd:/buptzym Sync Disk/Baidu Cloud/My Documents/data analysis/datasets/rungps/'; cookies='jsessionid=69df607b71b1f14afec090f520b14b55; logincookie=5699607196$6098898d08e533587e82b33dd9d02196; persistent_cookie=5699607196$42c885ad38f59dca407e09c95be1a60b; uname_forloginform= "[email protected]"; __utma=54733311.82935663.1447906150.1447937410.1456907433.7; __utmb=54733311.5.10.1456907433; __utmc=54733311; __utmz=54733311.1456907433.7.3.utmcsr=baidu|utmccn= (organic) |utmcmd=organic; Cookie_site=auto'userid='5699607196'; F= Codecs.open (folder +'desert.htm','R','Utf-8'); HTML=F.read (); F.close (); root=etree. HTML (HTML) tree=etree. ElementTree (root); ListNode=tree.xpath ('//*[@id = "Feedlist"]'); Numre=re.compile (U'Cycling | running | km |,| time consuming | consumption | kcal'); Urllists=[]records=[]; forChildinchListnode[0].iterchildren (): Record={}; Temp=child.xpath ('Div[2]/div[1]/a[2]')    ifLen (temp) = =0:Continue; SOURCE= temp[0].attrib['href']; record['ID']=source.split ('/') [-1]; Info=Temp[0].text; Numinfo=Numre.split (info); ifLen (numinfo) <6:        Continue; record['type']= Info[0:2]; record['Distance']= numinfo[1]; record[' Hot']=numinfo[6]; Urllists.append ('http://edooon.com/user/%s/record/export?type=gpx&id=%s'% (UserID, record['ID']));

It is important to note that because a cookie is required for downloading, the reader needs to replace the UserID and the cookie that is logged in with the beneficial GPS.

The next step is to download the process, get the XPath of the URL of the Export Data button, construct a request with a cookie, and then save the file, which is very easy.

Opener =Urllib.request.build_opener () opener.addheaders.append ('Cookies', cookies)); Path='//*[@id = "Exportlist"]/li[1]/a'; forEveryurlinchUrllists:id= Everyurl.split ('=') [-1]; Print(ID); URL='http://edooon.com/user/%s/record/%s'%(userid, id); F=opener.open (URL); HTML=F.read ();    F.close (); Root=etree. HTML (HTML) tree=etree.    ElementTree (root); FS=Str (Tree.xpath (path) [0]); ifFs isNone:Continue; Furl='http://edooon.com/user/%s/record/%s'%(userid, FS); F=Opener.open (Furl); HTML=F.read ();    F.close (); FileName=folder+'ID'+'. GPX'; XMLFile= Codecs.open (filename,'WB');    Xmlfile.write (HTML); Xmlfile.close ();

After that, we saved about 300 more GPX files.

3. Parse GPX Data

The so-called GPX data, is a general specification of the GPS data format, detailed information can be self-search.

We need to use Python's GPX parser, Gpxpy is a good choice to use

PIP3 Install Gpxpy can be installed.

GPXPY provides a rich interface, of course, for statistics, we only need to extract a subset of the data:

defREADGPX (x): File= Open (dir+x+'. GPX','R') txt=file.read () GPX=gpxpy.parse (TXT) MV=gpx.get_moving_data () Dat= {'Move Time': Mv.moving_time,'Standstill Time': Mv.stopped_time,'Moving Distance': Mv.moving_distance,'Pause Distance': Mv.stopped_distance,'Max speed': Mv.max_speed}; dat['Total Time']=(Gpx.get_duration ()) dat['ID']=str (x) UpDown=Gpx.get_uphill_downhill () dat['Uphill']=(Updown.uphill); dat['Downhill']=(Updown.downhill) timebound=gpx.get_time_bounds (); dat['Start Time']=(timebound.start_time) dat['End Time']=(timebound.end_time) p=gpx.get_points_data () [0] dat['lat']=p.point.latitude dat['LNG']=p.point.longitude file.close ()returnDat

The READGPX function reads the file name x and returns a dictionary. And get a table similar to the following:

Because we only need to draw the area of Beijing, a coordinate expression is needed to sift out areas outside Beijing. The filter code uses pandas and has more detailed code in the attachment.

exceptids= detailed [(Detailed. lng<116.1) | ( Detailed. lng>116.7) | (detailed. lat<39.9) | (detailed. lat>40.1)]. Id

 def   Filtercity (r): SP  =r.split ( Span style= "color: #800000;" > '      )  if  sp[1]!=   GPX    :  return      False;  if  sp[0] in   exceptids.values:  return   False;  return  True; 

 for inch if filtercity (R)]

In this way, we have screened out all the sports data we have done in Beijing.

4. Mapping GPS data

Repeatedly build the wheel is not fun, draw GPX has a relatively powerful library, address in http://avtanski.net/projects/gps/

Unfortunately, this library uses Perl as the development language and uses GD as the visual rendering library. I spent a lot of time on installing GD above.

Ubuntu default installation Perl, GD is need LIBGD, LIBGD is very difficult to download on the official web, download but found the wrong version, which let me in the foreign internet for several hours, are dying ... At the end of the day, I realized that installing the LIBGD library would be as follows:

Apt-get Install Libgd-gd2-perl

I think this is the Apt-get way pit Daddy, apt get gd or libgd can not find, if not to check, who know so write Ah! As for the Perl cpan management tool, hey, don't say all is tears.

Next download GD 2.56, is a very new version. Find a variety of Chinese version of the installation procedures, found that there are problems. This kind of thing, the best way is still to see readme.md Ah!

After unpacking, Perl./makefile.pl

After make

Make install

Then you can ...

This GPX Drawing library introduces itself this way:

This folder contains several Perl scripts for processing and PLOTTINGGPS track data in. GPX format.

Its readme has a lot of instructions on the use, of course, we do not nonsense, to copy all the GPX data into the SAMPLE_GPX folder, and then the gorgeous run

./runme.sh

If there is no problem, it should be the following:

I assume that your readers are already familiar with bash, and modify the runme.sh file to see more options.

The final results are as follows:

When I saw the result, I was shocked! This is the result of running 2000 km or so, the main road in the third ring of Beijing (mainly concentrated in Changan Street North) has traveled. In particular, the North third ring and North Tucheng Road (line Line 10 north section) by my various abuses. Each section of the white line is a story, every point is one of my footprints Ah!

5. Summary

This article is clearly not detailed enough, far from hand by hand. And there's no more data analysis (obviously I've done all of this) but I believe that the running programmer must be very powerful, and I'm going to give it a go.

In fact, can be made a Web service, run friends upload their own running software ID, you can automatically render a variety of beautiful running path and analysis diagram, it should be very meaningful!

It took me seven or eight hours to vomit, and a lot of time was spent on how to install GD instead of downloading data. Lesson tells me, be sure to read the documentation in the installation package, because the version between the library and the library is different, it may cause the version of Hell, the new version will not uninstall, the old version can not be used when not to mention I did not remind Ah!

It is worth mentioning that the benefit of the GPS download GPX file without newline characters, which caused the Gpx_disualization library cannot parse it (the goods regular expression is wrong), I do not bother to move Perl regular, so replaced by the addition of newline characters.

GD also need to libpng and other Perl libraries, in the attachment are available to download.

The attachment is the Python3 code for the GD library and crawls all GPX data.

Use Python and Perl to draw the Beijing run map