Python Simple Crawl data

Failure, even with the Firefox see headers, parameters are not exactly the same, crawling out of the page, but it is not to give data, try to disable JS, and then see the cookie (JS is no cookie), with this cookie crawl, or not, separated by time to see , the contents of the cookie have not changed, a little frustrated, but still sent out, but also to save themselves a small task what

If there is a big guy passing by, still look at the liberal enlighten

Another two Web site scripts can be used, and will directly put down the code, the process will not say

Target site http://www.geomag.bgs.ac.uk/data_service/models_compass/igrf_form.shtml

First, solve the conversion of date to decimal years, taking into account only the rough conversion of the day

defDate2dy (year, Month, day): Months= [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31] Oneyear= 365ifyear%100 = =0:ifyear%400 = =0:months[1] = 29oneyear= 366Else:        ifyear%4 = =0:months[1] = 29oneyear= 366 Days=0 I= 1 whileI <month:days= days +Months[i] I= i + 1 Days= days + day-1returnYear + days/366

The first small goal is to capture the 2016.12.1 data.

Open Firefox's F12 and switch to the network section

Submit data to get

Useful information is the request header, request URL and parameters, chop down to throw into the program inside try

This piece I tried for about a day more, can't catch down, I am good food. jpg

Put the code down. First, in case there is a big guy after the hope of the liberal enlighten

#!usr/bin/pythonImportRequestsImportSysweb_url= R'http://www.geomag.bgs.ac.uk/data_service/models_compass/igrf_form.shtml'Request_url= R'Http://www.geomag.bgs.ac.uk/cgi-bin/igrfsynth'filepath= Sys.path[0] +'\\data_igrf_raw_'+'. html'FID= Open (filepath,'W', encoding='Utf-8') Headers= {    'Host':'www.geomag.bgs.ac.uk',    'user-agent':'mozilla/5.0 (Windows NT 6.1; rv:53.0) gecko/20100101 firefox/53.0',    'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',    'Accept-language':'zh-cn,zh;q=0.8,en-us;q=0.5,en;q=0.3',    'accept-encoding':'gzip, deflate',    'Content-type':'application/x-www-form-urlencoded',    'Content-length':'136',    'Referer':'http://www.geomag.bgs.ac.uk/data_service/models_compass/igrf_form.shtml',    'Connection':'keep-alive',    'upgrade-insecure-requests':'1'}payload= {    'name':'-',#your name and email address    'coord':'1',#' 1 ': Geodetic ' 2 ': geocentic    'Date':'2016.92',#decimal Years    'alt':' Max',#Altitude    ' Place':"',    'Degmin':'y',#Position coordinates: ' Y ': in Degrees and Minutes ' n ': in Decimal Degrees    'latd':' -',#latitude degrees (degrees negative for south)    'LATM':'0',#Latitude minutes    'Lond':' -',#longitude degrees (degrees negative for West)    'lonm':'0',#Longitude minutes    'tot':'y',#Total Intensity (F)    'Dec':'y',#declination (D)    'Inc':'y',#inclination (I)    'Hor':'y',#horizontal Intensity (H)    'nor':'y',#North Component (X)    'EAS':'y',#East Component (Y)    'ver':'y',#Vertical Component (Z)    'Map':'0',#Include A MAP of the location: ' 0 ': NO ' 1 ': YES    'SV':'N'}#if required secular variation (rate of change), plus ' SV ': ' Y 'R = Requests.post (Request_url, Data=payload, headers=headers) fid.write (R.text) fid.close ();

Python Simple Crawl data