The way to the financial quantization of the Python-1-simple data processing and drawing

The first step in financial quantification: data statistics and analysis.

The textbook I chose is: Data analysis using Python O ' Reilly Publishing

Practical cases

1. Processing of 1.usa.gov data from bit.ly.

1) Data: http://www.usa.gov/About/developer-resources/1usagov.shtml

This data is a common JSON format

2) Convert JSON into a dictionary

Note: I saved the data in TXT format to be processed locally. The delimiter needs to be removed, and the characters need to be removed because there are BOM characters inside. These dictionaries are then read into the list.

Import OS
Import Json,pickle

From collections Import Defaultdict
From collections Import Counter

Records = [] forLineinchOpen"Haha6.txt", encoding ="UTF8"): Line= Line.strip ("\ n")        ifLine.startswith (U'\ufeff'): Line= Line.encode ('UTF8') [3:].decode ('UTF8')#Remove BOM charactersline = Json.loads (line, encoding ="Utf-8") Records.append (line)

Print (Records[0])

#output:
The first row of data is as follows: #{' u ': ' http://today.lbl.gov/2016/06/24/saudi-minister-of-energy-visits-lab-on-june-20/#main ',
# ' _id ': ' 27e6808c-3750-e5ac-002a-cfb577e72a48 ', ' r ': ' Direct ', ' SL ': ' 2963Ceb ', ' h ': ' 2963Ceb ',
# ' K ': ', ' a ': ' mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) applewebkit/537.36 (khtml ', ' C ': ' FR ',
# ' HC ': 1466804416, ' NK ': 0, ' ll ': [48.8582, 2.3387], ' g ': ' 2963Fqo ',
# ' t ': 1467187377, ' hh ': ' 1.usa.gov ', ' l ': ' Anonymous ', ' I ': ', ' tz ': ' Europe/paris '}

3) Find all the time zones and count them

Time_zones = [rec["TZ"] forRecinchRecordsif "TZ" inchRec]##时区统计, the statistics of the key of the dictionary element in the list#Method 1defget_counts (Sequence): Counts= {}     forXinchsequence:ifXinchCounts:counts[x]+ = 1Else: counts[x]= 1returncountscounts=get_counts (time_zones)Print(counts["america/new_york"])#Method 2defget_counts1 (Sequence): Counts=defaultdict (int) forXinchSequence:counts[x]+ = 1returncountscounts=get_counts1 (time_zones)Print(counts["america/new_york"])

#output: 353

4) Remove the first ten time zones and their count values

# Method 1 def top_counts (count_dict, n = ten):    for in  count_dict.items ()]    value _key_pairs.sort ()    return value_key_pairs[-N:]print(top_counts ( Counts)# Method 2counts = Counter (time_zones) Counts.most_common (ten) Print (Counts.most_common (10))

5) Use Pandas to simplify, count the time zone, and give the Top ten bar chart

#use Pandas to count time zones fromPandasImportDataFrameImportPandas as PDImportNumPy as Npframe=DataFrame (Records)#print (frame)#tz_counts = frame["TZ"].value_counts ()#print (Tz_counts[:10])Clean_tz = frame["TZ"].fillna ("missing")#Missing value handlingClean_tz[clean_tz = =""] ="Unknown" #empty string processingTz_counts =clean_tz.value_counts ()Print(Tz_counts[:10].plot (kind ="Barh", rot=0))

#output是柱状图

The way to the financial quantization of the Python-1-simple data processing and drawing