Introduction to the second chapter, "Data analysis using Python" study notes _1

"Example 1" 1.usa.gov data from bit.ly

1. File Location Description:

Forward slash \ Backslash \ mixed with all can. such as path= "D:/python/ch01.txt"

First of all, the problem encountered is pycharm Chinese coding problem, note ideencoding changed to Utf-8, while the file is the first to add #-*-encoding:utf-8-*-, while containing Chinese strings remember plus u.

2, the file read the first line:

Open (Path). ReadLine ()

3, JSON, a Web data format, Python has many modules to convert the JSON string into a python dictionary, usage:

Import JSON

Path= "Ch01.txt"

Records=[json.load (line) for line in open (path)]

At this point records becomes a set of Python dictionaries.

Show Dictionary first line : Records[0]

The index of Python starts at 0. Given a string representation of a key can be recorded. Example: records[0][' TZ ')

4, in the Ipython, drawing, should be opened in Pylab mode to see the picture. Enter in Ipython :%pylab or add a sentence plt.show ()

5. Dataframe is the most important data structure in pandas, which is used to represent data as a table. The command to create a DataFrame from a dictionary is:frame=dataframe (Records), at which point the frame is a column row structure, and frame[' TZ ' is a series object that has one method of counting the values . tz_counts=frame[' TZ '].value_counts ()

After counting, we can plot according to the statistic value, but there are two kinds of problems in the record, no this field or the field is empty. Then we need to fill in the replacement value first. method: Replace the missing value (NA) with the Fillna function, and the unknown value (NULL) is replaced with a Boolean array index :

clean_tz=frame[' TZ '].fillna (' Missing ')

clean_tz[clean_tz== ']= ' unkown '

Tz_counts=clean_tz.value_counts ()

5, Pandas missing value Supplementary tutorial , reference: https://jingyan.baidu.com/article/93f9803fe29aece0e46f55cb.html

6. The Tz_counts object is obtained after the data is populated. The object has a plot method that can be used to draw.

Tz_counts (:). Plot (kind= ' Barh ', rot=0)

7, Next, use string and expression to do some data processing work .

Results=series ([X.split () [0] for X in Frame.a.dropna ()])

#对于一个 Series,dropna Returns a Series that contains only non-empty data and index values

Remove the missing field first: Cframe=frame[frame.a.notnull ()]

Second, it calculates whether the rows are Windows based on the value of a, #np. The WHERE function is a vectorization ifelse function

Operating_system=np.where (cframe[' a '].str.contains (' windows '), ' windows ', ' no windows ')

Next, the data is grouped according to the time zone and operating system list: By_tz_os=cframe.groupby (' tz ', Operating_system)

The grouped results are then counted by size (similar to value_counts), and the count results are reshaped using unstack.

#下面是将tz按照operating_system进行分组并计数并用unstack进行展开并填充na为0

Agg_counts=by_tz_os.size (). Unstack (). Fillna (0)

AGG_COUNTS[:10]

Finally, we select the most frequently occurring time zone and construct an indirect indexed array based on the number of rows in agg_counts:

#下面注意 the SUM function by default axis = 0, which is normal Plus, Axis = 1 is in line Plus, Argsort is from small to large and returns to the following table

Indexer=agg_counts.sum (1). Argsort ()

INDEXER[:10]

Intercept the last 10 lines by take

Count_subset=agg_counts.take (indexer) [-10:]

Count_subset.plot (kind= ' Barh ', stacker=true)

Plt.show () #下面进行比例展示 normed_subset = Count_subset.div (count_subset.sum (1), Axis = 0) normed_subset.plot (kind = ' Barh ', stacked = True) plt.show ()

Introduction to the second chapter of data analysis using Python learning note _1