Python data aggregation and grouping operations (1)-groupby mechanics

Objective

Python's Pandas package provides a powerful and flexible data aggregation and grouping operation. The 9th chapter of the Python for Data analysis describes the usage in detail, but some of the details are easy to forget, so I intend to summarize this part of the book in a blog for review. According to the chapters in the book, this part of the knowledge includes the following four parts:

1.GroupBy Mechanics (GroupBy technology)

2.Data Aggregation (Data aggregation)

3.group-wise operation and Transformation (Group-level operations and transformations)

4.Pivot Tables and cross-tabulation (pivot table and cross table)

This article is the first part, introduces GroupBy technology.

First, the principle of grouping

Core:

1. Regardless of the grouping key is an array, a list, a dictionary, a Series, a function, as long as it is consistent with the length of the axis of the variable to be grouped can be passed into groupby for grouping.

2. The default axis=0 is grouped by rows, and you can specify Axis=1 to group columns.

The process of grouping data can be summarized as: Split-apply-combine three steps:

1. Group the data by key value (key) or grouping variable.

2. For each group to apply our functions, this step is very flexible, can be Python's own function, can be our own writing function.

3. Aggregate the results after the function is evaluated.

Figure 1: The Principle of grouping aggregation (image from Python for Data Analysis page 252)

ImportPandas as PDImportNumPy as Npdf= PD. DataFrame ({'Key1': ['a','a','b','b','a'],    'Key2': [' One',' Both',' One',' Both',' One'],    'data1': Np.random.randn (5),    'data2': Np.random.randn (5)})

We use Key1 as our grouping key value, group the data1, and then we ask for the mean value of each group:

grouped = df['data1'].groupby (df['key1'])

The syntax is simple, but here you need to pay attention to the grouped data type, which is not a data frame, but a GroupBy object.

Grouped

In fact, in this step, we did not perform any calculations just to create a GroupBy object after the Key1 group was created, and any operation of the function behind us is based on this object.

Mean value:

Grouped.mean ()

Just now we've just used Key1 to group, we can also use two grouping variables, and the results are reshaped by the Unstack method:

means = df['data1'].groupby ([df['key1'], df['  Key2']). Mean ()

Means

Means.unstack

All of our grouping variables are series within DF, in fact only arrays with key1 and so on can:

states = Np.array (['Ohio'California ' California'Ohio'Ohio '  = Np.array ([2005, 2005, 2006, 2005, 2006]) df['data1'].groupby ([ States, years]). Mean ()

Second, the grouping to iterate

The GroupBy object supports iterative operations, resulting in a two-tuple tuple of grouping variable names and data blocks:

 for  in Df.groupby ('key1'):    print  name      Print Group

If there are two groups of variables:

 for  in Df.groupby (['key1','key2') ):    print  k1,k2    Print Group

We can turn the above results into a list or dict to see what the results look like:

List (Df.groupby (['key1','key2'))

Look not very clear, let's take a look at the first element of this list:

List (Df.groupby (['key1','key2')]) [0]

Similarly, we can convert the results to Dict (dictionary):

Dict (List (Df.groupby (['key1','key2']))

Dict (List (Df.groupby (['key1','key2'])) [('  a','one')]

These are grouped based on rows, because by default groupby is grouped in axis=0 direction (row direction), we can specify Axis=1 direction (column direction) to group:

Grouped=df.groupby (Df.dtypes,axis=1) list (grouped) [0]

Dict (list (grouped))

Attention

" " The following two-paragraph statement functions as " " df.groupby ('key1') ['data1']df.data1.groupby (df.key1)

Iii. grouping by means of a dictionary

People = PD. DataFrame (Np.random.randn (5, 5), Columns=['a','b','C','D','e'], index=['Joe','Steve','Wes','Jim','Travis']) people.ix[2:3, ['b','C']] = Np.nan#Add missing valuePeople

If we want to aggregate by column, what should we do?

We create a dictionary of column names based on the actual situation, and then pass this dictionary to GroupBy, remembering to specify Axis=1, because we are grouping aggregations on columns:

mapping = {'a':'Red','b':'Red','C':'Blue',     'D':'Blue','e':'Red','F':'Orange'}by_columns=people.groupby (Mapping,axis=1) By_columns.mean ()

Now that we can group the columns by passing in the dictionary, we can certainly also group the columns by passing in the series (the index in the series is the key in the dictionary):

map_series = PD. Series (mapping) people.groupby (Map_series,axis=1). Count ()

Iv. Grouping by functions

Just when we grouped with Dict and series to build the map, for some complex requirements, we can directly to the GroupBy function transfer function name to group, take just the people data for example, if we want to group by row, group key is the letter length of each person's name, how to do? The more straightforward idea is to ask for a length relative to each name, create an array, and then pass this array to GroupBy, and we'll try it out:

 for inch People.index]people.groupby (L). Count ()

If the scheme is feasible, is there a quicker and more graceful way? Of course, we just have to pass Len this function name to GroupBy:

People.groupby (len). Count ()

In addition to the transfer function, we can also use the function with the Dict,series,array, after all, it will all be converted to an array:

Key_list = [' One ' one ' one '  'both 'and ']people.groupby ([Len, Key_list]). Min ()

V. Grouping by index level

Just now our data index has only one level, when the data has a multilevel index, you can specify the index we want to group by the level, note to use Axis=1 to represent by column:

columns = PD. Multiindex.from_arrays ([['Asian','Asian','Asian','America','America'],    [' China','Japan','Singapore','states','Canada']], names=['Continent','Country']) HIER_DF= PD. DataFrame (Np.random.randn (4, 5), columns=columns) HIER_DF

We group by continent and sum:

Python data aggregation and grouping operations (1)-groupby mechanics