A simple tutorial on using Python in data analysis

This article mainly introduces a simple tutorial on using Python for data analysis. it mainly introduces how to use Python for basic data analysis, such as data import, change, Statistics, and hypothesis testing, for more information, see the recent introduction of Analysis with Programming to Planet Python. As the first special blog of this website, I will share with you how to start data analysis using Python. The details are as follows:

Data import
Import local or web-side CSV files;
Data transformation;
Data Statistics description;
Hypothesis test
One sample t-test;
Visualization;
Create a UDF.

Data import

This is a key step. we need to import data for subsequent analysis. Generally, the data is in CSV format, and can be converted to at least CSV format, even if not. In Python, the operations are as follows:

import pandas as pd # Reading data locallydf = pd.read_csv('/Users/al-ahmadgaidasaad/Documents/d.csv') # Reading data from webdata_url = "https://raw.githubusercontent.com/alstat/Analysis-with-Programming/master/2014/Python/Numerical-Descriptions-of-the-Data/data.csv"df = pd.read_csv(data_url)

To read the local CSV file, we need the relevant modules in the pandas data analysis database. The read_csv function can read local and web data.

Data Transformation

Since there is data in the workspace, the next step is data transformation. Statisticians and scientists usually remove unnecessary data from the analysis at this step. Let's first look at the data:

# Head of the dataprint df.head() # OUTPUT Abra Apayao Benguet Ifugao Kalinga0 1243 2934  148 3300 105531 4158 9235  4287 8063 352572 1787 1922  1955 1074  45443 17152 14501  3536 19607 316874 1266 2385  2530 3315  8520 # Tail of the dataprint df.tail() # OUTPUT  Abra Apayao Benguet Ifugao Kalinga74 2505 20878  3519 19737 1651375 60303 40065  7062 19422 6180876 6311 6756  3561 15910 2334977 13345 38902  2583 11096 6866378 2623 18264  3745 16787 16900

For R programmers, the above operations are equivalent to print (head (df) to print the first six rows of data, and print (tail (df )) to print the last six rows of data. Of course, in Python, 5 rows are printed by default, while R is 6 rows. Therefore, the R code head (df, n = 10) is df. head (n = 10) in Python, and the print data tail is the same.

In the R language, the names of data columns and rows are extracted by colnames and rownames respectively. In Python, columns and index attributes are used for extraction, as shown below:

# Extracting column namesprint df.columns # OUTPUTIndex([u'Abra', u'Apayao', u'Benguet', u'Ifugao', u'Kalinga'], dtype='object') # Extracting row names or the indexprint df.index # OUTPUTInt64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78], dtype='int64')

The T method is used for data forwarding,

# Transpose dataprint df.T # OUTPUT   0  1  2  3  4  5  6  7  8  9 Abra  1243 4158 1787 17152 1266 5576 927 21540 1039 5424 Apayao 2934 9235 1922 14501 2385 7452 1099 17038 1382 10588 Benguet 148 4287 1955 3536 2530 771 2796 2463 2592 1064 Ifugao 3300 8063 1074 19607 3315 13134 5134 14226 6842 13828 Kalinga 10553 35257 4544 31687 8520 28252 3106 36238 4973 40140     ...  69  70  71  72  73  74  75  76  77 Abra  ... 12763 2470 59094 6209 13316 2505 60303 6311 13345 Apayao ... 37625 19532 35126 6335 38613 20878 40065 6756 38902 Benguet ...  2354 4045 5987 3530 2585 3519 7062 3561 2583 Ifugao ...  9838 17125 18940 15560 7746 19737 19422 15910 11096 Kalinga ... 65782 15279 52437 24385 66148 16513 61808 23349 68663     78 Abra  2623 Apayao 18264 Benguet 3745 Ifugao 16787 Kalinga 16900 

Other transformations, such as sorting, use the sort attribute. Now we extract data from a specific column. In Python, iloc or ix attributes can be used. But I prefer ix because it is more stable. Suppose we need the first five rows in the first column of the data, we have:

print df.ix[:, 0].head() # OUTPUT0  12431  41582  17873 171524  1266Name: Abra, dtype: int64

By the way, Python indexes start from 0 rather than 1. To retrieve the first three columns of data from 11 to 20 rows, we have:

print df.ix[10:20, 0:3] # OUTPUT Abra Apayao Benguet10 981 1311  256011 27366 15093  303912 1100 1701  238213 7212 11001  108814 1048 1427  284715 25679 15661  294216 1055 2191  211917 5437 6461  73418 1029 1183  230219 23710 12222  259820 1091 2343  2654

The preceding command is equivalent to df. ix [, ['aba', 'ayao', 'benguet '].

To discard columns in data, column 1 (Apayao) and column 2 (Benguet) are used. the drop attribute is used as follows:

print df.drop(df.columns[[1, 2]], axis = 1).head() # OUTPUT Abra Ifugao Kalinga0 1243 3300 105531 4158 8063 352572 1787 1074  45443 17152 19607 316874 1266 3315  8520

The axis parameter tells the function whether to discard columns or rows. If axis is equal to 0, the row is discarded.

Statistical Description

The next step is to describe the statistical characteristics of data through the describe attribute:

print df.describe() # OUTPUT    Abra  Apayao  Benguet  Ifugao  Kalingacount  79.000000  79.000000 79.000000  79.000000  79.000000mean 12874.379747 16860.645570 3237.392405 12414.620253 30446.417722std 16746.466945 15448.153794 1588.536429 5034.282019 22245.707692min  927.000000 401.000000 148.000000 1074.000000 2346.00000025%  1524.000000 3435.500000 2328.000000 8205.000000 8601.50000050%  5790.000000 10588.000000 3202.000000 13044.000000 24494.00000075% 13330.500000 33289.000000 3918.500000 16099.500000 52510.500000max 60303.000000 54625.000000 8813.000000 21031.000000 68663.000000

Hypothesis test

Python has a good statistical inference package. That is the stats in scipy. Ttest_1samp enables a single sample t-test. Therefore, if we want to test the average rice yield in the Abra column through the zero hypothesis, we assume that the average rice yield is 15000, and we have:

from scipy import stats as ss # Perform one sample t-test using 1500 as the true meanprint ss.ttest_1samp(a = df.ix[:, 'Abra'], popmean = 15000) # OUTPUT(-1.1281738488299586, 0.26270472069109496)

Returns the ancestor composed of the following values:

T: Floating point or array type
T statistic
Prob: Floating point or array type
Two-tailed p-value bilateral probability value

Through the above output, we can see that the p value is 0.267 much greater than α = 0.05, so there is no sufficient evidence that the average rice yield is not 150000. Apply this test to all variables, and assume that the mean value is 15000. we have:

print ss.ttest_1samp(a = df, popmean = 15000) # OUTPUT(array([ -1.12817385, 1.07053437, -65.81425599, -4.564575 , 6.17156198]), array([ 2.62704721e-01, 2.87680340e-01, 4.15643528e-70,   1.83764399e-05, 2.82461897e-08]))

The first array is the t statistic, and the second array is the corresponding p value.

Visualization

Python has many visualization modules, and the most popular one is the matpalotlib Library. We can also select the bokeh and seaborn modules. In my previous blog post, I have explained the function of the box map module in the matplotlib library.

# Import the module for plottingimport matplotlib.pyplot as plt plt.show(df.plot(kind = 'box'))

Now, we can use the ggplot topic integrated with R in the pandas module to beautify the chart. To use ggplot, we only need to add a line in the above code,

import matplotlib.pyplot as pltpd.options.display.mpl_style = 'default' # Sets the plotting display theme to ggplot2df.plot(kind = 'box')

In this way, we can get the table as follows:

It is much simpler than the matplotlib. pyplot topic. However, in this blog, I prefer to introduce the seaborn module, which is a data visualization library. Therefore, we have:

# Import the seaborn libraryimport seaborn as sns # Do the boxplotplt.show(sns.boxplot(df, widths = 0.5, color = "pastel"))

More sexy box chart, continue to look down.

plt.show(sns.violinplot(df, widths = 0.5, color = "pastel"))

plt.show(sns.distplot(df.ix[:,2], rug = True, bins = 15))

with sns.axes_style("white"): plt.show(sns.jointplot(df.ix[:,1], df.ix[:,2], kind = "kde"))

plt.show(sns.lmplot("Benguet", "Ifugao", df))

Create a UDF

In Python, we use the def function to implement a custom function. For example, if we want to define a function that adds two numbers, we can:

def add_2int(x, y): return x + y print add_2int(2, 2) # OUTPUT4

By the way, indentation in Python is very important. Define the function scope through indentation, just like using braces {…} in the R language {...} Same. Here is an example of our previous blog:

Generates 10 normal distribution samples, u = 3 and o.
Calculate x_bar and x_bar2 based on the confidence level of 95%;
Repeat 100 times; then
Calculate the percentage of the confidence interval containing the real mean.

In Python, the program is as follows:

import numpy as npimport scipy.stats as ss def case(n = 10, mu = 3, sigma = np.sqrt(5), p = 0.025, rep = 100): m = np.zeros((rep, 4))  for i in range(rep):  norm = np.random.normal(loc = mu, scale = sigma, size = n)  xbar = np.mean(norm)  low = xbar - ss.norm.ppf(q = 1 - p) * (sigma / np.sqrt(n))  up = xbar + ss.norm.ppf(q = 1 - p) * (sigma / np.sqrt(n))   if (mu > low) & (mu < up):   rem = 1  else:   rem = 0   m[i, :] = [xbar, low, up, rem]  inside = np.sum(m[:, 3]) per = inside / rep desc = "There are " + str(inside) + " confidence intervals that contain "   "the true mean (" + str(mu) + "), that is " + str(per) + " percent of the total CIs"  return {"Matrix": m, "Decision": desc}

The above code is easy to read, but the cycle is slow. The above code is improved below, thanks to Python experts.

import numpy as npimport scipy.stats as ss def case2(n = 10, mu = 3, sigma = np.sqrt(5), p = 0.025, rep = 100): scaled_crit = ss.norm.ppf(q = 1 - p) * (sigma / np.sqrt(n)) norm = np.random.normal(loc = mu, scale = sigma, size = (rep, n))  xbar = norm.mean(1) low = xbar - scaled_crit up = xbar + scaled_crit  rem = (mu > low) & (mu < up) m = np.c_[xbar, low, up, rem]  inside = np.sum(m[:, 3]) per = inside / rep desc = "There are " + str(inside) + " confidence intervals that contain "   "the true mean (" + str(mu) + "), that is " + str(per) + " percent of the total CIs" return {"Matrix": m, "Decision": desc}