Python is a simple tutorial for data analysis, and python uses data analysis
Recently, Analysis with Programming has joined Planet Python. As the first special blog of this website, I will share with you how to start data analysis using Python. The details are as follows:
Data Import
Import local or web-side CSV files;
Data transformation;
Data Statistics description;
Hypothesis Test
One sample T-test;
Visualization;
Create a UDF.
Data Import
This is a key step. We need to import data for subsequent analysis. Generally, the data is in CSV format, and can be converted to at least CSV format, even if not. In Python, the operations are as follows:
import pandas as pd # Reading data locallydf = pd.read_csv('/Users/al-ahmadgaidasaad/Documents/d.csv') # Reading data from webdata_url = "https://raw.githubusercontent.com/alstat/Analysis-with-Programming/master/2014/Python/Numerical-Descriptions-of-the-Data/data.csv"df = pd.read_csv(data_url)
To read the local CSV file, we need the relevant modules in the pandas data analysis database. The read_csv function can read local and web data.
Data Transformation
Since there is data in the workspace, the next step is data transformation. Statisticians and scientists usually remove unnecessary data from the analysis at this step. Let's first look at the data:
# Head of the dataprint df.head() # OUTPUT Abra Apayao Benguet Ifugao Kalinga0 1243 2934 148 3300 105531 4158 9235 4287 8063 352572 1787 1922 1955 1074 45443 17152 14501 3536 19607 316874 1266 2385 2530 3315 8520 # Tail of the dataprint df.tail() # OUTPUT Abra Apayao Benguet Ifugao Kalinga74 2505 20878 3519 19737 1651375 60303 40065 7062 19422 6180876 6311 6756 3561 15910 2334977 13345 38902 2583 11096 6866378 2623 18264 3745 16787 16900
For R programmers, the above operations are equivalent to print (head (df) to print the first six rows of data, and print (tail (df )) to print the last six rows of data. Of course, in Python, 5 rows are printed by default, while R is 6 rows. Therefore, the R code head (df, n = 10) is df. head (n = 10) in Python, and the print data tail is the same.
In the R language, the names of Data columns and rows are extracted by colnames and rownames respectively. In Python, columns and index attributes are used for extraction, as shown below:
# Extracting column namesprint df.columns # OUTPUTIndex([u'Abra', u'Apayao', u'Benguet', u'Ifugao', u'Kalinga'], dtype='object') # Extracting row names or the indexprint df.index # OUTPUTInt64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78], dtype='int64')
The T method is used for data forwarding,
# Transpose dataprint df.T # OUTPUT 0 1 2 3 4 5 6 7 8 9 Abra 1243 4158 1787 17152 1266 5576 927 21540 1039 5424 Apayao 2934 9235 1922 14501 2385 7452 1099 17038 1382 10588 Benguet 148 4287 1955 3536 2530 771 2796 2463 2592 1064 Ifugao 3300 8063 1074 19607 3315 13134 5134 14226 6842 13828 Kalinga 10553 35257 4544 31687 8520 28252 3106 36238 4973 40140 ... 69 70 71 72 73 74 75 76 77 Abra ... 12763 2470 59094 6209 13316 2505 60303 6311 13345 Apayao ... 37625 19532 35126 6335 38613 20878 40065 6756 38902 Benguet ... 2354 4045 5987 3530 2585 3519 7062 3561 2583 Ifugao ... 9838 17125 18940 15560 7746 19737 19422 15910 11096 Kalinga ... 65782 15279 52437 24385 66148 16513 61808 23349 68663 78 Abra 2623 Apayao 18264 Benguet 3745 Ifugao 16787 Kalinga 16900
Other transformations, such as sorting, use the sort attribute. Now we extract data from a specific column. In Python, iloc or ix attributes can be used. But I prefer ix because it is more stable. Suppose we need the first five rows in the first column of the data, we have:
print df.ix[:, 0].head() # OUTPUT0 12431 41582 17873 171524 1266Name: Abra, dtype: int64
By the way, Python indexes start from 0 rather than 1. To retrieve the first three columns of data from 11 to 20 rows, we have:
print df.ix[10:20, 0:3] # OUTPUT Abra Apayao Benguet10 981 1311 256011 27366 15093 303912 1100 1701 238213 7212 11001 108814 1048 1427 284715 25679 15661 294216 1055 2191 211917 5437 6461 73418 1029 1183 230219 23710 12222 259820 1091 2343 2654
The preceding command is equivalent to df. ix [, ['aba', 'ayao', 'benguet '].
To discard columns in data, column 1 (Apayao) and column 2 (Benguet) are used. The drop attribute is used as follows:
print df.drop(df.columns[[1, 2]], axis = 1).head() # OUTPUT Abra Ifugao Kalinga0 1243 3300 105531 4158 8063 352572 1787 1074 45443 17152 19607 316874 1266 3315 8520
The axis parameter tells the function whether to discard columns or rows. If axis is equal to 0, the row is discarded.
Statistical description
The next step is to describe the statistical characteristics of data through the describe attribute:
print df.describe() # OUTPUT Abra Apayao Benguet Ifugao Kalingacount 79.000000 79.000000 79.000000 79.000000 79.000000mean 12874.379747 16860.645570 3237.392405 12414.620253 30446.417722std 16746.466945 15448.153794 1588.536429 5034.282019 22245.707692min 927.000000 401.000000 148.000000 1074.000000 2346.00000025% 1524.000000 3435.500000 2328.000000 8205.000000 8601.50000050% 5790.000000 10588.000000 3202.000000 13044.000000 24494.00000075% 13330.500000 33289.000000 3918.500000 16099.500000 52510.500000max 60303.000000 54625.000000 8813.000000 21031.000000 68663.000000
Hypothesis Test
Python has a good statistical inference package. That is the stats in scipy. Ttest_1samp enables a single sample t-test. Therefore, if we want to test the average rice yield in the Abra column through the zero hypothesis, we assume that the average rice yield is 15000, And we have:
from scipy import stats as ss # Perform one sample t-test using 1500 as the true meanprint ss.ttest_1samp(a = df.ix[:, 'Abra'], popmean = 15000) # OUTPUT(-1.1281738488299586, 0.26270472069109496)
Returns the ancestor composed of the following values:
T: floating point or array type
T statistic
Prob: floating point or array type
Two-tailed p-value bilateral probability value
Through the above output, we can see that the P value is 0.267 much greater than α = 0.05, so there is no sufficient evidence that the average rice yield is not 150000. Apply this test to all variables, and assume that the mean value is 15000. We have:
print ss.ttest_1samp(a = df, popmean = 15000) # OUTPUT(array([ -1.12817385, 1.07053437, -65.81425599, -4.564575 , 6.17156198]), array([ 2.62704721e-01, 2.87680340e-01, 4.15643528e-70, 1.83764399e-05, 2.82461897e-08]))
The first array is the t statistic, and the second array is the corresponding P value.
Visualization
Python has many visualization modules, and the most popular one is the matpalotlib library. We can also select the bokeh and seaborn modules. In my previous blog post, I have explained the function of the box map module in the matplotlib library.
# Import the module for plottingimport matplotlib.pyplot as plt plt.show(df.plot(kind = 'box'))
Now, we can use the ggplot topic Integrated with R in the pandas module to beautify the chart. To use ggplot, we only need to add a line in the above Code,
import matplotlib.pyplot as pltpd.options.display.mpl_style = 'default' # Sets the plotting display theme to ggplot2df.plot(kind = 'box')
In this way, we can get the table as follows:
It is much simpler than the matplotlib. pyplot topic. However, in this blog, I prefer to introduce the seaborn module, which is a data visualization library. Therefore, we have:
# Import the seaborn libraryimport seaborn as sns # Do the boxplotplt.show(sns.boxplot(df, widths = 0.5, color = "pastel"))
More sexy box chart, continue to look down.
plt.show(sns.violinplot(df, widths = 0.5, color = "pastel"))
plt.show(sns.distplot(df.ix[:,2], rug = True, bins = 15))
with sns.axes_style("white"): plt.show(sns.jointplot(df.ix[:,1], df.ix[:,2], kind = "kde"))
plt.show(sns.lmplot("Benguet", "Ifugao", df))
Create a UDF
In Python, we use the def function to implement a custom function. For example, if we want to define a function that adds two numbers, we can:
def add_2int(x, y): return x + y print add_2int(2, 2) # OUTPUT4
By the way, indentation in Python is very important. Define the function scope through indentation, just like using braces {…} in the R Language {...} Same. Here is an example of our previous blog:
Generates 10 normal distribution samples, u = 3 and o.
Calculate x_bar and x_bar2 Based on the confidence level of 95%;
Repeat 100 times; then
Calculate the percentage of the confidence interval containing the real mean.
In Python, the program is as follows:
import numpy as npimport scipy.stats as ss def case(n = 10, mu = 3, sigma = np.sqrt(5), p = 0.025, rep = 100): m = np.zeros((rep, 4)) for i in range(rep): norm = np.random.normal(loc = mu, scale = sigma, size = n) xbar = np.mean(norm) low = xbar - ss.norm.ppf(q = 1 - p) * (sigma / np.sqrt(n)) up = xbar + ss.norm.ppf(q = 1 - p) * (sigma / np.sqrt(n)) if (mu > low) & (mu < up): rem = 1 else: rem = 0 m[i, :] = [xbar, low, up, rem] inside = np.sum(m[:, 3]) per = inside / rep desc = "There are " + str(inside) + " confidence intervals that contain " "the true mean (" + str(mu) + "), that is " + str(per) + " percent of the total CIs" return {"Matrix": m, "Decision": desc}
The above code is easy to read, but the cycle is slow. The above code is improved below, thanks to Python experts.
import numpy as npimport scipy.stats as ss def case2(n = 10, mu = 3, sigma = np.sqrt(5), p = 0.025, rep = 100): scaled_crit = ss.norm.ppf(q = 1 - p) * (sigma / np.sqrt(n)) norm = np.random.normal(loc = mu, scale = sigma, size = (rep, n)) xbar = norm.mean(1) low = xbar - scaled_crit up = xbar + scaled_crit rem = (mu > low) & (mu < up) m = np.c_[xbar, low, up, rem] inside = np.sum(m[:, 3]) per = inside / rep desc = "There are " + str(inside) + " confidence intervals that contain " "the true mean (" + str(mu) + "), that is " + str(per) + " percent of the total CIs" return {"Matrix": m, "Decision": desc}