Python's simple tutorial for data analysis _python

Recently, analysis and programming joined Planet Python. As the first of its special blogs, I'm here to share how to start data analysis through Python. The specific contents are as follows:

Data import
Import a local or web-side CSV file;
Data transformation;
Data statistics description;
Hypothesis Testing
Single sample t test;
visualization;
Create a custom function.

Data import

This is a critical step, and for subsequent analysis we first need to import the data. In general, the data is in CSV format, and if not, at least it can be converted to CSV format. In Python, our actions are as follows:

Import pandas as PD
 
# Reading data locally
df = pd.read_csv ('/users/al-ahmadgaidasaad/documents/d.csv ')
 
# Reading data from web
data_url = "https://raw.githubusercontent.com/alstat/Analysis-with-Programming/master/ 2014/python/numerical-descriptions-of-the-data/data.csv "
df = pd.read_csv (Data_url)

In order to read the local CSV file, we need to pandas the corresponding module in this data Analysis library. The Read_csv function can read local and web data.

Data transformation

Now that you have data in the workspace, the next step is the data transformation. Statisticians and scientists usually remove non-essential data from the analysis in this step. Let's look at the data first:

# Head of the data
print Df.head ()
 
# OUTPUT
 abra Apayao Benguet Ifugao Kalinga
0 1243 2934  148 3300 10 553
1 4158 9235  4287 8063 35257
2 1787 1922  1955 1074  4544
3 17152 14501  3536 19607 31687
4 1266 2385  2530 3315  8520
 
# Tail of the The Data
print Df.tail ()
 
# OUTPUT
  Abra Apayao be Nguet Ifugao Kalinga 2505 20878 3519 19737 16513 60303 40065 7062 19422 61808
76 6311 6756< 15910 23349 13345 38902 2583 11096 68663 c26/>3561
2623 18264  3745 16787 16900

For R-language programmers, the above operation is equivalent to printing the first 6 rows of data through print (Head (DF)) and the last 6 lines of data printed through print (tail (DF)). Of course python, the default print is 5 lines, and R is 6 rows. So R's code head (DF, n = 10) is df.head in Python (n = 10), and the same is true for print data tails.

In the R language, the names of data columns and rows are extracted separately by colnames and Rownames. In Python, we use the columns and index attributes to extract the following:

# Extracting column names
print df.columns
 
# OUTPUT
Index ([u ' Abra ', U ' Apayao ', U ' Benguet ', U ' Ifugao ', U ' Kalinga '], dtype= ' object ')
 
# extracting row names or the index
print df.index
 
# OUTPUT
Int64index ([0,  1, 2, 3, 4, 5.6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 34, 35 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60 4, A,,,,,,,,,,, and, in,,,,, and,,, and dtype= ' int64 '

Data transpose using the T method,

# Transpose Data
print DF. T
 
# OUTPUT
   0  1  2 3 4 5 6 7 8 9 
abra  1243 4158 1787 17152 1266 5576 927 21540 1039 5424 
Apayao 2934 9235 1922 14501 2385 7452 1099 17038 1382 10588 
Benguet 148 4287 1955 3536 2530 771 2796 2463 2592 1064 
Ifugao 3300 8063 1074 19607 3315 13134 5134 14226 6842 13828 
Kalinga 10553 352 4544 31687 8520 28252 3106 36238 4973 40140 
 
   ...  The 
Abra  of the "..." 12763 2470 59094 6209 13316, 2505 60303 6311 13345 
... 37625 19532 35126 6335 38613, 20878 40065 6756 38902 
...  2354 4045 5987 3530 2585, 3519 7062 3561 2583 
...  9838 17125 18940 15560 7746, 19737 19422 15910 11096 
... 65782 15279 52437 24385 66148 16513 61808 23349 68663 
abra  2623 
Apayao 18264 Benguet 3745< C42/>ifugao 16787 
Kalinga 16900 

Other transformations, such as sorting, are used with the sort attribute. Now we extract a particular column of data. Python, you can use either the Iloc or the IX attribute. But I prefer to use IX because it's more stable. Let's say we need the first 5 lines of data, we have:

Print df.ix[:, 0].head ()
 
# OUTPUT
0  1243
1  4158
2  1787
3 17152
4  1266
Name:abra, Dtype:int64

By the way, Python's index starts at 0 rather than 1. To remove the first 3 columns of data from 11 to 20 rows, we have:

Print df.ix[10:20, 0:3]
 
# OUTPUT
 abra Apayao Benguet
Ten 981 1311  2560-
27366 15093  3039
1100 1701  2382
7212 11001  1088  1048 1427 2847 25679 15661 2942
2191  2119
5437 6461 734 1055
1029 1183  2302  23710 12222 2598
1091 2343  2654

The above command is equivalent to df.ix[10:20, [' Abra ', ' Apayao ', ' Benguet ']].

To discard the columns in the data, this is column 1 (Apayao) and column 2 (Benguet), we use the Drop property as follows:

Print Df.drop (df.columns[[1, 2]], Axis = 1). Head ()
 
# OUTPUT
 Abra Ifugao Kalinga
0 1243 3300 10553
1 41 8063 35257
2 1787 1074  4544
3 17152 19607 31687
4 1266 3315  8520

The axis argument tells the function whether to discard columns or rows. If axis equals 0, then the row is discarded.

Statistical Description

The next step is to describe the statistical characteristics of the data through the describe attribute:

Print Df.describe ()
 
# OUTPUT
    abra  Apayao  benguet  Ifugao  Kalinga
Count  79.000000  79.000000 79.000000  79.000000  79.000000
mean 12874.379747 16860.645570 3237.392405 12414.620253 30446.417722
std 16746.466945 15448.153794 1588.536429 5034.282019
min  927.000000 401.000000 148.000000 1074.000000 2346.000000
25%  1524.000000 3435.500000 2328.000000 8205.000000 8601.500000
50%  5790.000000 10588.000000 3202.000000 13044.000000 24494.000000
75% 13330.500000 33289.000000 3918.500000 16099.500000 52510.500000
Max 60303.000000 54625.000000 8813.000000 21031.000000 68663.000000

Hypothesis Testing

Python has a good statistical inference package. That is the stats inside the scipy. Ttest_1samp realizes a single sample t test. So, if we want to test the average paddy yield of data Abra, by 0 hypothesis, here we assume that the total paddy yield is 15000, we have:

From scipy import stats as SS
 
# Perform one sample t-test using 1500 as the true mean
print ss.ttest_1samp (a = df. ix[:, ' Abra '], Popmean = 15000)
 
# OUTPUT
(-1.1281738488299586, 0.26270472069109496)

Returns the Ganso consisting of the following values:

T: floating point or array type
T Statistical quantity
Prob: floating-point or array type
two-tailed p-value Two-sided probability value

With the above output, we see that the P value is 0.267 far greater than alpha equals 0.05, so there is no sufficient evidence that the average paddy yield is not 150000. Apply this test to all variables, assuming that the mean value is 15000, we have:

Print Ss.ttest_1samp (a = df, Popmean = 15000)
 
# OUTPUT
(Array ([-1.12817385, 1.07053437,-65.81425599,-4.564575, 6.17156198]),
 array ([2.62704721e-01, 2.87680340e-01, 4.15643528e-70,
   1.83764399e-05, 2.82461897e-08])

The first array is the T statistic, and the second array is the corresponding P value.

Visualization of

Python has many visual modules, the most popular of which is the Matpalotlib library. With a little mention, we can also select Bokeh and Seaborn modules. In the previous blog post, I have explained the Matplotlib library in the box diagram module function.

# import the module for plotting
Import Matplotlib.pyplot as Plt
 plt.show (df.plot (kind = ' box '))

Now, we can use the Pandas module to integrate R's Ggplot theme to beautify the chart. To use Ggplot, we just need to add one more line to the code above,

Import Matplotlib.pyplot as plt
pd.options.display.mpl_style = ' Default ' # Sets the plotting display theme to ggplot2< C6/>df.plot (kind = ' box ')

So we get the following chart:

It's much simpler than the Matplotlib.pyplot theme. But in this blog post, I prefer to introduce the Seaborn module, which is a statistical data visualization library. So we have:

# import the Seaborn Library
import Seaborn as SNS
 # do the BoxPlot
plt.show (Sns.boxplot (df, widths = 0.5, C Olor = "pastel"))

What a sexy box chart, keep looking down.

Plt.show (Sns.violinplot (df, widths = 0.5, color = "pastel"))

Plt.show (Sns.distplot (df.ix[:,2], rug = True, bins = 15))

With Sns.axes_style ("white"):
 plt.show (Sns.jointplot (df.ix[:,1), df.ix[:,2], kind = "KDE")

Plt.show (Sns.lmplot ("Benguet", "Ifugao", DF))

Create a custom function

In Python, we use the DEF function to implement a custom function. For example, if we want to define a function that adds two numbers, it can be as follows:

def add_2int (x, y): return
 x + y
 
print add_2int (2, 2)
 
# OUTPUT
4

By the way, indenting in Python is important. Defining function scopes by indenting is like using curly braces {...} in the R language The same. Here's an example of our previous blog:

Produces 10 normal distribution samples, of which U=3 and O.
Calculate X_bar and x_bar2 based on 95% confidence level;
Repeat 100 times; And then
Calculate the percentage of the confidence interval that contains the true mean value

In Python, the program is as follows:

Import NumPy as NP
import scipy.stats as SS
 
def case (n = ten, mu = 3, sigma = Np.sqrt (5), p = 0.025, rep = m): 
   m = Np.zeros ((Rep, 4)) for
 
 I in range (rep):
  norm = np.random.normal (loc = mu, scale = sigma, size = N)
  Xbar = Np.mean (norm) Low
  = XBAR-SS.NORM.PPF (q = 1-p) * (SIGMA/NP.SQRT (n)) up
  = Xbar + SS.NORM.PPF (q = 1-p) * (s IGMA/NP.SQRT (n))
 
  if (Mu > Low) & (Mu < up):
   rem = 1
  else:
   rem = 0
 
  m[i,:] = [Xbar, Low , up, REM]
 
 inside = np.sum (m[:, 3]) per
 = Inside/rep
 desc = "There are" + str (inside) + "confidence int Ervals that contain "
   " "true Mean (" + str (MU) + "), which is ' + str (PER) + ' percent of the total CIs '
 
 retur n {"Matrix": M, "Decision": desc}

The above code is simple to read, but the loop is slow. The following improvements are made to the above code, thanks to the Python expert.

Import NumPy as NP
import scipy.stats as SS
 
def case2 (n = ten, mu = 3, sigma = Np.sqrt (5), p = 0.025, rep = m): 
   scaled_crit = SS.NORM.PPF (q = 1-p) * (SIGMA/NP.SQRT (n))
 norm = np.random.normal (loc = mu, scale = sigma, size = (Rep, N))
 
 Xbar = Norm.mean (1) Low
 = xbar-scaled_crit up
 = Xbar + scaled_crit
 
 rem = (mu > Low) & (Mu < up) 
   m = Np.c_[xbar, low, up, REM]
 
 inside = np.sum (m[:, 3]) per
 = inside/rep desc
 = "There are" + str (insid E) + "confidence intervals that contain"
   "true Mean (" + str (MU) + "), which is" + str (PER) + "percent of the Total CIs ' return
 {' Matrix ': M, ' Decision ': desc}