Python Shipping Simple tutorials for data analysis

More recently, analysis with programming joined Planet Python. As the first special blog of the site, I'll share how to start data analysis with Python. The specific contents are as follows:

Data import
Import a local or web-side CSV file;
Data transformation;
Data statistical description;
Hypothesis Testing
Single sample t test;
visualization;
Create a custom function.

Data import

This is a critical step, and for subsequent analysis we need to import the data first. In general, the data is in CSV format, and even if not, it can be converted to CSV format at least. In Python, we do the following:

Import pandas as PD  # Reading Data LOCALLYDF = pd.read_csv ('/users/al-ahmadgaidasaad/documents/d.csv ')  # Reading data from webdata_url = "https://raw.githubusercontent.com/alstat/Analysis-with-Programming/master/2014/ Python/numerical-descriptions-of-the-data/data.csv "df = pd.read_csv (Data_url)

In order to read the local CSV file, we need to pandas the corresponding module in this data Analysis library. The Read_csv function can read both local and web data.

Data transformation

Now that there's data in the workspace, the next step is data transformation. Statisticians and scientists typically remove nonessential data from the analysis in this step. Let's look at the data first:

# Head of the Dataprint Df.head ()  # OUTPUT Abra Apayao Benguet Ifugao Kalinga0 1243 2934  148 3300 105531 4158 9235  4287 8063 352572 1787 1922  1955 1074  45443 17152 14501  3536 19607 316874 1266 2385 2530  3315  8520  # Tail of the Dataprint Df.tail ()  # OUTPUT  Abra Apayao benguet Ifugao Kalinga74 2505 20878  3519 19737 1 651375 60303 40065  7062 19422 6180876 6311 6756  3561 15910 2334977 13345 38902  2583 11096 6866378 2623 1826 4  3745 16787 16900

For R language programmers, this is equivalent to printing the first 6 rows of data through print (Head (DF)), and the last 6 lines of data through print (tail (DF)). Of course, in Python, the default print is 5 lines, while R is 6 rows. So the code Head of R (DF, n = 10) is df.head (n = 10) in Python, and the tail of the printed data is the same.

In the R language, the names of data columns and rows are extracted separately by colnames and Rownames. In Python, we use the columns and index properties to extract, as follows:

# extracting column Namesprint df.columns  # Outputindex ([u ' Abra ', U ' Apayao ', U ' Benguet ', U ' Ifugao ', U ' Kalinga '), Dtype= ' object ')  # extracting row names or the Indexprint df.index  # Outputint64index ([0, 1, 2, 3, 4, 5, 6, 7, 8, 9 , 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67 ,-----------------------------], dtype= ' Int64 '

Data transpose using the T method,

# transpose Dataprint DF. T  # OUTPUT   0  1  2  3  4  5  6  7  8  9Abra  1243 4158 1787 17152 1266 5576 927 21540 1039 5424Apayao 2934 9235 1922 14501 2385 7452 1099 17038 1382 10588Benguet 148 4287 1955 3536 2530 771 2796 246 3 2592 1064Ifugao 3300 8063 1074 19607 3315 13134 5134 14226 6842 13828Kalinga 10553 35257 4544 31687 8520 28252 3106 3623 8 4973 40140 ...  The  "  77Abra  ..." 12763 2470 59094 6209 13316 2505 60303 6311 13345Apayao ... 37625 19532 35126 6335 38613 20878 40065 6756 38902Benguet  ... 2354 4045 5987 3530 2585 3519 7062 3561 2583Ifugao  ... 9838 17125 18940 15560 7746 19737 19422 15910 11096Kalinga ... 65782 15279 52437 24385 66148 16513 61808 23349 68663     78Abra  2623Apayao 18264Benguet 3745Ifugao 16787Kalinga 169 00

Other transformations, such as sorting, are using the Sort property. Now we extract a specific column of data. In Python, you can use the Iloc or IX property. But I prefer IX because it is more stable. Let's say we need the first 5 rows of the data column, we have:

Print df.ix[:, 0].head ()  # OUTPUT0  12431  41582 17873 171524  1266name:abra, Dtype:int64

By the way, Python's index starts at 0 rather than 1. To take out the first 3 columns of data from 11 to 20 rows, we have:

Print df.ix[10:20, 0:3]  # OUTPUT Abra Apayao Benguet10 981 1311  256011 27366 15093 303912 1100 1701  238213 7212 11001  108814 1048 1427  284715 25679 15661  294216 1055 2191  211917 5437 6461  73418 1029 1183< c14/>230219 23710 12222  259820 1091 2343  2654

The above commands are equivalent to df.ix[10:20, [' Abra ', ' Apayao ', ' Benguet ']].

In order to discard the columns in the data, this is column 1 (Apayao) and column 2 (Benguet), we use the Drop property as follows:

Print Df.drop (df.columns[[1, 2]], Axis = 1). Head ()  # OUTPUT Abra Ifugao Kalinga0 1243 3300 105531 4158 8063 352572 17 1074  45443 17152 19607 316874 1266 3315  8520

The axis parameter tells the function to discard columns or rows. If axis equals 0, then the row is discarded.

Statistical description

The next step is to describe the statistical characteristics of the data through the describe attribute:

Print Df.describe ()  # OUTPUT    Abra  Apayao  benguet  Ifugao  kalingacount  79.000000  79.000000 79.000000  79.000000  79.000000mean 12874.379747 16860.645570 3237.392405 12414.620253 30446.417722STD 16746.466945 15448.153794 1588.536429 5034.282019 22245.707692min  927.000000 401.000000 148.000000 1074.000000 2346.00000025%  1524.000000 3435.500000 2328.000000 8205.000000 8601.50000050%  5790.000000 10588.000000 3202.000000 13044.000000 24494.00000075% 13330.500000 33289.000000 3918.500000 16099.500000 52510.500000max 60303.000000 54625.000000 8813.000000 21031.000000 68663.000000

Hypothesis Testing

Python has a good statistical inference package. That is the stats inside the scipy. The Ttest_1samp implements a single-sample t-Test. So, if we want to test the average grain yield of the data abra column, by 0 hypothesis, here we assume that the overall paddy yield is 15000, we have:

From scipy import stats as SS  # Perform One sample t-test using as the true Meanprint ss.ttest_1samp (a = df.ix[:, ' Abra '], Popmean = 15000)  # OUTPUT (-1.1281738488299586, 0.26270472069109496)

Returns a Ganso consisting of the following values:

T: floating-point or array type
T-Statistic
Prob: floating-point or array type
two-tailed p-value Two-sided probability value

With the above output, we see that the P-value is 0.267 far greater than α equals 0.05, so there is insufficient evidence that the average paddy yield is not 150000. Apply this test to all variables and also assume that the mean value is 15000, we have:

Print Ss.ttest_1samp (a = df, Popmean = 15000)  # OUTPUT (Array ([-1.12817385, 1.07053437,-65.81425599,-4.564575, 6.17 156198]), Array ([2.62704721e-01, 2.87680340e-01, 4.15643528e-70,   1.83764399e-05, 2.82461897e-08]))

The first array is the T statistic, and the second array is the corresponding P-value.

Visualization of

There are many visual modules in Python, the most popular being the Matpalotlib library. With a little mention, we can also choose the bokeh and Seaborn modules. In the previous blog post, I have explained the function of the box Whisker diagram module in the Matplotlib library.

# Import the module for Plottingimport Matplotlib.pyplot as Plt plt.show (df.plot (kind = ' box '))

Now we can beautify the chart with the Ggplot theme of integrated R in the Pandas module. To use Ggplot, we just need to add a line to the above code,

Import Matplotlib.pyplot as Pltpd.options.display.mpl_style = ' Default ' # Sets the plotting display theme to Ggplot2df.plo T (kind = ' box ')

So we get a table like this:

Much more concise than the Matplotlib.pyplot theme. But in this blog post, I prefer to introduce the Seaborn module, which is a statistical data visualization library. So we have:

# Import The Seaborn libraryimport Seaborn as SNS # do the Boxplotplt.show (Sns.boxplot (df, widths = 0.5, color = "pastel") )

How sexy the box chart, continue to look down.

Plt.show (Sns.violinplot (df, widths = 0.5, color = "pastel"))

Plt.show (Sns.distplot (df.ix[:,2], rug = True, bins = 15))

With Sns.axes_style ("white"): Plt.show (Sns.jointplot (df.ix[:,1], df.ix[:,2], kind = "KDE"))

Plt.show (Sns.lmplot ("Benguet", "Ifugao", DF))

Creating a Custom function

In Python, we use the DEF function to implement a custom function. For example, if we want to define a function that adds two numbers, as follows:

def add_2int (x, y): return x + y  print add_2int (2, 2)  # OUTPUT4

By the way, indentation in Python is important. Define the function scope by indenting, just like using curly braces {...} in the R language. The same. Here is an example of our previous blog post:

Produces 10 normal distribution samples, of which U=3 and O.
Calculates X_bar and X_BAR2 based on the 95% confidence level;
Repeat 100 times; And then
Calculates the percentage of the confidence interval that contains the true mean

In Python, the program is as follows:

Import NumPy as Npimport scipy.stats as SS  def case (n = ten, mu = 3, sigma = Np.sqrt (5), p = 0.025, rep = +): M = np. Zeros (Rep, 4)) for   I in range (rep):  norm = np.random.normal (loc = mu, scale = sigma, size = N)  Xbar = Np.mean (Norm)  Low = XBAR-SS.NORM.PPF (q = 1-p) * (SIGMA/NP.SQRT (n)) up  = Xbar + SS.NORM.PPF (q = 1-p) * (SIGMA/NP.SQRT (n)) 
  if (Mu > Low) & (Mu < up):   rem = 1  else:   rem = 0    m[i,:] = [Xbar, low, up, REM]   inside = Np.sum (m[:, 3]) per = Inside/rep desc = "There is" + str (inside) + "confidence intervals that contain"   "the T Rue mean ("+ str (MU) +"), that's "+ str (PER) +" percent of the total CIs "   return {" Matrix ": M," decision ": desc}

The code reads simple, but the loop is slow. The following improvements have been made to the above code, thanks to the Python expert.

Import NumPy as Npimport scipy.stats as SS  def case2 (n = ten, mu = 3, sigma = Np.sqrt (5), p = 0.025, rep = +): scaled _crit = SS.NORM.PPF (q = 1-p) * (SIGMA/NP.SQRT (n)) norm = np.random.normal (loc = mu, scale = sigma, size = (rep, n)) 
  xbar = Norm.mean (1) low = xbar-scaled_crit up = Xbar + scaled_crit   rem = (mu > Low) & (Mu < up) m = np.c _[xbar, low, up, REM]   inside = np.sum (m[:, 3]) per = Inside/rep desc = "There is" + str (inside) + "confidence int Ervals that contain "   " the True Mean ("+ str (MU) +"), which is "+ str (PER) +" percent of the total CIs "return {" Ma Trix ": M," decision ": desc}