Statistical learning methods-Logistic regression notes and Python source code

Logical regression (logistic regression)

Logistic regression is a classical classification method in statistical learning. It is more used on the two classification {0,1} issue.

Definition 1:

Set X is a continuous random variable, and x obeys the logistic regression distribution refers to the following distribution function and density function for x:

The distribution function belongs to the logistic function and its shape is an S-shaped curve.

Definition 2:

The two logistic regression models are the following conditional probability distributions:

As can be seen from the above, logistic regression is normalized to linear regression, and the output range is specified in {0,1}.

Now, the characteristics of logistic regression, probability , refers to the ratio of the probability of occurrence of an event to the probability of not occurring. To find the logarithm of the above equation, we can get the following formula.

This means that in a logistic regression model, the logarithmic probability of the output y=1 is a linear function of the input x.

The value of the linear function w*x of the input x by line classification is the real field. A logistic regression model allows you to convert a linear function into a probability,

This means that the closer the linear function is to the positive infinity, the closer the probability is to 1, the closer the value of the linear function is to the negative infinity, the closer the probability is to 0. Such a model is called a logistic regression model.

Loss function:

as in the Perceptual machines section, we need to construct the loss function and update the weight parameters. We use the maximum likelihood estimation method to estimate the model parameters, i.e. W. The maximum likelihood estimation method is the most probable condition (condition i.e. model parameter) that the result is already known and then seeks to make the result tenable.

Likelihood function:

Logarithmic likelihood function:

In this way, we have a loss function, where we simply maximize the function, and ask for the maximum value of W.

Optimization Solution:

Gradient Descent method

always change in the negative direction until the minimum value is found. In junior maths, the derivation of a function can get a letter

number at a certain point of the slope K (indicating the growth rate of the function, changing in the positive direction) if I they take the slope minus

- k, then the rate of growth in the negative direction is obtained. In this case, since we are going to make the logarithmic likelihood function substantially, the

You don't need to add a minus sign here.

To update a formula:

Among them, Alpha is the learning rate.

Python Source:

#coding =utf-8#author=altmanimport NumPy as Npimport Matplotlib.pyplot as Pltdef loaddata (): train_x = [] train_y = [] Filein = open (' data.txt ') for line in Filein.readlines (): Linearr = Line.strip (). Split () t Rain_x.append ([1.0, Float (linearr[0]), float (linearr[1])]) train_y.append (float (linearr[2])) train_x = np.ar Ray (train_x) train_y = Np.array (train_y). T return train_x,train_ydef sigmod (x): Return 1.0/(1.0+np.exp (-X)) def train (matrix,labels): size = matrix.shape[1 ] W = np.ones (size) while true:x = Np.dot (matrix,w) y = sigmod (x) diff = labels-y tmp w = w + 0.01*np.dot (matrix.            T,diff) diff2 = (tmpw-w) **2 sum_diff2 = SUM (diff2) sq = sum_diff2**0.5 If sq < 0.001: Break else:w = TMPW return wdef Test (matrix,labels,w): x = Np.dot (matrix,w) y = sigmod (x Error = 0.0 for i,result in Enumerate (y): if result > 0.5: Predict = 1.0 if predict! = Labels[i]: Error +=1 else:predict = 0.0 if predict! = Labels[i]: Error +=1 print ("Error rate:%3.2f"% (error/100.0)) def show (Data,labels, W): x1=[] y1=[] x2=[] y2=[] for I in range (len (labels)): if labels[i] = = 0:x1.append (dat a[i,1]) y1.append (data[i,2]) else:x2.append (data[i,1]) y2.append (data[i,2)) PL T.scatter (x1,y1,edgecolors= ' R ') Plt.scatter (x2,y2,edgecolors= ' k ') max_x = (Np.max (data[:,1])) min_x = (np.min (dat a[:,1]) y_min_x = float (-w[0]-w[1] * min_x)/w[2] y_max_x = float (-w[0]-w[1] * max_x)/w[2] Plt.plot ([min_ X, max_x], [y_min_x, y_max_x], '-G ') Plt.show () def main (): Matrix,labels = LoadData () weights = Train (Matrix,labe LS) test (matrix,labels,weights) show (matrix,labels,weights) if __name__ = = ' __main__ ': Main ()

Experimental result diagram:

Experimental data set:

-0.01761214.0530640-1.3956344.6625411-0.7521576.5386200-1.3223717.15285300.42336311.05467700.4067047.06733510.66739412.74 14520-2.4601506.86680510.5694119.5487550-0.02663210.42774300.8504336.92033411.34718313.17550001.1768133.1670201-1.7818719 .0979530-0.5666065.74900310.9316351.5895051-0.0242056.1518231-0.0364532.6909881-0.1969490.44416511.0144595.75439911.98529 83.2306191-1.693453-0.5575401-0.57652511.7789220-0.346811-1.6787301-2.1244842.67247111.2179169.5970150-0.7339289.0986870 -3.642001-1.61808710.3159853.52395311.4166149.6192320-0.3863233.98928610.5569218.29498411.22486311.5873600-1.347803-2.406 05111.1966044.95185110.2752219.54364700.4705759.3324880-1.8895679.5426620-1.52789312.1505790-1.18524711.3093180-0.4456783 .29730311.0422226.1051551-0.61878710.32098601.1520830.54846710.8285342.6760451-1.23772810.5490330-0.683565-2.16612510.229 4565.9219381-0.95988511.55533600.49291110.99332400.1849928.7214880-0.35571510.3259760-0.3978228.05839700.82483913.7303430 1.5072785.02786610.0996716.8358391-0.34400810.71748501.7859287.7186451-0.91880111.5602170-0.3640094.7473001-0.8417224.11908310.4904261.9605391-0.0071949.0 7579200.35610712.44786300.34257812.2811620-0.810823-1.46601812.5307776.47680111.29668311.60755900.47548712.0400350-0.7832 7711.00972500.07479811.0236500-1.3374720.4683391-0.10278113.7636510-0.1473242.87484610.5183899.88703501.0153997.5718820-1 .658086-0.02725511.3199442.17122812.0562165.0199811-0.8516334.3756911-1.5100476.0619920-1.076637-3.18188811.82109610.2839 9003.0101508.4017661-1.0994581.6882741-0.834872-1.7338691-0.8466373.84907511.40010212.62878101.7528425.46816610.0785570.0 5973610.089392-0.71530011.82566212.69380800.1974459.74463800.1261170.9223111-0.6797971.22053010.6779832.55666610.76134910 .6938620-2.1687910.14363211.3886109.34199700.31702914.7390250

Statistical learning methods-Logistic regression notes and Python source code