Machine Learning: Logistic regression

**************************************

Note: This blog series is for bloggers to learn the "machine learning" course notes from Professor Andrew Ng of Stanford University. Bloggers deeply learned the course, do not summarize is easy to forget, according to the course plus their own to do not understand the problem of the addition of this series of blogs. This blog series includes linear regression, logistic regression, neural network, machine learning application and system design, support vector machine, clustering, dimension, anomaly detection, recommender system and large scale machine learning.

**************************************

Logistic regression

category (Classi?cation)

Examples of classification problems:

Mail: spam/non-spam?

Online Trading: Is fraud (yes/no)?

Tumour: malignant/benign?

The above problem can be referred to as the two classification problem, we will be the dependent variable (dependant variable) may belong to the two classes are referred to as the negative class (negative class) and the forward class (positive class), the dependent variable y? {0,1}, where 0 represents a negative class and 1 represents a forward class.

For multi-classification problems, the dependent variable y:y∈{0,1,2,3 can be defined as follows,..., n}

If the classifier is a regression model, and you have trained a model, you can set a threshold value:

If hθ (x) ≥0.5, then the predicted Y=1, both y belongs to the positive example;

If hθ (x) is <0.5, then the y=0 is predicted, and y is a negative example;

However, for the two classification problem, the hypothesis output value of the linear regression model hθ (x) can be greater than 1 or less than 0. This time we elicit logistic regression, where the hypothesis output of the logistic regression is between 0 and 1, i.e.:

0≤hθ (x) ≤1

hypothesis expression (hypothesis representation)

As mentioned in the previous section, we need to define the output of hypothesis between 0 and 1, both: 0≤hθ (x) ≤1

But linear regression is not possible, here we introduce a function g, so that the hypothesis of logistic regression is represented as:

hθ (x) =g (ΘTX), where G is called the logistic function

For example, if hθ (x) = 0.7 is computed for a given x by the parameters that have been determined, then there is a 70% chance that Y is a positive class, and the probability of y being a negative class is 1-0.7=0.3.

Now suppose we have a model: hθ (x) =g (θ0+θ1x1+θ2x2), and the parameter θ is a vector [-3 1 1]. The model predicts Y=1 when -3+X1+X2 is greater than or equal to 0, that is, X1+X2 is greater than or equal to 3 o'clock.

We can draw the line x1+x2=3, which is the dividing line of our model, separating the area predicted to 1 and the area predicted as 0.

The above is just a linear decision boundary, and when hθ (x) is more complex, we can get a nonlinear decision boundary, for example:

costfunction (cost function)

For linear regression models, the cost function we define is the sum of squares of all model errors. Theoretically, we can also use this definition for a logistic regression model, but the problem is that when we

When brought into the cost function defined in this way, the cost function we get is a non-convex function (Non-convex functions).

This means that our cost function has many local minimums, which will affect the gradient descent algorithm to find the global minimum value. So we redefine the cost function of logistic regression:

The Cost (hθ (x), y) function of this construction is characterized by the fact that when the actual Y=1 and hθ also have a 1 o'clock error of 0, when Y=1 but hθ is not 1 o'clock, the error becomes larger with the hθ; when the actual y=0 and hθ are also 0 o'clock at 0, when y=0 but hθ is not an 0 o'clock error The difference becomes larger as the hθ becomes larger.

After obtaining such a cost function, we can use the gradient descent algorithm to find the parameters which can make the cost function least. The algorithm is:

Note that this algorithm is almost identical to the gradient descent algorithm in linear regression, except that the representation of hθ (x) is different.

In addition to the gradient descent algorithm, there are some algorithms that are often used to minimize the cost function, which are more complex and superior, and usually do not require manual selection of the learning rate, usually faster than the gradient descent algorithm. These algorithms are: Conjugate gradient (conjugate Gradient), local optimization method (Broyden Fletcher Goldfarb Shann,bfgs) and finite memory local optimization (LBFGS).

Multi-classclassification:one-vs-all (Multi-Class classification problem)

Examples of multi-class classification problems:

e-Mail Classification/labeling: Work mail, friend Mail, home mail, hobby Mail

Medical Chart (Medicaldiagrams): No illness, cold, flu

Weather: Sunny, cloudy, rain, snow

One-vs-all (one-vs-rest):

For the multi-class classification problem, it can be seen as a category two classification problem: keep One of the categories, the rest as another category. For example, for the following example, you can calculate the probability of one of the classes relative to the other class separately:

******************

hao_09

Date: 2015/8/9

Article Address: http://blog.csdn.net/lsh_2013/article/details/47381227

******************

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Machine Learning: Logistic regression