Logistic regression in Machine Learning)

1. Classification

In this article, we will discuss the classification problem (classification problems). That is to say, the variable Y you want to predict is a discrete value. We will use the logistic regression algorithm to solve the classification problem.

In the previous article, the classification of spam is actually a classification problem. There are many other examples. For example, an online transaction website determines whether a transaction is fraudulent (some people can use stolen credit cards, you know ). Another example is whether a tumor is benign or malignant, which is also a classification problem.

In the preceding examples, we want to predict a binary variable, either 0 or 1, or a spam or a fraudulent transaction, or not; or a malignant tumor, or not.

The dependant variable classes can be called negative classes and positive classes ). You can use 0 to represent the negative class, and 1 to represent the positive class.

At present, our classification problem is limited to two categories: 0 or 1. Then we will discuss the problem of multiclass classification. That is to say, variable Y can take multiple values, such as 0, 1, 2, and 3.

So how can we solve a classification problem? Here is an example:

Now we have such a classification task. We need to judge whether the tumor is benign Based on the tumor size. As shown in the training set, the horizontal axis indicates the tumor size, and the vertical axis indicates whether the tumor is benign or not. Note that the vertical axis has only two values: 1 (representing a malignant tumor) and 0 (representing a benign tumor ).

Through the previous blog, we know that linear regression is used to process the above data sets. In fact, we use a straight line to fit the data. Therefore, the hypothesis you get may be as follows:

If you want to make a prediction, a feasible method is as follows:

From the above example, it seems that linear regression can also solve the classification problem. Now, let's make some changes to the above questions.

Expand the horizontal axis to the right and add a training sample as follows:

In this case, we use linear regression to get a new line:

At this point, we use 0.5 as the threshold to predict whether the tumor is benign or not.

2. hypothesis Representation

3. demo-boundary

It is emphasized that the decision boundary is not the property of the training set, but the property of the hypothesis itself and its parameters. Given the parameter vector θ, the decision boundary is determined. We do not use a training set to define the decision boundary. We use a training set to fit the θ parameter. We will talk about how to do this in the future. However, once you have the parameter θ, it determines the decision boundary.

4. Cost Function 1

Now we will discuss how to fit the model parameter θ in logistic regression.

Specifically,We need to define optimization objective or cost function to fit the θ parameter.This is the fitting of the logistic regression model in supervised learning.

As shown in, we have a training set with M training samples. Like before, each of our samples is represented by a n + 1-dimensional feature vector (X0 = 1 ). In addition, due to the classification problem, all y values in the training set are 0 or 1. Assume that the function parameter is θ. How can we fit the θ parameter (or select the θ parameter) for the given training set )?

Previously, when we used a linear regression model to fit the hypothesis parameter θ, we used the following cost function, and we slightly changed it, put the original 1/2 in the first 1/2 m to the summation symbol.

Now we use another method to write the cost function:

Now, we can see more clearly that the cost function isCost Function (cost item)Sum the sum in the range of the training set, and then calculate the mean value (multiplied by 1/M ).

The cost function is defined as follows:

The understanding of this cost item (cost function) is as follows: y the value I expected, if you want to reach this value through the learning algorithm, then suppose h (x) the price is the price. The expected predicted value is h (x), while the actual value is Y. simply remove all the superscripts.

Obviously, in linear regression, the cost item (cost function) is defined as the square of the difference between 1/2 multiplied by the predicted value H and the actual value Observed Result Y. This value generation can be well used in linear regression.
But it is not suitable for logistic regression.

2

If we can minimize the cost item (cost function) in the cost function J (θ), we can indeed use this cost item. But in fact, if we use this cost item, the cost function J (θ) will become a non-convex function about the parameter θ. Why?

For logistic regression, the H function here is non-linear:

It can be said that:

Is a very complex non-linear function, so if we use the H function to construct the cost item we use in linear regression (cost function ), then we use this cost item to construct the cost function J (θ ).

Then J (θ) may be a function with many local optimal values:

In fact, this is a non-convex function.

It is not hard to find that if you use the Gradient Descent Method on such a function, we cannot guarantee that it will converge to the global minimum value.

Obviously, we want our cost function J (θ) to be a convex function, that is, a single bow function, as shown in:

If the gradient descent method is used for this function, we can ensure that the gradient descent method converges to the global minimum value of the function.

Therefore, the problem with using this cost item (cost function) in logistic regression is that the emergence of the non-linear sigmoid function leads J (θ) to become a non-convex function.

3

What we need to do is to find another cost item (cost function) that is itself a convex function, so that we can use an algorithm similar to gradient descent to find a global minimum value. Here is a price item (cost function) We will use in logistic regression ):

5. simplified cost function and Gradient Descent

Note that θ is a variable. Our goal is to find the θ value that minimizes J (θ.

Logistic regression in Machine Learning)