Soft-margin SVM

Consider this situation:

If we use Hard-margin SVM and do not tolerate a little error, we will get the result on the right. Obviously, the result on the left is more reasonable, so in practice we use a SVM that can accept a certain error (tolerate noise), namely: Soft-margin SVM.

Combine two conditions to:

There are several problems with this:

One is that our objective function is no longer linear, so we cannot use QP, and the second is that we treat large errors and small errors equally.

So we'll change ∞ to ε:

(This is also a question: ε is different for each data, and even becomes a target variable.) Not very easy to understand)

Now the form of Soft-margin SVM has been clarified, how to solve?

1. According to the process of solving duality problem which we deduced in Hard-margin SVM, we can get the following results, and also have primal-innner optimal (complementary slackness) in kkt condition.

Solving:

The question here is how does B get it? According to the previous Primal-innner optimal conditions:

at this point, the solution process of Soft-margin SVM is finished .

Previously Hard-margin we divided all the data points into support vectors and general vectors based on the value of Alpha.
Now there are three cases of α in soft-margin, which can be divided into three categories:

For Gaussian kernel functions, how to select the appropriate C and gamma?

You can use validation. You can also use #sv because:

The difference between using the two choices is as follows:

Soft-margin SVM