Can machine learning really work? (2) (take the two-dimensional PLA algorithm as an example)

One problem: In most cases, M (the size of the hypothesis set) is infinitely large, such as the PLA algorithm. So is our principle 1 impossible to use?

We try to do something about it:

STEP1: Look for the effective number of hypothesis set to replace M

What do you mean? This is the derivation, but, for example, in the PLA algorithm, H1 and H2 are so similar (considering the straight line on the plane), so if D is good for H1, then H2 is also good. That is: There are too many overlapping parts, we over-estimatinng.

Now let's change the idea. Look at the problem from the dataset's point of view.

What do you mean? For only one point of D, all H is divided into two categories: one is to divide X1 into one, and the other is to divide into X.

By now, we have understood this way of thinking. For each h, we look at the classification of each of the data in D. It's called a dichotomy. Then the effective number of H is the sum of dichotomies.

However, in this case, our calculation process relies on the specific data of D, so we use growth function to remove the dependency on the data: we use | H (x1; x2; : : : ; x N) | The upper limit. Used to indicate:

If growth function is polynomial rather than exponential, then we can use principle 1 to design learning algorithm A.

Step2:effective number is polynomial, not exponential.

As we know above, we just need to prove that effective number is polynomial, not exponential, and we're done. However, we need to introduce several concepts to assist our proving process.

Concept: Break point & Shatter

These are two concepts that are important for solving growth function problems.

There are 2 input points, and the H of PLA can fully achieve four classifications. At this point, the 2 points are said to be shatter.

There are 3 input points, the PLA H can fully achieve 8 classifications, at this point, said the 3 points are shatter.

However, there are no 4 points that can be used by the PLA H shatter.

In this case, 4 is the break point of H.

Set K is the break point of H, then there is the proof process. Normal and ingenious.

Define bounding function:

Then there are:

How to solve the remaining B (n,k)?

Take B (4,3) as an example to see if we can use B (3,?). Solve.

B (4,3) = 11, can be divided into two categories: one is x4 in pairs appear, a class is x4 into a single appearance.

Because k=3, so any 3 points can not shatter, namely: Α+β≤b (3,3).

And because for 2α, X4 is in pairs appear, so, x1,x2,x3 any two points must not shatter, otherwise, plus X4, there will be three points are shatter. namely: Α≤b (3,2).

Can be proved by mathematical inductive method. At this point the right end of the inequality is the upper limit of growth function. (can prove that the above ≤ is actually =)

Can machine learning really work? (2) (take the two-dimensional PLA algorithm as an example)