An Introduction to "Iterative Methods" in Machine Learning"

An Introduction to "Iterative Methods" in Machine Learning"

Zouxy09@qq.com

Http://blog.csdn.net/zouxy09

 

First, let's take a look at the eight-part article (from Baidu encyclopedia): the iterative method, also known as the tossing method, is a process of constantly using the old value of the variable to recursive the new value, what corresponds to the iteration method is a direct method (or a solution), that is, a one-time solution. Iterative algorithms are a basic method for solving problems with computers. It allows computers to execute a group of commands (or a certain step) repeatedly and execute these commands (or these steps) at a time by taking advantage of the high computing speed and the ability to perform repetitive operations) the original value of the variable.

To solve the problem using iterative algorithms, we need to do the following three aspects:

1. Determine the iteration variable.

Among the problems that can be solved by iterative algorithms, there is at least one variable that is directly or indirectly recursive from the old value to the new value. This variable is the iteration variable.

2. Establish an iterative relationship.

The so-called iterative relationship refers to how to derive the formula (or relationship) of the next value from the previous value of the variable ). The establishment of the iteration relationship is the key to solving the iteration problem. It can usually be done using a recursive or inverted method.

3. Control the iteration process.

When will the iteration end? This is an issue that must be considered when writing iterative programs. The iteration process cannot be repeated endlessly. The control of the iteration process can be divided into two situations: one is that the number of iterations required is a definite value and can be calculated; the other is that the number of iterations required cannot be determined. In the previous case, you can build a fixed number of cycles to control the iteration process. In the latter case, you need to further analyze the conditions used to end the iteration process.

 

I wonder if you have found that since we entered the field of machine learning, iterations have been lost and can be seen everywhere. No matter which paper or algorithm, we can see its shadows. Have you ever wondered why? Why is iteration so powerful? Why is it everywhere? Many conventional methods cannot solve the problem, and the iteration will take the lead. So powerful iteration?

Why is it so frequently used? My personal understanding is that no matter what machine learning algorithm, we have to turn to the computer to solve the problem. It is also manifested in searching for a specific function space based on a certain optimization goal. What does it mean? Our goal is to let machines learn and understand the physical world, which is random. So we need to guess that we have a lot of guesses about something, which of the following guesses is the most reliable? You have to have an indicator to measure it. What is the indicator? Is there a minimum error or the maximum performance indicator? So how do you find the minimum or maximum value? Do you mean we have a demand, but we have a Lagan? Yes, but what are the essential conditions they can use? It is the analytical expression of these error functions or performance functions. However, many signals in the world are non-stable, or it is difficult to know their statistical characteristics. At this time, we cannot get an accurate analytical expression. How can we find the maximum or minimum values. Iteration? What is iteration?

The optimization problem is either the maximum value, the minimum value, or the climbing from the foot of the hill to the top of the hill, or the downhill to the foot of the hill. Here is an example of climbing mountains. How do you know where the mountain is when you stand at the foot of the hill and cannot see the top of the hill? You only know that if I climb up the top of the hill every step, I will certainly be able to climb up the top of the hill. So you climb up step by step. When you crawl for a short period, stop and see which direction is the fastest, that is, the steep slope (that is, the gradient of the Performance surface ), then you will climb in this direction for a short while, and then stop and observe the fastest way up the hill until you climb the top of the hill. This is the shortest method (Hehe, the standard is generally called the shortest descent method when looking for the minimum value ).

If you see the top of the hill at the foot of the mountain, you will not care about it, you will go straight to the top of the mountain until you reach the top of the mountain. This is the Newton Iteration Method.

If you start at the foot of the hill and climb for a short step, then you will be blinded by the next step, because you believe that, god will take you to the top of the hill (God is the hand with the probability of manipulation). This is a random search algorithm.

Well, when you climb to the top of a mountain, you find that there is still a higher mountain, but there is no way. If you think of that higher mountain, you have to go downhill first and then climb that mountain. At this time, some people will be satisfied. They will see the beautiful scenery they want to see and enjoy his local maximum value. But some people are not satisfied. You are eager for the kind of state of mind that "the top of the hill is my peak and the mountains are small", so you will jump to the foot of the hill. If you are lucky, then you will roll to the foot of the highest mountain. At this time, you will climb up to reach the highest peak. This is an algorithm that avoids local maximum values and looks for global maximum values, such as simulated annealing or impulse.

In my personal understanding, it depends on three factors:

1) which direction do you go in each step;

2) How far are you going in each step;

3) whether the requirement for the global maximum jump from the local maximum is high.

I personally feel that these three factors constitute different optimization algorithms, or iterative algorithms.

In fact, life is an iterative process, every day in a new State, every day is constantly updated, slowly approaching your beautiful ideal. Sometimes you find that your life is very tired and you feel that your success is nowhere in the future. It is because your convergence speed is too slow. Sometimes you are too eager for success and benefits, and you may miss the goal and accidentally go over the hill. Therefore, the iteration direction and pace of your life are very important. Haha.

In addition, the iterative method is very helpful for solving some computation with a large amount of computing. For example, how to calculate the inverse of a matrix? Well, in the image field, the matrix is large enough, so we usually use the Klein law to solve the inverse of the matrix, that is, to calculate the ratio of the matrix's adjoint matrix to its determinant, this amount of computing is amazing. Although faster algorithms and faster hardware appear, this speed is unacceptable for programs with high real-time requirements. However, iteration will make you very comfortable, because the last result used in this calculation result, for example, the inverse of the nth evaluate is used for the nth + 1 time, in this way, we do not need to re-calculate the inverse of the square matrix of N + 1 at the nth plus 1 moment (you can understand it in this way ). For example, if you want to build a 100-storey building, adding a floor on the 99-storey building is much faster than opening a stove and building a 100-storey building on an open space. This is like a sequential iteration method or something.

Iteration is a very meaningful thing, so it is difficult for me to express his lofty and open-minded in such a pale language. Only in the vast sea of knowledge will I continue to meet, meet, know, and love each other!