On manifold Learning

Reprinted from: http://blog.pluskid.org/?p=533

Always feel that even the "talking about" two words, or let the title some too big, not to mention I have just contacted such a field. But too lazy to think of other headlines, think about this topic, but also because with friends chatting about my own recently in the direction of doing. Manifold learning or just manifold itself usually sounds a bit esoteric, but if you don't want to do a rigorous theoretical deduction, you can get some perceptual understanding from a lot of intuitive examples, just to take this opportunity to talk about it briefly, Or at least I know about it so far.

These two words, before talking about manifold, may wish to say first learning, namely machine learning. and said machine learning not mention Artificial Intelligence words seem to appear somewhat not kind. People say AI is one of the most tragic disciplines, because whenever one of its sub-domain development smarty pants, then immediately on their own, and AI "No more involved", and machine learning is probably the latest one of the typical bar. This is a bit strange, for example, math, is it enough to categorize? Regardless of how the points, the brothers and sisters also admit that they are called "math". What about the AI? I think a big part of it is the problem of its own positioning.

Anyway, now I'm not quite sure what AI is doing and I don't know what other people are really clear about. Wikipedia says

Artificial Intelligence (AI) is the intelligence of machines and the branch of Computer Science, which aims to create it.

But this is equivalent to a tautology, because what is the intelligence of machines ? At first, Daniel was ambitious, and seemed to be full of confidence, as if it had been widely believed that "Newton's theorem reveals the truth of the universe, the rest of the science as long as the formula to do the calculation can be", we may feel that, in a few decades, humans can not think, to AI to do. But I don't want to say anything about "thinking", What is "smart" and the accompanying "Turing test" topics. What I want to say is, in the end, what the AI is, this is still a problem, or AI at first set a high goal, decades later, found that the situation is not as optimistic as it was, but some can not.

At this time, some side branches or sub-domains of AI decisively put down the face, lost the near-fantasy goal, gradually developed into a "normal" discipline, so it is no longer good to call AI. Or now the AI has two meanings, a generalized AI, including all relevant and derived fields, the other is a narrow or classical AI, especially those who are still clinging to the pursuit of real "intelligent" part, or not a little better, is the rest of the part.

Machine learning as a typical runaway, although the name of a learning a word, let people at first glance feel compared with Intelligence is just a change, but in fact here the meaning of learning is much simpler. Let's take a look at the typical process of machine learning, which sometimes feels like applying math or more popular mathematical modeling, usually we have data that needs to be analyzed or processed, and based on some experience and assumptions, we can build a model This model will have some parameters (even non-parametric methods can be seen similarly), according to the data to solve the model parameters of the process, called Parameter estimation, or model Fitting, but the machine learning people, usually called it learning (Or, change an angle, called Training)--because it's a useful model based on the data, and it's quite similar to the "learning" process of human beings. However, if you throw away the boring literal kind game, we can see that machine learning has abandoned the "smart" high hat, its purpose is to solve the specific problem-and do not care whether it is through a "smart" way to solve the class.

Here, in fact, we construct the model is similar to writing a class, the data is the parameters of the constructor function, learning is the process of the construction function, after the successful construction of an object, we completed the study. Some machine learning problems come to an end, and others use the resulting model (object) to do some processing of later data, usually inferencing. At this time, there are some things like statistics, so-called "statistical inference." In fact, many of the problems of the original statistics and machine learning research are overlapping, but the two factions look at the same problem from different angles. And, indeed, there are statistical learning such a claim, which can be seen as a sub-domain of machine learning (or a molecule or even machine learning itself).

Here, if you haven't been annoyed by the constant keying of words, I can't stand the patience. So, I assume you already know what is called learning, or is already disgusting to bother to understand. So we moved on to the next topic: manifolds, which is manifold. I wonder if you are confused about the picture of the earth that I put on the beginning of this article? This is because the spherical surface is an example of a typical manifold, and the earth is a very typical "spherical" (just like a spherical surface).

Sometimes the "low-dimensional manifold embedded in high-dimensional space" is often seen in the paper, but high-dimensional data is always unthinkable for our poor low-dimensional organisms, so the most intuitive examples are usually two-dimensional or one-dimensional epidemics embedded in three-dimensional space. For example a piece of cloth, you can think of it as a two-dimensional plane, which is a two-dimensional Euclidean space, now we (in three-dimensional) twist it, it becomes a manifold (of course, do not twist, it is a manifold, Euclidean space is a special case of manifolds).

So, intuitively speaking, a manifold is like a D-dimensional space, in a m-dimensional space (M > D) is distorted after the result. It is important to note that the manifold is not a "shape" but a "space", and if you feel that the "distorted space" is hard to imagine, then remember the example of a previous piece of cloth. If I am not mistaken, general relativity seems to be the study of our time and space as a four-dimensional flow (three dimensions plus time one dimension), gravitational force is the result of this manifold distortion. Of course, these are intuitive concepts, in fact, the manifold does not need to rely on embedded in a "peripheral space" and exist, a little more formally, a D-dimensional manifold is an arbitrary point out of the local embryo in (simply, is the positive inverse mapping is a smooth one by one map) Euclidean space. In fact, it is this kind of local and Euclidean space of the same embryo gives us a lot of benefits, which makes us in daily life many of the geometrical problems can be solved with simple Euclidean geometry, because compared to the scale of the Earth, our daily life is even a very small part of it-I suddenly think of "Seven Dragon Bead" That the king of the realm of the kind of private small planet, walk a few steps around a circle of feeling, it seems that the king not only to be physically good (the above gravity seems to be 10 times times the Earth), but also the brain is better, junior high school must be Riemann geometry!

So, in addition to this simple example of Earth, the actual application of the data, how to know that it is not a manifold it? Then you may return to the intuitive feeling. Again from the sphere, if we do not know the existence of the spherical surface, then the point on the spherical surface is actually a three-dimensional Euclidean space point, you can use a ternary group to represent its coordinates. But unlike the normal points in space, where they allow the position to be limited to the sphere, you can look at its parametric equation:

As you can see, these three-dimensional coordinates are actually generated by two variables, or it can be said that the degree of freedom is two, which corresponds to a two-dimensional manifold. With this feeling, it is natural to look at the examples of faces that are often used in manifold learning. is a result of the Isomap paper:

Here's the picture from the same face (well, is actually a human face model), each picture is a grayscale image of 64x64, if the bitmap in accordance with the column (or row) together, you can get a 4096-dimensional vector, so that each picture can be regarded as a 4096-dimensional Euclidean space in a point. Obviously, not one point in a 4096-dimensional space can correspond to a face picture, which is similar to a spherical case, and we can assume that all 4096-dimensional vectors that can be human faces are actually distributed in a subspace of D-dimensional (d < 4096). and specifically to the Isomap face of this example, in fact we know that all 698 pictures are taken from the same person face (model), but in different pose and light shooting, if the pose (up and down and left and right) as two degrees of freedom, and light as a degree of freedom, So these pictures actually have only three degrees of freedom, in other words, there is a spherical equivalent of the parametric equation (of course, the analytic formula can not be written out), given a set of parameters (that is, the upper and lower, the left and right pose and illumination of the three values), it is possible to generate a corresponding 4096-dimensional coordinates. In other words, this is a 3-dimensional manifold embedded in the 4096-D Euclidean space. In fact, the graph above is Isomap the data set from 4096 to 3-dimensional space, and shows the results of 2-dimensional, the figure is each face in this two-dimensional space in the corresponding coordinate position, some of the red circle points are selected, and next to the point of the corresponding picture of the original It is very intuitive to see that these two dimensions correspond exactly to the results of the smooth changes of the two degrees of freedom of pose. As far as I know, the introduction of manifolds to machine learning has two main purposes: first, the original algorithm used in the Euclidean space is modified to make it work on the manifold, directly or indirectly, the structure and properties of the convection shape to use; the second is to directly analyze the structure of the manifold and attempt to map it to a Euclidean space, Then, the results are used to study the Euclidean space algorithm. Here Isomap happens to be a very typical example, as it is actually "transforming a manifold into a Euclidean space" by "retrofitting an algorithm that would otherwise be suitable for Euclid spaces". :) This method Isomap is called  multidimensional Scaling (MDS)  ,mds is a dimensionality reduction method, Its purpose is to make the distance between the point 22 after the dimensionality is as constant as possible (that is, the distance between the two points corresponding to the original space is the same). Only the MDS is designed for Euclidean space, and the calculation of distances is done using Euclidean distance. If the data is distributed across a streamThe Euclidean distance is not applicable.

Let's go back to Earth-this two-dimensional manifold in three-dimensional space, assuming we want to calculate the distance between the north and South poles in three-dimensional space, which is easy, that is, the length of the two-point line segment, but, if you want to be in this manifold at a distance can not do so, we can not drill a hole from the To walk along the surface of the Earth, of course, if I go along any route, and then count out the total number of steps as a distance, this is not, because this way if I go along different routes, will not get a different distance value? All in all, we now need a new definition of the distance measurement on the Earth's surface (manifold), theoretically, any function that satisfies the 4 conditions of the measure can be defined as a distance, however, in order to correspond with the Euclidean space, select a generalization definition of a straight line distance.

Remember the first high school "between two points, the shortest line"? Now, instead of extending the concept of line segments to "the shortest curve between two points is a segment", the distance definition on the manifold is equivalent to Euclidean space: the distance between two points on a manifold is the length of the "line segment" connecting two points. Although it is only a replacement concept, but now the two unified, but in the manifold line is probably not necessarily "straight" (so the line is not necessarily "straight"), often referred to as "geodesic." For spherical this simple manifold, any line must be in a "great circle", so the line on the spherical surface is actually some great circles, but also caused the spherical surface of the manifold without parallel lines such as a series of embarrassing situation (any two straight lines intersect), if you have seen some of the math science gossip class books, Should remember a lot of things!

Back to Isomap, one of the main things it did was to change the distance in the original space in MDS from Euclidean distance to the geodesic distance on the manifold. Of course, if the structure of the manifold is not known beforehand, this distance is impossible to calculate, so Isomap by connecting the data points to form an adjacency Graph to approximate the original manifold, and the geodesic distance corresponding to the Graph on the shortest path to approximate. As shown in the following:

This thing is called the Swiss roll, let's think of it as a piece of cloth rolled up. The points of the two black circles in the graph, if calculated by Euclidean distances in the outer Euclidean space, are very close points, but in the manifold they are actually far away: The red line is the distance from the manifold that Isomap. It can be imagined that if it were the original MDS, it would definitely be violent to project it into a two-dimensional space, completely ignoring the manifold structure, and Isomap could successfully "unfold" the manifold before projecting it.

In addition to ISOMAP, manifold embedding has many algorithms, including locally Linear embedding, Laplacian eigenmaps, Hessian eigenmaps, Local Tange NT Space Alignment, semidefinite embedding (Maximum Variance unfolding) and so on. Which is good or bad is not good to say, they all have their own characteristics, but also have a lot of variants. There is a Matlab demo on the Internet, and gives a few popular manifold embedding algorithms on some synthetic manifold results and comparisons, can have an intuitive understanding.

On the other hand, the transformation of the existing algorithm to fit the manifold structure and even specifically for the characteristics of manifolds to design a new algorithm, more typical graph regularized semi-supervised learning. Simply put, in supervised learning, we can only use the data with the label, and (usually there are a lot of) the data without the label is wasted. In the manifold hypothesis, although these data do not have a label, but still can help learn out the structure of the manifold, and after learning the manifold structure in fact, we are a little more understanding of the original problem, so it is natural to expect to get better results.

Of course, all of this is based on the same hypothesis that the data is distributed on a manifold (some of the algorithms may have slightly looser assumptions), but what exactly is real world data on manifolds? This is hard to say. However, in addition to the typical face and hand written digit, there is also a flow-based algorithm that works directly on data such as text that seems to have no relation to the manifold, and the effect seems to be good.

In other words, although it is a very important issue to combine with practical applications, it is not conclusive, especially for those who engage in theoretical aspects. I think, for them, it is actually like the application, but the mathematics of things to apply to machine learning problems, and the relationship between the complex, graph theory, Algebra, topology, Geometry ... As if the 18-way princes gathered in a hall, so I always feel that it will take another 500 years to do a bad math!

However, I can understand the potential of the existence of happy, like you play the game, the first play "Heaven and earth Rob: God Evil Supreme Biography", and then Play "Heaven and earth Rob Preface: Phantom Sword Record" in the time, will find inside the characters, plot gradually linked together, Perhaps even just in one place has appeared in a completely insignificant supporting role, when you again in another place suddenly see he is, a series of clues instantly be linked together, the overall picture gradually emerges, that kind of happy feeling, probably is called "dispel"!

So ah, these things are almost the same, sometimes think of those who stand at the top of the peak "list of the mountain small" feeling, the kind of "mastery" of the feeling, really let people all kinds of envy! But I had to climb the old mountain and climb the North Peak to experience the experience.

Finally, the conclusion is: the so-called machine learning in the learning, is to establish a model, by the given data to solve the model parameters. The manifold learning, in the model, contains the manifold assumptions about the data. Well, what else is there? Also, I painted the ice glass on the city wall of Loulan, not a bit like ... >_< the City fans don't cut me ... Update 2011.02.28: According to the information in the message, "manifold" this Chinese word from the Wen Tianxiang "heaven and earth have upright, miscellaneous flow shape", the word for the first time as the current mathematical significance of the use of the mathematical Department of Peking University, an old Professor Jiang Zehan. The old gentleman is the pioneer of algebra topology in China. It is also the teacher of the university time that the tutor of Caszgh classmate.

Tags:manifold

Comments to brief introduction to manifold learning «older comments 12

• Glimpse September 2nd at 9:06 pm • Reply

  Landlord Hello! I just contacted manifold, read your blog, understand a lot, thank you! Please also provide more comments on manifold application.

• Wangkexue October 18th at 2:35 pm • Reply

  Like blogger blog style, thumb up!

• hai na bai chuan December 17th at 3:19 pm • Reply

  Thank you so much, for me this new contact manifold Learning Novice, you this blog is the Savior Ah!!

• Kide December 28th at 2:04 pm • Reply

  Landlord Hello, I still want to ask you in the article cited image example, is considered as a 698 (picture number) multiplied by 4096 (each picture pixel) data, and then use the flow to reduce the dimension to 698 times 3 of the data, so that the characteristic value is there, then how do you carry out the verdict? Now there is a picture of the 64x64, if it is unfolded as a 4096-dimensional vector, he is a 1x4096 data, this data is not able to use manifolds to reduce the dimension of it (Isomao to calculate the distance matrix, this is a point, the distance matrix is 0?). ), how to make a verdict? Do not know my elaboration is not a problem, is very want to know, you use manifold learning to do feature extraction, get is 698x3, then you are how to carry out the verdict (pending decision data for 64x64), thank you!

  • Pluskid December 29th at 1:51 am • Reply

    Hello, if you say that the discriminant is to be classified, the reduced-dimension algorithm can not be directly used for classification, but also need to cooperate with other classification algorithms.

On manifold Learning