Normal mapping principle

The appearance of normal map is to simulate the " illumination information " of the model with high polygon number for the model of low polygon number. Illumination information The most important thing is the angle of the light incident direction and the entry normal. Normal mapping is essentially a matter of recording this angle. The calculation of illumination is closely related to the normal direction of a plane .

We know what's in the computer.

A model is an approximation of an object by a combination of multiple polygon faces. It is not sleek. The more polygons, the closer the real object is. When you light a point in a face, the normals are interpolated through the vertices of the polygon. Interpolation is actually to simulate the "correct" normal direction of this point, Otherwise, the normals of all points in the whole face are consistent, and the light goes up, and we see the model exaggeration points like a mirror stitching together. But the normal interpolation is still distorted. The higher the number of faces in the model, the less natural the distortion is

The number of polygons is high, and the amount and memory needs to be calculated are high. The predecessor found the normal map (formerly the Bump map) This method, so that the low modulus can be approximated to enjoy the higher-modulus illumination details. The price is there, Is the need for a file that records this information. This is a common practice in the program for calculating time for storage space. 3D programs prefer to use this technique. The unit price of the storage hardware is much faster than the unit price of the computing hardware.

The graphics card includes a graphics API that is supplemented with it, and the original source of the read data is the image. So the file that records this information is saved as a picture format. 2 words behind the normal map that's it. Well, you already know half of it.

Because of the small number of polygons, a surface of an area on a low modulus may be several facets of the same area on the high modulus. Comparison of high modulus and low modulus (for simplicity we abstract 2-D segments)

The curved convex curve of the upper concave represents a high modulus. The lower one is a smoother representation of the low modulus. Because of the higher modulus details, it is more natural to change direction in a certain area than the flat plate. I can't understand what I can do.

Seeing this picture, some people should feel something. It doesn't matter if you don't feel it, then come again.

Regardless of the high or low modulus, anyway, the end is to be colored. Assuming that the model has been rendered with color, now we imagine using scissors to expand the model (similar to the animal Mr Skinner process), get 2 of the same size of the skin, after all, the area will not be too much. High modulus of skin of course white body tender precision, The low-modulus skin is a little rough. Now imagine the process: gradually move the high-modulus skin above the lower-mold skin to a certain height until the horizontal overlap.

Do you feel this way now? It doesn't matter if you don't feel it, then come again.

Although the model precision is different, in any case, these 2 skins each point is colored (the value of the interpolation). Two of the same color on the skin, high modulus this skin is more real, because in the calculation of the final color information depends on the normal, high modulus   point more accurate than the point on the low modulus. How do we give low modulus this skin beauty, so that it can approach the effect of high modulus? It is impossible for a qualitative change to mating.

approach is violent. Now imagine you are using a needle, from top to bottom, pierce the high-modulus skin, and then stab the low-modulus skin. Make sure the needle is perpendicular so that it sticks to the same point. and imagine if the needle was magical, it pierced the higher-die skin, stealing some information and transmitting it to the low-die skin. Low-profile The success of the transformation into black fungus. What is this information about? Of course, the normal information. Now the high modulus of this skin is densely filled with the needle, in other words, the preservation of the information of the leak, must be point-to-point. That is, every point on this skin must be preserved. So normal maps are the same size as the original Each point in the map holds the normal information corresponding to a point at a high modulus. The actual calculation is only concerned with the normal information obtained from the map, and the normal on the low modulus is discarded.

Do you feel this way now? It doesn't matter if you don't feel it, then come again.

Why did I emphasize vertical? Not just for needles to stick to the same point. Now please take this process, Imagine being in the. The arrows in the figure indicate the normal direction of a point on the high modulus. How do I record this direction information? Now imagine that the high-modulus and low-modulus overlap, in order to make it easier to imagine, the lower modulus is smaller than the high modulus. Or you can simply imagine that the high-modulus face is directly above the low-die surface. Imagine a beam of light (the equivalent of a needle), From the top down, the normal on the high modulus is projected onto the low modulus.

The foreplay is done, now let's deal with a little bit of detail. This is a projection process. But the shadow is a 2-dimensional one. The vector is made up of three components of x, Y, Z. A point on the high modulus is projected onto the plane of the corresponding point on the lower There are only 2 components left in the projection. As we now only know the normal in the X-y projection direction, that in the direction of the Z axis? As long as we ensure that the normal is a unit vector before the projection, it is very simple z=1-x*x-y*y. So we can save space for Z. In fact, since we already know the normal direction ( The normal direction in the high modulus object space), and is also the unit, the direct preservation is possible. The projection process is just a thought experiment, and there is really no light projecting from top to bottom.

To this can be clear, "orthodox" normal map generation, is a high modulus, low modulus is indispensable. Because there is no high modulus of the normal direction, there is no low modulus, it is not known that the high modulus of a certain point of the normal corresponding to the low-modulus point.

because the normal information of a point is saved to the corresponding pixel point on the normal map. The actual calculation is to put the normal x, Y, The z-direction size is mapped to the color space RGB. Is the X value exists in R, the Y value exists in G, the z value exists in B. Because RGB is 8 bytes. Therefore, the high-modulus normal information is stored in pixels to lose precision. And it is impossible to match the high modulus and the low modulus It's a simulation process. Natural normal maps are not invincible.

Now we can answer the question that the previous Photoshop generated the normal map based on the diffuse map. The actual diffuse map does not contain the normal information on the model at all. So it's a normal map based on the diffuse map is simply wrong. But why can it be applied? Imagine the high-modulus accuracy of the scary, high to the after the rendering of the high-modulus leather after the cut down, it became a picture. Before imagining the stickers on top of the mold are covered with rust. So you get a rust photo. Photoshop handles This rust photo in fact according to some algorithms (Sobel, etc.) The color value is converted to a gradient value, and the normals are approximated. Because we don't really care about the exact distribution of rust, like that, so it is possible to do so in this case, and the potholes are best suited to this approach. Photoshop's approach to getting out of high-modulus low-modulus is confusing, Causes the novice to think that the normal is from the diffuse map, or simply blocked the idea.

The normals we used to calculate the normal map above, and where we came from. If this normal direction, is in world coordinates, which is known as the Universe normal. If it is in the object's own local coordinates, it is called object Space Normal. It's easy to imagine that world space Normal once extracted from the map, it can be used directly, high efficiency. But there is a drawback that this world space normal is fixed, if the object does not maintain the original direction and position, the original generated normal The map is void. So someone saved the object space normal. It is extracted from the map, and it needs to be multiplied by the Model-view matrix to convert to world coordinates, or to other coordinates depending on the calculation process and requirements. Object space Normal generated map, objects can be rotated and shifted. It's basically satisfying. But there is one drawback. A map can only correspond to a specific model, and the model cannot have a deformation (deform).

>>  tangent space normal map

Let's take a look at object space. When a low-modulus rotation, because the rigid body does not deform, The equivalent of each point is multiplied by a rotation matrix R, after which the point relationship remains unchanged. In fact, we keep the object from spinning, rotating the coordinate system of object space (x, Y, Z, three axes) to get the same result. This relationship is believed to be understandable. In other words, normals are for object Space is fixed, the object remains in the object space fixed, just follow the movement of the coordinate system, the rotation is OK. Now we're imagining that a point in a low modulus needs to be deformed, which in principle can also be done by making the object The space coordinate system is multiplied by a variant matrix T to achieve. But different points have different deformations, and it is not possible to have a matrix T that fits this point and is suitable for this point. Therefore, the object space coordinate system is not available. Which single coordinate system can have a transformation matrix that is common to all points? Obviously can't imagine.

When the deformation, the vertex relationship changed, that is, the shape of the polygon, the direction changed. If there is a fixed coordinate system on the face That when the object is deformed, moved, rotated, and the coordinate system is bound to move with the face, then a point or vector in the coordinate system (e.g. we convert the high-modulus normals into this coordinate system) does not need to be changed. When the entire face changes, we only need to calculate the coordinate system on the plane to the world coordinate system transformation matrix, Then the point or coordinate defined on this surface (fixed), multiplied by this matrix, is the coordinates in the world. How this coordinate is constructed is not important to us at this time, so be sure to understand this concept. We're just looking for a local coordinate system, a point coordinate in the local coordinate system, multiplied by the local coordinate system to the World Coordinate transformation matrix ( This matrix is dynamically computed at low-modulus rendering, and the coordinates of points in the local coordinate system are obtained in the world coordinate system. The fixed values stored in the normal map (normal direction) can be calculated in a meaningful way.

See here it's obvious, This approach requires thousands of different coordinate systems that are defined on the polygon. How many faces there are on the lower die, how many of these coordinate systems are needed. The calculation of this method is naturally larger than the object space normal map. On each face of the low modulus, To construct this coordinate system. This coordinate system term is called tangent space.

object In space normal map, the object space coordinate system of the low modulus is coincident with the object space coordinate system in the high modulus. So it doesn't need to be built, so a point on a low modulus can replace its normal with a high-modulus normal. The concept of coordinate system overlap is important. In the new method, This tangent space on a low modulus must also be tangent space with a coordinate system on the high modulus. Because a face on a low modulus may correspond to several faces on a higher modulus (high accuracy), each face has a local coordinate system according to the new method, for each face on the low modulus, the high modulus has several faces , there will be a few faces, which is certainly not possible. So the tangent space used for high modulus is the low modulus. Generating a normal map, it is necessary to confirm which faces on the low modulus are corresponding to which faces on the lower die, and then the normals of the faces on the higher modulus, are converted to the coordinates of the tangent built on this side of the low modulus. In this way, when the low modulus deformation, that is, the triangular surface changes, its tangent space will also change, the normal stored in the map multiplied by the low-modulus side of the tangent Space to the external coordinate system of the transformation matrix can be obtained external coordinates. By the way, this normal of high modulus preservation is the normal of object space in higher modulus, and you should see here that it is natural. What you might see when you search for an article convert the light into tangent space, Make sure that you are in the same coordinate system. Have a little 3d mathematics knowledge of the people know, but also to you mention the perfunctory? I thought to make sure tangent sapce coincidence and practice, is to let the person epiphany tangent space of the trick point.

When I think of this paragraph myself, tangent space's normal mapping principle is enlightened. Next we build this tangent space coordinate system.

When the face is moving, tangent space also has to follow. The vertical normals on the surface are followed, so the normal n can be used as an axis of tangent space. It is very important to note that the vertical normal on the surface is not the normal of the interpolation, and that normal is what we need to preserve. n simply refers to the direction perpendicular to the face.

We examine, for a triangular face, its side v2v1,v3v1, V3v2 we can always be sure of it. The edges will also move in the shape of the deformation. So we can select an edge as the second axis of tangent space T. The third axis is simple, directly according to the cross product to B=t * N. This axis is ordered. In fact, The axis selection can be almost arbitrary, as long as you can make sure that you can build it every time. For example, you can select V1v3,v1v2 as the axis, N=v1v3 * v1v2. Here N is exactly the same direction as before. But so the v1v2,v1v3 in this coordinate system is not perpendicular, The non-orthogonal coordinate bases are not convenient in the matrix operation, but also have to be orthogonal. So we choose the first most intuitive and clearest and most convenient method.

Since the three axes are determined, the matrix that constructs the object space to tangent space is simple, and we o-tbn the T,b,n, respectively, as the x, Y, Z axis of the tangent space. Based on the three coordinate bases we construct the matrix as follows:

A certain point of normal in object space is multiplied by this matrix, which is to get the normal direction in tangent space, and then map this value to the RGB space and save it as a map. Why is this matrix so, this is an off-topic. Let me briefly say: Object Space's three axes (1,0,0), (0,1,0), (0,0,1) multiplied by this matrix, must be exactly the axis of the tangent space, the natural matrix is the top. and the coordinates of the other points of object space are X, Y, Z Three a linear combination of unit coordinates. So this matrix must be correct for the other points.

actually in vertex shader, we only know the current vertex information, We don't know the other two vertices of the triangle. But modern shader can provide a tangent information for vertices, Represents the tangent at the vertex. You can imagine a tangent on a football that passes through a certain point. So we'll take the tangent direction of the vertex as the T vector above. This is also tangent space called the origin of this name .  you will see a lot of articles mentioned in the texture of U, V direction. Because the u,v of a point on a polygon is interpolated along each edge, the u,v direction is the same as the direction of the edge. In fact, we already have a ready-made tangent to use.

  OK, now the normal values of each point on the high-modulus surface are converted to the tangent on the low modulus Space coordinates. Now we are considering the specific low-modulus rendering calculations. Suppose that a surface on a low modulus we calculate this matrix and take out the corresponding normal value in the normal map of a point on the face. Now we need to calculate the light. We can convert the tangent Space. You can also convert the resulting normal vector to world space and the light vector. The result is the same. As a practical consideration, you will find that the latter method is not good. Because for each point on the polygon, the normal to world space is calculated once. and the previous method , for all points on a polygon, the calculation of the light vector to tangent space is computed once. Then consider the process of vertex shader and fragment shader, and you'll find that we can calculate the light to vertex in shader tangent Space conversion, in the fragment Sader take out the normal value and the previous tangent space in the direction of the light to do the calculation. Here is a reminder that the direction of light we get in general Verteix shader is based on world space, The normal map holds the direction of the high modulus object space and then to tangent space, so in vertex shader we must first convert the light to object space and then to the tangent Space. This ensures that the light and normals are based on the same coordinate system at the end of the calculation.

The above is the principle of normal mapping. Because the principle of the application of a wide range, but also to string a lot of knowledge points, is very well worth figuring out

The results of the original, normal, and program calculations are as follows:

Normal mapping principle