Processing 2D images and textures-projection textures

Creating a mirror: Projection texture Problem

You want to create a mirror in the scene. For example, create a rearview mirror in a racing game. You can also use this technology to create a reflection texture.

Solution

First, you need to draw the scene in the mirror into a texture. Then, draw the scene (including the empty mirror) You see in the camera and paste the texture on the mirror.

To draw a scenario that looks like a mirror, you need to define a second camera called the mirror camera (mirror camera ). You can use the Position, Target, and Up vectors of the first common image camera to obtain the Position, Target, and Up vectors of the image camera. When you use an image camera to watch a scene, you can see the same as what you see in the mirror. Figure 3-24 shows this principle.

Figure 3-24 image principles

After saving the result to the texture, you will draw the scene you see in the normal camera and use the projection texture to draw the texture to the mirror, the projection texture maps correct pixels to the corresponding position of the mirror. This method is not applicable if an object exists between the image camera and the mirror.

You can solve this problem by defining an image cropping plane, so that all objects after the mirror will be cropped.

Working Principle

First, add the following variables to the project:

RenderTarget2D renderTarget;Texture2D mirrorTexture; VertexPositionTexture[] mirrorVertices; Matrix mirrorViewMatrix; 

You need to draw the scenes in the mirror into a custom rendering target, so you need the renderTarget and render texture variables. To create an image camera, you need to define an image View matrix. Because you need to add the mirror to the scene, you need some vertices to define the mirror position.

The variable renderTarget is initialized in the LoadContent method. For more information about creating and using custom rendering targets, see tutorial 3-8.

PresentationParameters pp = device.PresentationParameters; int width = pp.BackBufferWidth; int height = pp.BackBufferHeight; renderTarget = new RenderTarget2D(device, width, height, 1, device.DisplayMode.Format); 

TIPS:You can also reduce the width and height of the rendering target. In this way, the video card costs less, but the image in the mirror looks rough.

After initializing the rendering target, define the mirror position:

private void InitMirror() {    mirrorVertices = new VertexPositionTexture[4];         int i = 0;     Vector3 p0 = new Vector3(-3, 0, 0);     Vector3 p1 = new Vector3(-3, 6, 0);     Vector3 p2 = new Vector3(6, 0, 0);     Vector3 p3 = p1 + p2 - p0;         mirrorVertices[i++] = new VertexPositionTexture(p0, new Vector2(0,0));     mirrorVertices[i++] = new VertexPositionTexture(p1, new Vector2(0,0));     mirrorVertices[i++] = new VertexPositionTexture(p2, new Vector2(0,0));     mirrorVertices[i++] = new VertexPositionTexture(p3, new Vector2(0,0));         mirrorPlane = new Plane(p0, p1, p2); } 

You can use this method to create a mirror of any shape. In this example, a simple rectangular mirror is created. Display the content in the Custom rendering target using the mirror plane. You need to use TriangleStrip to draw two triangles to define this rectangle, so you only need four vertices.

In 3D space, you only need three points to define a rectangle. This method allows you to specify the vertex p0, p1, and p2 of the mirror. The Code calculates the vertex p3 so that the fourth vertex is also in the same plane, corresponding to this line of code:

Vector3 p3 = p0 + (p1 – p0) + (p2 - p0); 

Create four vertices from these four locations. This method does not need to pass texture coordinates to (see the vertex shader later), but because I am too lazy to define a VertexPosition structure, I simply pass some arbitrary texture coordinates, for example (0, 0), because they are not used (see tutorial 5-14 to learn how to create custom vertex formats ).

Note:In this example, all zcoordinates are 0, so the mirror is in the XY plane.

Create a View matrix for an image camera

Next, you want to create an image View matrix to draw a scenario that looks like a mirror. To create this image View matrix, you need the Position, Target, and Up vectors of the image camera. The Position, Target, and Up vectors of the Image View matrix are images of a common camera about the mirror plane, as shown in Figure 3-24.

private void UpdateMirrorViewMatrix() {    Vector3 mirrorCamPosition = MirrorVector3(mirrorPlane, fpsCam.Position);     Vector3 mirrorTargetPosition = MirrorVector3(mirrorPlane, fpsCam.TargetPosition);     Vector3 camUpPosition = fpsCam.Position + fpsCam.UpVector;     Vector3 mirrorCamUpPosition = MirrorVector3(mirrorPlane, camUpPosition);     Vector3 mirrorUpVector = mirrorCamUpPosition - mirrorCamPosition;     mirrorViewMatrix = Matrix.CreateLookAt(mirrorCamPosition, mirrorTargetPosition, mirrorUpVector); } 

Position and TargetPosition can be simply mirrored because they are in absolute 3D space. However, the Up vector indicates a direction and cannot be immediately mirrored. You need to first convert the Up direction to a 3D position, somewhere above the camera, which can be done by adding the Up direction to the 3D position of the camera.

Because this is a 3D location, you can get its image. You get a 3D position above the image camera, so you can get the Up direction of the image camera after subtracting the image camera position.

After you know the Position, Target, and Up directions of the image camera, you can create a View matrix.

The MirrorVector3 method mirrors Vector3 Based on the passed pluplane parameter. Because the mirror created in this tutorial is on the XY plane, you only need to change the Z Component symbol to find the image position:

private Vector3 MirrorVector3(Plane mirrorPlane, Vector3 originalV3) {     Vector3 mirroredV3 = originalV3;     mirroredV3.Z = -mirroredV3.Z;         return mirroredV3; } 

You will learn how to mirror any plane, but the mathematical principles in this method will distract you. Call the UpdateMirrorViewMatrix method from the Update method:

UpdateMirrorViewMatrix(); 

Draw the scenario you can see from the mirror

With the Image View matrix, you can use this matrix to draw the scenario you can see from the mirror. Because you need to draw a scenario twice (one in the mirror and one in the normal camera), the good idea is to refactor your code and place the scenario in an independent method:

Private void RenderScene (Matrix viewMatrix, Matrix projectionMatrix) {Matrix worldMatrix = Matrix. createScale (0.01f, 0.01f, 0.01f) * Matrix. createTranslation (0, 0, 5); myModel. copyAbsoluteBoneTransformsTo (modelTransforms); foreach (ModelMesh in myModel. meshes) {foreach (BasicEffect effect in mesh. effects) {effect. enabledefalightlighting (); effect. world = modelTransforms [mesh. parentBone. index] * worldMatrix; effect. view = viewMatrix; effect. projection = projectionMatrix;} mesh. draw ();} // draw other objects of your scene ...}

This method uses the View and Projection matrices as parameters to draw scenarios.

Add this code in the Draw method, enable the custom rendering target, and Draw the scene seen in the mirror to the rendering target through the image View matrix. Then, set the backup buffer to the current rendering target to close the custom rendering target (see tutorial 3-9) and store the content in the Custom rendering target to a texture:

//render scene as seen by mirror into render target device.SetRenderTarget(0, renderTarget); graphics.GraphicsDevice.Clear(Color.CornflowerBlue); RenderScene(mirrorViewMatrix, fpsCam.ProjectionMatrix); //deactivate custom render target, and save its contents into a texture device.SetRenderTarget(0, null); mirrorTexture = renderTarget.GetTexture();

Note:In the image scenario, you want to use the same projection matrix as the normal scenario to draw custom rendering targets. If, for example, the angle of your normal projection matrix is greater than that of the rendering target, the coordinates calculated in the shader will be combined.

After saving the texture, you will clear the backup buffer and then draw the scene in the normal camera. In other words, use the normal View matrix:

//render scene + mirror as seen by user to screen graphics.GraphicsDevice.Clear(Color.Tomato); RenderScene(fpsCam.ViewMatrix, fpsCam.ProjectionMatrix); RenderMirror(); 

The method called by the last line of code adds the mirror to the scene. In this example, the mirror is just a simple rectangle defined by two triangles. Its color samples from the texture that contains the scenes seen in the mirror. Because the mirror needs to display the correct texture, you should not simply place the image on the rectangle, but create a HLSL technique.

HLSL

First define the output structure of the XNA-HLSL variables, texture samplerzer, vertex and pixel shader:

//XNA interface float4x4 xWorld; float4x4 xView; float4x4 xProjection; float4x4 xMirrorView; //Texture Samplers Texture xMirrorTexture; sampler textureSampler = sampler_state {    texture = <xMirrorTexture>;     magfilter = LINEAR;     minfilter = LINEAR;     mipfilter=LINEAR;     AddressU = CLAMP;     AddressV = CLAMP; };struct MirVertexToPixel {    float4 Position : POSITION;     float4 TexCoord : TEXCOORD0; }; struct MirPixelToFrame {    float4 Color : COLOR0; } 

As usual, you need the World, View, and Projection matrices to calculate the 2D screen position of each 3D vertex. The View matrix of the image camera is also required to calculate the texture coordinates of each vertex in the mirror in vertex shader.

Technique needs to contain the texture of the scene viewed in the mirror. It uses this texture to obtain the color of each pixel in the mirror.

The output structure of vertex shader is the texture coordinate and the 2D screen coordinate of the current vertex. Pixel shader only calculates the pixel color.

Vertex Shader

As usual, vertex shader calculates the 2D screen coordinates of each vertex, which can be done by multiplying the 3D position by the WorldViewProjection matrix.

//Technique: Mirror MirVertexToPixel MirrorVS(float4 inPos: POSITION0) {    MirVertexToPixel Output = (MirVertexToPixel)0;     float4x4 preViewProjection = mul(xView, xProjection);     float4x4 preWorldViewProjection = mul(xWorld, preViewProjection);     Output.Position = mul(inPos, preWorldViewProjection); } 

In image technique, vertex shader also calculates the pixel corresponding to the vertex in xforwartexture for each vertex in the mirror. For example, if you want to find the pixel in xforwardedtexture that corresponds to the top left vertex of the mirror, the key to finding the answer is to look at the mirror from the mirror camera. You need to obtain the 2D coordinates corresponding to the vertex in the xforwardedtexture saved by the image camera. This is actually the 3D coordinates you have converted from the WorldViewProjection matrix of the image camera.

float4x4 preMirrorViewProjection = mul (xMirrorView, xProjection); float4x4 preMirrorWorldViewProjection = mul(xWorld, preMirrorViewProjection);Output.TexCoord = mul(inPos, preMirrorWorldViewProjection); return Output; 

Note:The word Mirror in the Code does not represent an additional matrix; it only indicates a matrix of the image camera. For example, xMirrorView does not mean that the Mirror matrix is multiplied by the View matrix, but the View matrix of the image camera.

Pixel Shader

Now in pixel shader, you have texture coordinates for the four vertices in the mirror. Which of the following is the scope you do not want. You know that the texture coordinates are between 0 and 1, as shown in the left figure of 3-25. The screen coordinate ranges from-1 to 1, as shown in the figure 3-25 on the right.

Figure 3-25 texture coordinates (left) and screen coordinates (right)

Fortunately, it is very easy to map from [-]. For example, you can divide it by 2 so that the range is [-0.5, 0.5], then add 0.5, and the range is [0, 1]. Besides, because you are processing a float4 (homogeneous) Coordinate, You need to divide them by the fourth coordinate before using the first three components. This is the first operation performed by pixel shader:

MirPixelToFrame MirrorPS(MirVertexToPixel PSIn) : COLOR0 {    MirPixelToFrame Output = (MirPixelToFrame)0;         float2 ProjectedTexCoords;     ProjectedTexCoords[0] = PSIn.TexCoord.x/PSIn.TexCoord.w/2.0f +0.5f;     ProjectedTexCoords[1] = -PSIn.TexCoord.y/PSIn.TexCoord.w/2.0f +0.5f;         Output.Color = tex2D(textureSampler, ProjectedTexCoords);         return Output; }

The "-" in the Code for calculating the second texture coordinate is required, because the scene needs to be drawn up and down to the frame buffer, so you need to adjust it. The last line of code queries the corresponding color in xforwardedtexture. This color is returned to pixel shader.

Note:The first two parts represent the 2D screen coordinates. You need to divide the first three parts by the fourth, but what is the third coordinate? It is actually a 2D depth. In other words, this value is between 0 and 1 in the graphics card z buffer, indicating that the vertex is in the near cut plane, and 1 indicates in the far cut plane. When pixel shader is called to calculate the pixel color, the video card first checks whether the pixel should be drawn based on the depth value of the pixel in the z buffer. For more information, see the last part of tutorial 2-1.

The following is the definition of technique:

technique Mirror {    pass Pass0     {        VertexShader = compile vs_1_1 MirrorVS();         PixelShader = compile ps_2_0 MirrorPS();     }}

Use Technique in XNA

You also need to define the DrawMirror method in the XNA project. This method uses the created technique to draw the rectangle:

private void RenderMirror() {    mirrorEffect.Parameters["xWorld"].SetValue(Matrix.Identity);     mirrorEffect.Parameters["xView"].SetValue(fpsCam.ViewMatrix);     mirrorEffect.Parameters["xProjection"].SetValue(fpsCam.ProjectionMatrix);     mirrorEffect.Parameters["xMirrorView"].SetValue(mirrorViewMatrix);     mirrorEffect.Parameters["xMirrorTexture"].SetValue(mirrorTexture);         mirrorEffect.Begin();     foreach (EffectPass pass in mirrorEffect.CurrentTechnique.Passes)     {        pass.Begin();         device.VertexDeclaration = new VertexDeclaration(device, VertexPositionTexture.VertexElements);         device.DrawUserPrimitives<VertexPositionTexture> (PrimitiveType.TriangleStrip, mirrorVertices, 0, 2);         pass.End();     }    mirrorEffect.End(); }

Set the World, View, and Projection matrices, as well as the xnet matrix and the x1_texture that contains the scenes in the image. The two triangles of the rectangle are drawn in the form of TriangleStrip. You need to import the. fx file in the XNA project and connect it to the mirrorEffect variable.

Any image plane

In the previous example, an image plane at a special position is selected, so it is easy to perform an image on a point. However, in actual situations, you also want to define any image plane. The MirrorVector3 method needs to be improved so that it can mirror any 3D point on any image plane:

private Vector3 MirrorVector3(Plane mirrorPlane, Vector3 originalV3) {    float distV3ToPlane = mirrorPlane.DotCoordinate(originalV3);     Vector3 mirroredV3 = originalV3 - 2 * distV3ToPlane * mirrorPlane.Normal;         return mirroredV3; } 

First, you want to know the shortest distance between the point and the plane. This can be calculated using the DotCoordinate method of the mirror plane (this shortest distance is the distance between the point and the plane ). If you multiply the plane normal by the distance and subtract the result vector from this point, you will be on the plane. But you don't want to be on the plane. You want to move twice the distance! Therefore, you need to double the result vector and subtract the vector from the initial coordinate.

This Code allows you to use a mirror based on any three points.

Define a mirror clipping plane

There is still a big problem: when objects are behind the mirror, these objects will be seen by the mirror camera and stored in the mirror texture. When Mirror pixel shader samples the color from this texture, these objects will be drawn to the Mirror, but these objects are actually behind the Mirror and should not be displayed.

The solution is to define a user clipping plane, which can be done by defining a plane and notifying XNA that the objects on the other side of the mirror do not need to be drawn. Of course, this plane should be the plane of the mirror, so the objects after the mirror do not need to be drawn.

However, the four coefficients of the cropping plane must be defined in the cropping space (so that your video card can easily determine which objects need to be drawn and which need to be clipped ). To map coefficients from a 3D space to a clipping space, you need to transform them through the inverse-transpose matrix of ViewProjection, as shown in the following code:

private void UpdateClipPlane() {    Matrix camMatrix = mirrorViewMatrix * fpsCam.ProjectionMatrix;     Matrix invCamMatrix = Matrix.Invert(camMatrix);     invCamMatrix = Matrix.Transpose(invCamMatrix);     Vector4 mirrorPlaneCoeffs = new Vector4(mirrorPlane.Normal, mirrorPlane.D);     Vector4 clipPlaneCoeffs = Vector4.Transform(-mirrorPlaneCoeffs, invCamMatrix);     clipPlane = new Plane(clipPlaneCoeffs); }

Calculate the inverse matrix. Then, accept the four coefficients defined by the mirror plane in the 3D space, map them to the clipping space through the reverse matrix, and then create the clipping plane using the result.

Note:"-" Indicates which plane needs to be removed. It is related to the normal direction of the plane, which is defined according to the sequence of the defined points p0, p 1, p2 and p3.

Because the clipPlane variable depends on viewMatrix, and viewMatrix needs to be updated when the camera position changes, the following method is called in the Update method:

UpdateClipPlane(); 

The next step is to pass the cropping plane to the video card, enable the plane before you draw the scene you see in the mirror, and remember to disable the plane before you draw the scene that a common camera sees, because the objects behind the mirror can be seen by ordinary cameras, they should be displayed:

//render scene as seen by mirror into render target device.SetRenderTarget(0, renderTarget); device.Clear(ClearOptions.Target | ClearOptions.DepthBuffer,Color.CornflowerBlue, 1, 0); device.ClipPlanes[0].Plane  = clipPlane; device.ClipPlanes[0].IsEnabled = true; RenderScene(mirrorViewMatrix, fpsCam.ProjectionMatrix); device.ClipPlanes[0].IsEnabled = false; 

Note:If you look closely at the image, the image in the mirror looks a little blurred compared to the original image. This is because the calculated texture coordinates usually do not exactly correspond to a pixel, so the video card will take the average of the nearest pixel, the average process corresponds to the fuzzy operation (see tutorial 2-12 ).

Code

To load technique, you need to load the. Fx file to an Effect variable. You also need to define the mirror:

private void InitMirror() {    mirrorVertices = new VertexPositionTexture[4];         int i = 0;     Vector3 p0 = new Vector3(-3, 0, 1);     Vector3 p1 = new Vector3(-3, 6, 0);     Vector3 p2 = new Vector3(6, 0, 0);     Vector3 p3 = p1 + p2 - p0;         mirrorVertices[i++] = new VertexPositionTexture(p0, new Vector2(0,0));     mirrorVertices[i++] = new VertexPositionTexture(p1, new Vector2(0,0));     mirrorVertices[i++] = new VertexPositionTexture(p2, new Vector2(0,0));     mirrorVertices[i++] = new VertexPositionTexture(p3, new Vector2(0,0));         mirrorPlane = new Plane(p0, p1, p2); } 

When the camera position changes, you need to update the preview viewmatrix and clipPlane variables, because they depend on the normal View matrix:

private void UpdateMirrorViewMatrix() {    Vector3 mirrorCamPosition = MirrorVector3(mirrorPlane, fpsCam.Position);     Vector3 mirrorTargetPosition = MirrorVector3(mirrorPlane,fpsCam.TargetPosition);     Vector3 camUpPosition = fpsCam.Position + fpsCam.UpVector;     Vector3 mirrorCamUpPosition = MirrorVector3(mirrorPlane, camUpPosition);     Vector3 mirrorUpVector = mirrorCamUpPosition - mirrorCamPosition;     mirrorViewMatrix = Matrix.CreateLookAt(mirrorCamPosition, mirrorTargetPosition, mirrorUpVector); }

private Vector3 MirrorVector3(Plane mirrorPlane, Vector3 originalV3) {    float distV3ToPlane = mirrorPlane.DotCoordinate(originalV3);     Vector3 mirroredV3 = originalV3 - 2 * distV3ToPlane * mirrorPlane.Normal;         return mirroredV3; }

private void UpdateClipPlane() {    Matrix camMatrix = mirrorViewMatrix * fpsCam.ProjectionMatrix;    Matrix invCamMatrix = Matrix.Invert(camMatrix);     invCamMatrix = Matrix.Transpose(invCamMatrix);     Vector4 mirrorPlaneCoeffs = new Vector4(mirrorPlane.Normal, mirrorPlane.D);     Vector4 clipPlaneCoeffs = Vector4.Transform(-mirrorPlaneCoeffs, invCamMatrix);         clipPlane = new Plane(clipPlaneCoeffs); } 

During the rendering process, you first draw the scene you see from the image camera to a texture, then clear the screen, and draw the scene you see from the normal camera. Then draw the mirror:

protected override void Draw(GameTime gameTime) {    //render scene as seen by mirror into render target     device.SetRenderTarget(0, renderTarget);     device.Clear(ClearOptions.Target | ClearOptions.DepthBuffer, Color.CornflowerBlue, 1, 0);         device.ClipPlanes[0].Plane = clipPlane;     device.ClipPlanes[0].IsEnabled = true;    RenderScene(mirrorViewMatrix, fpsCam.ProjectionMatrix);     device.ClipPlanes[0].IsEnabled = false;         //deactivate custom render target, and save its contents into a texture     device.SetRenderTarget(0, null);     mirrorTexture = renderTarget.GetTexture();         //render scene + mirror as seen by user to screen     graphics.GraphicsDevice.Clear(Color.Tomato);     RenderScene(fpsCam.ViewMatrix, fpsCam.ProjectionMatrix);     RenderMirror();         base.Draw(gameTime); } 

The mirror is drawn as a simple rectangle using mirror technique, whose color comes from the fill texture variable:

private void RenderMirror() {    mirrorEffect.Parameters["xWorld"].SetValue(Matrix.Identity);     mirrorEffect.Parameters["xView"].SetValue(fpsCam.ViewMatrix);     mirrorEffect.Parameters["xProjection"].SetValue(fpsCam.ProjectionMatrix);     mirrorEffect.Parameters["xMirrorView"].SetValue(mirrorViewMatrix);     mirrorEffect.Parameters["xMirrorTexture"].SetValue(mirrorTexture);         mirrorEffect.Begin();     foreach (EffectPass pass in mirrorEffect.CurrentTechnique.Passes)     {        pass.Begin();         device.VertexDeclaration = new VertexDeclaration(device, VertexPositionTexture.VertexElements);         device.DrawUserPrimitives<VertexPositionTexture> (PrimitiveType.TriangleStrip, mirrorVertices, 0, 2);         pass.End();     }    mirrorEffect.End(); }

For each vertex in the mirror, vertex shader calculates the location of the 2D screen and the position corresponding to the xforwardedtexture:

MirVertexToPixel MirrorVS(float4 inPos: POSITION0) {    MirVertexToPixel Output = (MirVertexToPixel)0;         float4x4 preViewProjection = mul(xView, xProjection);     float4x4 preWorldViewProjection = mul(xWorld, preViewProjection);     Output.Position = mul(inPos, preWorldViewProjection);     float4x4 preMirrorViewProjection = mul (xMirrorView, xProjection);     float4x4 preMirrorWorldViewProjection = mul(xWorld, preMirrorViewProjection);         Output.TexCoord = mul(inPos, preMirrorWorldViewProjection);         return Output; } 

Pixel shader uses the coordinate division by the Second coordinate to map the position from the [-] rendering target range to the [0, 1] texture coordinate range. Use the result texture coordinates to sample the color of the xforwardedtexture position and return the result:

MirPixelToFrame MirrorPS(MirVertexToPixel PSIn) : COLOR0 {    MirPixelToFrame Output = (MirPixelToFrame)0;         float2 ProjectedTexCoords;     ProjectedTexCoords[0] = PSIn.TexCoord.x/PSIn.TexCoord.w/2.0f +0.5f;     ProjectedTexCoords[1] = -PSIn.TexCoord.y/PSIn.TexCoord.w/2.0f +0.5f;     Output.Color = tex2D(textureSampler, ProjectedTexCoords);         return Output; }