Multi-Level Image Processing (FrameBuffer) in WebGL ),

Multi-Level Image Processing (FrameBuffer) in WebGL ),

When using webgl, we usually want to perform multi-level processing on texture and display it on the surface.

For more precise edge detection, gray-scale shader, fuzzy shader, and edge shader must be used for processing, and each processing object is the last processed texture, this will overwrite and save the processed results.

 

Among the many webgl libraries, there is a direct option rederToTarget to render the texture after the shader processing and overwrite the original texture. How does one complete this step?

This leads to the main character of this article-FrameBuffer

 

What is FrameBuffer?

 

FBO (Frame Buffer Object) is an extension recommended for rendering data to a texture Object.

FrameBuffer is like a webgl display container. We usually use gl. drawArrays or gl. drawElements draws the object in the default window. When we specify a FrameBuffer as the current window, we use these two methods to draw the object, the object is drawn in the specified FrameBuffer.

 

Use of FrameBuffer

 

Internalformat, int x, int y, sizei width,
Sizei height, int border );
Target: TEXTURE_2D, TEXTURE _

FBO creation:

// Create a Framebuffervar fb = gl. createFramebuffer (); // bind the fb to the current window gl. bindFramebuffer (gl. FRAMEBUFFER, fb );

 

In this way, we have created a new buffer that can be drawn and it will not be displayed.

But can this be done? What we think of is to render the texture after the shader rendering and hand it to the next shader. Then the method framebufferTexture2D is introduced.

Reference from OpenGL ES Reference Pages about FramebufferTexture2D:
 
To render directly into a texture image, a specified image from a texture object can be attached as one of the logical buffers of the currently bound framebuffer object by calling the command
To render the image directly to a texture image, the image specified in a texture object can be bound to a logical cache on the currently used FBO using the following method.
Void FramebufferTexture2D (enum target, enum attachment, enum textarget, uint texture, int level );
 
Target:
• FRAMEBUFFER
Attachment:
• If attachment is COLOR_ATTACHMENT0, then image must have a colorrenderable internal format. (color)
• If attachment is DEPTH_ATTACHMENT, then image must have a depthrenderable internal format. (depth)
• If attachment is STENCIL_ATTACHMENT, then image must have a stencilrenderable internal format. (Template)
Textarget:
• TEXTURE_2D (two-dimen1_texture)
• TEXTURE_CUBE_MAP_POSITIVE_X (three-dimenure + x texture)
• TEXTURE_CUBE_MAP_POSITIVE_Y (three-dimenure + y texture)
• TEXTURE_CUBE_MAP_POSITIVE_Z (three-dimenure + z texture)
• TEXTURE_CUBE_MAP_NEGATIVE_X (three-dimen1_x texture)
• TEXTURE_CUBE_MAP_NEGATIVE_Y (three-dimen1_- y texture)
• TEXTURE_CUBE_MAP_NEGATIVE_Z (three-dimen1_- z texture)
Texture:
Texture object
Level:
Specifies the mipmap level of the texture image to be attached to the framebuffer and must be 0.

 

We use this method for binding (this article only describes the binding of colors, and tries to be similar to the template, but there are differences, not discussed here)

// Create a texture object var texture = gl. createTexture (); // use the following settings to create texture. In this way, texture settings allow us to process images of any size in gl. bindTexture (gl. TEXTURE_2D, texture); gl. texParameteri (gl. TEXTURE_2D, gl. TEXTURE_WRAP_S, gl. CLAMP_TO_EDGE); gl. texParameteri (gl. TEXTURE_2D, gl. TEXTURE_WRAP_T, gl. CLAMP_TO_EDGE); gl. texParameteri (gl. TEXTURE_2D, gl. TEXTURE_MIN_FILTER, gl. NEAREST); gl. texParameteri (gl. TEXTURE_2D, gl. TEXTURE_MAG_FILTER, gl. NEAREST );
Var fb = gl. createFramebuffer (); gl. bindFramebuffer (gl. FRAMEBUFFER, fb); // use this method to bind the color value of texture to FBO gl. framebufferTexture2D (gl. FRAMEBUFFER, gl. COLOR_ATTACHMENT0, gl. TEXTURE_2D, texture, 0 );

 

After binding, when we execute gl. drawArrays or gl. when the drawElements method is used, it will be directly rendered to the currently bound FBO, and FBO is bound to the texture color, so the color will also be rendered to texture during painting.

In this way, we can use two fbrs for queue processing:

OriginalImage --> texture1

Texture1 --> gray --> texture2

Texture2 --> blur --> texture1

Texture1 --> edge --> texture2

 

The specific implementation process is as follows:

var FBOs = [],    textures = [];    for(var i = 0; i < 2; i++){    var texture = gl.createTexture();    gl.bindTexture(gl.TEXTURE_2D, texture);    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);    var fb = gl.createFramebuffer();    gl.bindFramebuffer(gl.FRAMEBUFFER,fb);    gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, texture, 0);        //store corresponding texture and fb    textures.push(texture);    FBOs.push(fb);}gl.bindTexture(gl.TEXTURE_2D, originalImageTexture);for(var i = 0; i < 3; i++){    switch(i){case 0:            //set gray shader to current shader program            //handle arguments to vs shader and fs shader            break;        case 1:            //set blur shader to current shader program            //handle arguments to vs shader and fs shader            break;        case 2:            //set edge shader to current shader program            //handle arguments to vs shader and fs shader            break;    }        gl.bindFramebuffer(gl.FRAMEBUFFER, FBOs[i%2]);    //set the viewport fits the images size    gl.viewport(0, 0, imgWidth, imgHeight);    gl.drawArrays(....); //or gl.drawElements(....);        //set the rendered texture to current texture for next frambuffer using    gl.bindTexture(gl.TEXTURE_2D, texture[i%2]);}

 

The complete process is:

originalTexture --> gray program --> set FBO1 --> draw --> FBO1 --> set texture1texture1 --> blur program --> set FBO2 --> draw --> FBO2 --> set texture2texture2 --> edge program --> set FBO1 --> draw --> FBO1 --> set texture1

 

In this process, FBO1 and texture1 are color rendering bound, so after set FBO1 is rendered, it will be directly rendered to texture1

When we finish the entire painting, we need to display the processed image normally, then we need to jump out of FBO:

//set FBO to null to use default framebuffergl.bindFramebuffer(gl.FRAMEBUFFER, null);

 

Other functions of FrameBuffer

 

Gl. readPixels

Read pixel color data from FrameBuffer

Reference from 《webgl_2.0_reference_card》/《OpenGL ES Reference Pages about readPixels》:
 
Pixels in the current framebuffercan be read back into an ArrayBufferView object.
void readPixels(int x, int y, long width, long height,enum format, enum type, Object pixels)
 
x,y
• Specify the window coordinates of the first pixel that is read from the frame buffer. This location is the lower left corner of a rectangular block of pixels.
width,height
 
• Specify the dimensions of the pixel rectangle. width and height of one correspond to a single pixel.
format
• Specifies the format of the pixel data. The following symbolic values are accepted  RGBA in WebGL
type
• Specifies the data type of the pixel data. Must be UNSIGNED_BYTEin WebGL
pixels
　　• Returns the pixel data.
 

 

During use, we need to create a pixels object to store data.

//using ArrayBufferView to store pixels data only, Unit8Array is the best because each color data is a bytevar pixels = new Uint8Array(ImageWidth * ImageHeight * 4);gl.readPixels(0, 0, ImageWidth, ImageHeight, gl.RGBA, gl.UNSIGNED_BYTE, pixels);

 

In this way, we can obtain the color data of the entire FBO.

 

Gl. CopyTexImage2D Gl. CopyTexSubImage2D

Both functions are used to copy data from FBO to the currently bound texture

CopyTexImage2D method:

Reference from 《OpenGL ES Reference Pages about CopyTexImage2D》:
 
copy pixels into a 2D texture image
void CopyTexImage2D(enum target, int level,enum internalformat, int x, int y, sizei width,sizei height, int border);
 
target:
• TEXTURE_2D
• TEXTURE_CUBE_MAP_POSITIVE_{X, Y, Z},
• TEXTURE_CUBE_MAP_NEGATIVE_{X, Y, Z}
internalformat:
• ALPHA
• LUMINANCE
• LUMINANCE_ALPHA
• RGB
• RGBA
x,y
Specify the window coordinates of the lower left corner of the rectangular region of pixels to be copied.
width
Specifies the width of the texture image. Must be 0 or 2 n + 2  border for some integer n.
height
Specifies the height of the texture image. Must be 0 or 2 m + 2  border for some integer m.
border
Specifies the width of the border. Must be either 0 or 1.

CopyTexSubImage2D method:

Reference from 《OpenGL ES Reference Pages about CopyTexSubImage2D》:
 
copy a two-dimensional texture subimage
void CopyTexSubImage2D(enum target, int level, int xoffset,int yoffset, int x, int y, sizei width, sizei height);
 
target:
• TEXTURE_2D
• TEXTURE_CUBE_MAP_POSITIVE_{X, Y, Z},
• TEXTURE_CUBE_MAP_NEGATIVE_{X, Y, Z}
level:
Specifies the level-of-detail number. Level 0 is the base image level. Level n is the nth mipmap reduction image.
xoffset:
Specifies a texel offset in the x direction within the texture array.
yoffset:
Specifies a texel offset in the y direction within the texture array.
x,y:
Specify the window coordinates of the lower left corner of the rectangular region of pixels to be copied.
width:
Specifies the width of the texture subimage.
height:
Specifies the height of the texture subimage.

The differences between the two methods are as follows:

 

CopyTexSubImage2D adds offset to CopyTexImage2D to change the copy region.

The final copying region is [x, xoffset + width-1] and [y, yoffset + height-1].

 

CopyTexImage2D has an internelformat parameter greater than CopyTexSubImage2D to control the types of pixel data replication.

 

Conclusion:

With flexible texture operations, we can make more interesting things, And framebuffer is also a very important role in it.

  Appendix:

WebGL-1.0 reference card: http://files.cnblogs.com/files/zhiyishou/webgl-reference-card-1_0.pdf

OpenGL-ES-2.0 reference card: http://files.cnblogs.com/files/zhiyishou/OpenGL-ES-2_0-Reference-card.pdf

OpenGL-ES-2.0 Reference Manual: https://www.khronos.org/opengles/sdk/docs/man/

 

The end.