"Image processing" ISP image sensor camera principle

1. Color Filter ARRAY-CFA

With the popularization of digital camera and mobile phone, Ccd/cmos image sensor has been widely concerned and applied in recent years. Image sensors generally use a certain pattern to capture image data, commonly used in BGR mode and CFA mode. BGR mode is an image data mode that can be processed directly, such as display and compression, which is determined by the values of R (red), G (green) and B (blue ) to determine 1 pixels , such as SUPER CCD image sensor used by Fuji Digital camera. The advantage of this mode is that the image data produced by the sensor can be displayed directly without interpolation, and the image effect is the best, but the cost is high, and it is often used in professional cameras. General digital camera Sensors (CCD or CMOS) about the total cost of 10%~25%, in order to reduce costs, reducing volume, digital cameras on the market mostly use the CFA model , that is, the surface of the pixel array is covered by a layer of color filter array (color Filter ARRAY,CFA), there are a variety of color filter array, now the most widely used is the Bayer format Filter array , to meet the GRBG law, green pixels are twice times the number of red or blue pixels, This is because the peak of the visible spectral sensitivity of the human eye is in the middle band, which corresponds to the green spectral composition.

is a CFA -based image sensor with an effective resolution of 640 x 480, which uses only 1 values from the R, G, and B 3 values to represent 1 pixel points . In this way, each pixel can only capture one of the three primary colors r,g,b, and the other two color values are missing, and a mosaic image is obtained. in order to obtain full-color images, it is necessary to use the color information of the surrounding pixels to estimate the missing two other colors , which are called color interpolation , also known as color interpolation or de-mosaic .

is a 8 x 8 pixel-sized CFA pattern image data array with 1 squares representing 1 pixels, and a numeric subscript for R, G, and B representing its position in the 8 x 8 image array. Due to the relatively simple structure of the Image Color filter array used in CFA mode, the resulting image data is only 1/3 of the data of the primary primary color information of the original image, so the cost is lower. However, in the CFA mode image data compared with the BGR mode image data, the lack of 2/3 of the image color information, so to the CFA mode image data display, compression and other subsequent processing, it is necessary to pre-interpolation operation to restore the CFA mode image data missing 2/3 color information, thereby The CFA mode image data is reconstructed to match the image data of the BGR mode image. The more common is bilinear interpolation algorithm: The algorithm in a pixel point of a color value of the interpolation operation, the pixel adjacent pixel points corresponding to the color value of the arithmetic average to estimate.

2. Image denoising

In the process of image acquisition and transmission, the image quality is often affected by various noises and decreased. Because the acquisition and a variety of components susceptible to strong interference will produce impulse noise, due to unstable lighting, lens dust and non-linear channel transmission caused by the degradation of the image will produce different kinds of noise its main impact on human visual effects, making it difficult to identify some details of the image, the other noise to some image processing algorithms bring serious impact , such as gradient operators, due to the introduction of some object-independent points, so that the use of useless information caused more serious consequences, interfering with the observable information of the image. The noise discussed here is confined to the noise pollution caused by the image sensor to obtain the image data, because of this time the amount of data is less, the noise directly affects the interpolation algorithm behind, and the details of the image can not be reflected, both affect the image interpolation effect, also affect the human visual experience. Therefore, the removal of noise in image processing is a very important link.

3. Auto Focus

Autofocus is designed to obtain a higher resolution image. There are two kinds of focusing methods commonly used, one is the traditional focusing method and the other is the image focusing method based on digital image processing method. In the traditional way, automatic focusing is achieved by means of infrared or ultrasonic ranging. This approach requires the installation of transmitters and receivers, which increases the cost of the camera and the ultrasound does not automatically focus on the object behind the glass. This type of focus has been limited in some cases. Therefore, in the increasingly integrated, miniaturized, low-cost applications, based on digital image processing of the auto-focus method is more advantageous.

According to the lens Imaging analysis, the optical transfer function of the lens can be approximated as a Gaussian function, and its effect is equivalent to a low-pass filter. The higher the Defocus amount, the lower the cutoff frequency of the optical transfer function. From the frequency domain, the amount of defocus increases and the loss of high frequency energy of the image makes the details of the image gradually blurred. From the airspace, the increase of defocus, the more dispersed the intensity distribution function of the point light source imaging, the larger the distinguishable imaging spacing, the overlapping of image pixels and the serious loss of image detail. Therefore, the image definition evaluation function is based on the high frequency energy of the image edge.

In the digital processing method, the key of auto focus is to construct the image definition evaluation function. It has been proposed that the evaluation function of image sharpness includes gray variance, gradient energy, entropy function and some frequency domain function methods. The image definition evaluation function must have good single-peak and sharp, and the calculation amount is moderate, so that the precision focus can be realized quickly.

4. Auto Exposure

Exposure is the amount of physical quantity used to calculate the luminous flux from the scene to the camera. The image sensor only obtains the correct exposure, can obtain the high quality photograph. Overexposed, the image looks too bright to be exposed, and the image looks too dark. The size of the luminous flux reaching the sensor is determined mainly by two factors: the length of exposure time and the size of the aperture.

Automatic exposure using aperture, mainly based on the scene to control the aperture size, so that the amount of light is maintained within a certain range. Exposure control via aperture is cost-efficient. The mainstream technology that is now seen in the market for mid-low-end cameras is automatic exposure by adjusting exposure time.

At present, there are two methods of automatic exposure control algorithm, one is to use the reference luminance value, the image is evenly divided into a number of sub-images, the brightness of each sub-image is used to set the reference luminance value, the brightness value can be set by the speed of the shutter to obtain. Another method is to conduct exposure control by studying the relationship between luminance and exposure values under different illumination conditions. Both of these methods have studied a large number of image examples and many different lighting conditions. And all of them need the image database which is collected under different illumination conditions. The actual AE research needs to solve the following problems, the first is to determine whether the image needs automatic exposure, followed by automatic exposure, how to adjust the photoelectric conversion digital signal to find the automatic exposure ability compensation function, and finally adjust to what extent the most appropriate.

5. Gamma correction

In the video capture display system, the conversion characteristics of photoelectric conversion (Ccd/cmos) and electro-optic conversion (CRT/LCD) are nonlinear. These nonlinear periods have a power function that can reflect the characteristics of a nonlinear device. This characteristic is called the gamma characteristic, in the video because of the existence of the gamma characteristic, can cause the image signal brightness distortion, reduces the communication quality, affects the user experience. This distortion is therefore compensated for by gamma correction.
The nonlinearity of the photoelectric converter will cause the nonlinear distortion of the image, and the nonlinear distortion of the image is mainly reflected in the distortion of the gray level, that is, the image brightness is compressed and expanded, and its image is characterized by the appearance of being bleached or too dark. The gamma characteristic size of the camera/camera is typically 0.4-0.7, and the gamma characteristic size of the display is generally between 1.3 and 2.5.
The specific implementation methods of gamma correction are various, and the simpler implementation method is the check table method. Gamma correction is divided into two steps. First, a gamma correction table is established for the device used, and then the gamma correction data is obtained based on the input pixel values.

6. White balance

White balance, the literal understanding is the white balance. With the knowledge of color to explain, white refers to the light reflected in the eyes of the human eye due to the same proportion of blue, green, red and a certain degree of brightness of the visual response. White light is composed of red, orange, yellow, green, cyan, blue, violet seven kinds of light, and these seven kinds of color is red, green, blue three primary colors in different proportions of the formation, when a light in the three primary components of the same proportion of the time, the habit of people called the color, black, white, gray, gold and silver the reflection of the The popular understanding of white is the brightness that does not contain the color component. The human eye sees the white or other color root object itself the natural color, the light source color temperature, the object's reflection or the transmission characteristic, the human eye visual induction and so on many factors are related, for a simple example, when has the shade illumination to the Achromatic object, the object reflected light color and the incident light color is the same, the red illumination under the white object When two or more shades are illuminated at the same time, the object color is additive-colored, such as red and green light at the same time the White object, the object will appear yellow. The color of the object is a subtractive effect when light is illuminated on a colored object. If the yellow object is red under the magenta light, it is green under the cyan illumination, and appears gray or black under the blue light.
Because the human eye has the unique adaptability, sometimes cannot discover the color temperature change. For example, under the tungsten filament for a long time, and will not feel the white light under the red, if suddenly the fluorescent lamp changed to tungsten lighting, will be aware of the color of white paper is red, but this feeling can only last a while. The camera is not as adaptable as the human eye, so if the color adjustment of the camera is inconsistent with the color temperature of the scene illumination, the bias will occur. White balance is aimed at different color temperature conditions, by adjusting the internal color circuit of the camera to offset the color of the image, more close to the visual habits of the human eye. White balance can also be simply understood to be in any color temperature conditions, the camera shot by the standard white after the adjustment of the circuit, so that the image is still white.

7. Color space

Color space is also known as a color model (also known as color space or color system), and its purpose is to describe the color in a generally acceptable way under certain criteria. In essence, the color model is the elaboration of coordinate system and subspace. Each color in the system has a single point representation. In color image processing, it is important to choose the appropriate color model. From the point of view of application, there are two kinds of color models proposed by people. A class for hard devices such as color displays or color printers (but can be related to specific devices or independent of specific devices) such as RFB, CMY, YUV models. Another type of application for visual perception or for the purpose of color processing analysis, such as color graphics in animation, a variety of image processing algorithms, such as HSI, HSV model.

8. YUV Color Space

The luminance signal (y) and chroma signal (U,V) are independent of each other, that is, the Y-signal component of the black-and-white grayscale image with the u,v signal is composed of another two monochrome graphs are independent of each other. Because the y,u,v is independent, these monochrome graphs can be encoded separately. The black and white machine can receive the color TV signal, which is the independence between YUV components. The advantage of using YUV color space is that the human eye has a lower resolution than black-and-white image for color image details, so the color difference signal, U, V, can use "large area coloring principle". That is, with the luminance signal y transmission details, with the chromatic aberration signal u, V for large area coloring. Therefore, the clarity of the color signal is guaranteed by the bandwidth of the luminance signal, and the bandwidth of the chromatic aberration signal is narrowed. It is for this reason, in the multimedia computer, the YUV color space, the digital representation, usually using Y:U:V = 8:4:4, or y:u:v = 8:2:2. For example 8:2:2 specific practice is: to the luminance signal y, each pixel with 8-bit 2 binary number (can have 256 levels of brightness), and the U, V color signal every 4 pixels with a 8-digit point, that is, the picture of the particle is thicker, but this can save storage space, A pixel with a 24-bit compression to 12-bit representation, saving 1/2 of storage space, and the human eye basically does not feel the loss of this detail, which is actually a method of image compression technology.

The YUV format typically has two main classes: the packaged (packed) format and the planar format. The former stores YUV components in the same array, usually several adjacent pixels to form a macro pixel (Macro-pixel), and the latter uses three arrays to store the YUV three components separately, as if it were a three-dimensional plane.

9. Image Scaling

Image scaling (Scaler) technology, also known as image scale conversion, image resampling and image resolution conversion technology, is a key technology in video image processing, and is widely used to realize the conversion of FPD image resolution. For example, a high-definition digital TV will need to be converted into an HDTV (1920x1080) format after it receives a standard-definition digital TV signal in NTSC or PAL format for display on HDTV TVs, plus a progressive display of plasma (PDP) TVs, TFT-LCD TVs, It is necessary to improve the received image resolution to match the physical resolution of the LCD screen, in order to display the video image on the terminal, therefore, the quality of the Scaler performance will directly determine the display image.

Image scaling can be understood as the resampling process of image, the key is to use continuous model function to fit the original discrete image, after the continuous model parameters are obtained, the continuous image is resampled according to the desired magnification, and a discrete image conforming to the target resolution is obtained. The essence of Digital Image resampling is the process of interpolating discrete image points. According to the sampling/reconstruction theory, the ideal interpolation kernel is the sinc function, but it is not physically achievable. The usual interpolation kernel functions are finite-width interpolation functions that approximate the sinc function. Nearest neighbor method is the simplest scaling algorithm, but it causes the processed image to produce obvious jagged edges and mosaic effects. Bilinear interpolation can solve the problem of nearest neighbor domain method, but it is easy to blur the edge of image. As an improvement, the sinc kernel function is proposed, and the high-order interpolation algorithm is obtained, such as cubic interpolation and high-order spline interpolation.

"Image processing" ISP image sensor camera principle