Analysis of automatic driving binocular vision Solution

Over the next decade, in order to complete the evolutionary path from perception + early warning to decision + execution, the Advanced Auxiliary Driving System (ADAS) will access more sensors to achieve more complex computations with higher security.

A brief introduction to binocular vision

Compared with single visual perception, the key difference between binocular vision (Stereo Vision) is that two cameras can be used to image the same target from different angles, so as to obtain parallax information and calculate target distance. specifically to the visual Adas application, if the use of Monocular camera, in order to identify pedestrians and vehicles and other targets, usually require large-scale data acquisition and training to complete machine learning algorithm, and difficult to identify irregular objects, while the use of millimeter-wave radar and lidar for ranging accuracy is higher, but the cost and difficulty is higher. Therefore, the biggest advantage of binocular vision is to maintain low development cost under the premise of achieving a certain precision target recognition and ranging, complete FCW (Forward collision warning) and other Adas functions.

The basic principle of binocular vision ranging is not complicated, as shown in the following figure, p is the target point, and its imaging points on the left and right two cameras (lens center respectively A and b) are respectively E and F, the parallax of P point in two cameras is D=EC+DF. According to the similarity between the Triangle Ace and POA and the triangle BDF and Pob, the d= (FQ)/z is deduced, in which F is the camera focal length, q is the distance between the optical axis of the two cameras, and z is the distance from the target to the camera plane. Then the distance z= (FQ)/d, and F and Q can be considered as fixed parameters, so the Parallax signal D can be obtained from the distance Z.

Principle of binocular vision ranging

According to the principle of binocular vision ranging, the realization process is divided into five steps: Camera calibration, image acquisition, image preprocessing, feature extraction and stereo matching, and three-dimensional reconstruction. The camera calibration is to obtain the internal and external parameters and distortion coefficients of the camera, can be done off-line, while the synchronization of left and right camera image acquisition, the quality and consistency of image preprocessing, as well as the real-time requirements of stereo matching (obtaining parallax information) and three-dimensional reconstruction (obtaining distance information) algorithm, This paper presents a challenge to realize binocular vision Adas on embedded platform.

Monocular and Binocular Detailed comparison

Vision Scheme to complete the Adas task, generally to achieve ranging (this vehicle and the distance between the obstacles) and identify (what is the obstacle) two work. According to the different car camera module, the current mainstream Adas camera can be divided into monocular and binocular two technical routes.

The algorithm of the Monocular camera is to identify the rear range first, then identify it by image matching, and then estimate the obstacle and the vehicle time according to the image size and height. In the identification and estimation phase, it is necessary to compare with the established sample database. To identify all kinds of cars, we need to build a database of models, to identify elk, we need to build elk database.

The algorithm of binocular camera is the first to identify: first, using parallax directly to measure the distance between the object and the car, the principle and human eyes similar. When two eyes look at the same object, there will be parallax, that is to close the left and right eyes to see the object, will find the sensory displacement. This kind of displacement can further measure the distance of the target object. Then in the recognition phase, the binocular still needs to use the same Monocular feature extraction and depth learning algorithms to further identify the obstacles in the end.

Because the more distant disparity, the industry has the view that the binocular in 20 meters has a clear advantage, in the distance of 20 meters, parallax reduces the difficulty of ranging, you can use a high pixel camera and better algorithms to improve ranging performance, which is difficult and core competitiveness.

Binocular lens spacing and ranging are two of these parameters, the smaller the lens spacing, the closer the detection distance, the larger the lens spacing, the farther the detection distance. Consider the interior of the car and Adas needs, small size long distance binocular products are more popular.

The performance requirements and costs of the entire camera module are higher because of the addition of a lens, which brings more computation. And in both of the calibration work, the binocular is more complex than the monocular. and the choice of binocular approach to the market can not completely bypass the problem of the single eye solution, in the second stage, you still need a large database, still need to polish the algorithm.

Single Binocular Comparison

In addition to the single binocular, there are many cameras made up of the platform. In some scenarios, a telephoto and wide-angle camera is used in conjunction with the Adas main camera, taking into account the surrounding environment and the distant object detection. For example, the Mobileye program is described below.

There is also a situation where the Adas function is superimposed on the scan platform. For example, to look around to do lane departure warning (LDW), with a single eye to achieve this function than have a certain advantage. In heavy rain or in front of the strong light source, the front-view camera may not see the lane line, looking at the camera oblique down the lane line and can provide a number of angles, the basic will not be affected by the surface water reflection, function can be more stable than the previous view. But at the same time also want to consider the side without headlight irradiation, the night performance of the camera.

These schemes have no essential difference in the technical route and Monocular, more based on different platform, play the advantages of different types of camera module distribution tasks, or provide more perspective to solve the situation of the single eye in some complex environment.

Scenario Example: Binocular vision Adas solution based on s32v234

Tu-zhi-pu visual Adas Special soc:s32v234 structure chart

S32V234 employs 4 arm cortex A53 as the core CPU to achieve higher performance power consumption ratios. In addition, s32v234 contains an ARM cortex M4 as a chip MCU, mainly for real-time control of critical IO (such as CAN-FD), and supports Autosar operating systems.

The chip integrates two-channel MIPI-CSI2 and two-channel 16bit parallel Camera interface, as well as Gbit Ethernet controller, which offers a variety of options for image sensor input. At the same time, a programmable image signal processing (ISP) hardware module is included inside the chip. With the embedded ISP, the external matching image sensor can output raw data, thus reducing the material cost and saving the space size. In addition, the chip also contains two visual acceleration engines called APEX2CL. Each APEX2CL has 64 local computing units (CU) and is equipped with local memory to accelerate the process of image recognition through SIMD/MIMD (single instruction multiple data/multiple instruction multiple data) processing.

It is also worth noting that, in view of the stringent requirements for security and reliability of the ADAS system, s32v234 has added security mechanisms such as ECC (Error checking and correction), FCCU (fault collection and control unit), m/l BIST (Memory/logic built-in self-test) in design. To meet the needs of ISO26262 ASIL b~c.

The s32v234 chip has two MIPI-CSI2 camera interface, each maximum can provide 6Gbps transmission rate, can be used for left and right two camera video input. Since the two cameras input two MIPI channels, we need to consider the synchronization problem between them. With the help of external image sensors, s23v324 can support different synchronization modes. As shown in Figure 3, the image sensor usually has a field synchronous signal (VSYNC) and a line synchronization signal (HSYNC) for signal synchronization: When the two-way camera is working in master-slave mode, the synchronous signal is sent by master to slave; when both cameras are working in the mode, Synchronization signals can be generated by the S32V234 internal timer and sent to both cameras.

Synchronization scheme of Binocular camera

After the s32v234 obtains the image signal of the external camera, it can be preprocessed by the internal ISP. The ISP module contains a number of processing units optimized for ISP functions, using on-chip SRAM to cache input signals and intermediate processing results, and using a special coprocessor based on ARM Cortex m0+ to manage the sequence of ISP processing units, so as to achieve pixel level processing of image signals. Because the ISP is located inside the chip and can be flexibly programmed, it can not only save the cost of the external ISP of binocular camera, but also its computing resources and bandwidth can support the real-time processing of the dual-channel up to 1080P@30FPS image signal, which guarantees the quality and consistency of the two-channel image signal.

In the application of binocular vision Adas, the greatest challenge comes from the huge amount of computation required for stereo matching and three-dimensional reconstruction of two-way images. Taking the application of FCW as an example, it is required that the extraction of parallax signal has sufficient precision to ensure the range accuracy, and the processing frame frequency is maintained to ensure the response speed of early warning, so the embedded platform is required to have sufficient processing ability. s32v234 Integrated image acceleration engine APEX2 structure as shown in Figure 4, its parallel computing structure and special DMA design to ensure that the image signal has a very high processing efficiency. Specifically, after the ISP to the image signal preprocessing to send the DDR,APEX2 engine after the image segmentation through the dedicated DMA into each CU corresponding to the local memory Cmem, and stereo matching block matching (blocks Matching) and other algorithms can be in different cu in parallel processing, The processed data is sent back to the DDR via DMA, which is further processed by the CPU (such as generating an alert signal), or sent to a dedicated DCU (Display control unit) module output.

APEX2 architecture and image processing diagram

To sum up, the s32v234 based binocular vision Application Data flow as shown in Figure 5. In this application, the data flow flows according to the ISP-APEX2-DCU direction, A53 as the main control CPU to complete the logical control and the necessary data processing. Through this kind of pipelining, we can make full use of each part of the computing resources and improve the efficiency of calculation.

Binocular vision Data flow based on s32v234

Using s32v234 Development Board to build binocular vision platform, the two-channel 720P@30FPS video signal processing, the output results shown in Figure 6. The distance between the target and the camera is 1m,2m,3m from the top to bottom three images, and the result shows the target distance with the change of the warm and cold tones. The results show that the s32v234 can deal with binocular vision signals in real time, get the three-dimensional ranging result correctly, and can meet the needs of binocular vision Adas system with the safety design of the chip.

A gentle sweep welcome attention ~