Fs_s5pc100 Linux camera driver development (1)

Author: Liu Hongtao,Hua Qing vision embedded college lecturer.

Note:
Understanding the camera driver requires four prerequisites:
1) The basic working principle of the camera and the working principle of the camera controller integrated with s5pc100
2) working principles of platform_device and platform_driver
3) Linux kernel v4l2 driver architecture
4) Linux kernel I2C driver architecture

1. How cameras work

Ov9650/9655 is an image sensor chip with a CMOS interface. It can sense external visual signals, convert them to digital signals, and output them. The following figure shows how it works:

We need to provide the camera with a clock through xvclk1. The Reset is a reset line, and pwdn should always be low when the camera is working. Href indicates the line reference signal, pclk indicates the pixel clock, and vsync indicates the field synchronization signal. Once the camera provides a clock and resets the camera, the camera starts to work and transmits Digital Image Signals synchronously through href, pclk, and vsync. Data is transmitted through D0 ~ D7 data lines are concurrently sent.

The image format transmitted from the ov9650 to the outside is YUV, and YUV is a compressed image data format, which also contains many specific formats, our camera corresponds to YCbCr (8 bits, interpolated color ). make sure you understand the format. The format set in the driver later must be consistent with this format.

Many registers need to be configured in the ov9650. to configure these registers, You need to configure them through the sccb bus in the chip. Sccb is actually a weakened I2C bus. We can directly connect the camera to the s5pc100 I2C controller and use the I2C bus to read and write registers. Of course, you can also directly use gpio to simulate I2C. Both operation modes are implemented in our driver code.

Data collected from ov9650 cannot be directly sent to the CPU for processing. The s5pc100 chip integrates the camera controller, fimc (fully interactive mobile camera ). The camera must first transmit the image data to the Controller, and then process it with the CPU after the Controller completes processing (cropping and pulling, directly previewing or encoding.

In fact, fimc provides the clock required for camera operation.

2. driving development ideas

Because the driver is a bridge between hardware and software, we need to understand the two aspects of camera driver development: first, the hardware interface of the camera, that is, how it is connected to the chip, how to control it, how to reset the camera, and how to transmit the data format. Second, the software interface of the camera. The camera in the Linux kernel is a standard v4l2 device, but this camera is just a sensor, the specific operations must be controlled by fimc, which seems to be complicated.

In comparison, the hardware interface is easy to understand. by reading the chip manual and schematic diagram, there is basically no problem. The software interface is complicated and there is a camera controller in the middle. The following mainly analyzes software interfaces.

3. hardware interface

The camera's hardware schematic is as follows:

After obtaining the schematic diagram, we need to pay attention to the two pins 1 and 2 connected to i2c_sda1 and i2c_scl1 respectively. This shows that the camera can be configured through I2C controller 1. In addition, when debugging the camera, you can use an oscilloscope to measure the waveform based on this schematic to verify that the code is correct.

It is also worth noting that it is best to use a multimeter to measure whether each pin of the camera is correctly connected to the chip before the development driver. Otherwise, the image will not be visible even if the code is correct.

In addition, you need to carefully read the descriptions in the camera Controller Chapter in the chip manual. The following information is clarified:

Fimc supports the above three video industry standards, ov9650 supports ITU-R 601 YCbCr 8-Bit mode, which is very important for later driver writing.

Both mpll and apll can be used as the camera clock source, but mpll is recommended. This is also helpful for subsequent driver development.

4. Software Interface (how to connect with fimc driver)

After clarifying the hardware problems, you can focus on the software interfaces. There are two ways to implement the driver: the first is to make the camera driver into a common v4l2 device and directly call the registers in fimc to capture and process video data; the second method is to mount our camera driver under the fimc driver by using the fimc driver that has been implemented by the kernel.

The first method is to implement a large amount of code, because it is difficult to directly operate the fimc register. The second method is to use the fimc driver that has been completed by the kernel, the difficulty lies in how to integrate the camera driver and fimc driver.

In Android, the first method is not feasible, because the fimc module is not only a control interface of the camera, it also undertakes the output function of v4l2 and the overlay function, these two functions are very important to the android display system. Therefore, the best solution is the second one. Find the interface between the camera driver and the fimc driver. If you specify this interface, the subsequent tasks will be easy and the workload will be low.

4-1: fimc-driven Overall Structure Analysis

Position of the fimc driver in the kernel:

Drivers/Media/Video/Samsung/fimc
Fimc40_regs.c
Fimc43_regs.c
Fimc_capture.c
Fimc_dev.c
Fimc_output.c
Fimc_overlay.c
Fimc_v4l2.c

The most basic of these source codes is fimc_dev.c, where a platform_driver is registered and the corresponding platform_device description is provided in the corresponding platform code. Controllers on such SOC are usually attached to platform_bus to achieve matching between device and driver during system initialization.

In the probe function of driver, it mainly completes Resource Acquisition and registration of v4l2 devices. Because fimc has three identical controllers (fimc0, fimc1, fimc2), an array is used to describe the driver:

Struct video_device fimc_video_device [fimc_devices] = {
[0] = {
. Fops = & fimc_fops,
. Ioctl_ops = & fimc_v4l2_ops,
. Release = fimc_vdev_release,
},
[1] = {
. Fops = & fimc_fops,
. Ioctl_ops = & fimc_v4l2_ops,
. Release = fimc_vdev_release,
},
[2] = {
. Fops = & fimc_fops,
. Ioctl_ops = & fimc_v4l2_ops,
. Release = fimc_vdev_release,
},
};

In the probe function, call video_register_device () to register the three video_devices. In the user space, three Video device nodes, video0, video1, and, video2. fops, a member of each video_device, corresponds to basic operations on v4l2 devices, which are defined as follows:

Static const struct v4l2_file_operations fimc_fops = {
. Owner = this_module,
. Open = fimc_open,
. Release = fimc_release,
. IOCTL = video_ioctl2,
. Read = fimc_read,
. Write = fimc_write,
. MMAP = fimc_mmap,
. Poll = fimc_poll,
};

Another member, ioctl_ops, is very important because it describes all the ioctl operation sets of v4l2. Fimc_v4l2_ops is defined in fimc_v4l2.c:

Const struct v4l2_ioctl_ops fimc_v4l2_ops = {
. Vidioc_querycap = fimc_querycap,
. Vidioc_reqbufs = fimc_reqbufs,
. Vidioc_querybuf = fimc_querybuf,
. Vidioc_g_ctrl = fimc_g_ctrl,
. Vidioc_s_ctrl = fimc_s_ctrl,
. Vidioc_cropcap = fimc_cropcap,
. Vidioc_g_crop = fimc_g_crop,
. Vidioc_s_crop = fimc_s_crop,
. Vidioc_streamon = fimc_streamon,
. Vidioc_streamoff = fimc_streamoff,
. Vidioc_qbuf = fimc_qbuf,
. Vidioc_dqbuf = fimc_dqbuf,
. Vidioc_enum_fmt_vid_cap = fimc_enum_fmt_vid_capture,
. Vidioc_g_fmt_vid_cap = fimc_g_fmt_vid_capture,
. Vidioc_s_fmt_vid_cap = fimc_s_fmt_vid_capture,
. Vidioc_try_fmt_vid_cap = fimc_try_fmt_vid_capture,
. Vidioc_enum_input = fimc_enum_input,
. Vidioc_g_input = fimc_g_input,
. Vidioc_s_input = fimc_s_input,
. Vidioc_g_parm = fimc_g_parm,
. Vidioc_s_parm = fimc_s_parm,
. Vidioc_g_fmt_vid_out = fimc_g_fmt_vid_out,
. Vidioc_s_fmt_vid_out = fimc_s_fmt_vid_out,
. Vidioc_try_fmt_vid_out = fimc_try_fmt_vid_out,
. Vidioc_g_fbuf = fimc_g_fbuf,
. Vidioc_s_fbuf = fimc_s_fbuf,
. Vidioc_try_fmt_vid_overlay = fimc_try_fmt_overlay,
. Vidioc_g_fmt_vid_overlay = fimc_g_fmt_vid_overlay,
. Vidioc_s_fmt_vid_overlay = fimc_s_fmt_vid_overlay,
};

It can be seen that the fimc driver implements all the interfaces of v4l2, which can be divided into v4l2-input device interface, v4l2-output device interface and v4l2-overlay device interface. Here we focus on the v4l2-input device interface, because the camera is a video input device.

Fimc_v4l2.c registers many callback functions to implement the standard interfaces of v4l2. However, these callback functions are basically not implemented in fimc_v4l2.c, but are implemented accordingly. C is implemented separately. For example:

Operation Implementation of v4l2-input equipment: fimc_capture.c
Operation Implementation of v4l2-output equipment: fimc_output.c
Operation Implementation of v4l2-overlay equipment: fimc_overlay.c

These codes are not related to specific hardware operations. This driver writes all the code for operating the hardware register to a file, which is fimc40_regs.c. In this way, the hardware-related code and hardware-independent code are separated to implement code reuse to the maximum extent.

The relationship between these driver source codes is as follows:

4-2: fimc-driven camera interface Analysis

The key to the interface is the probe function in fimc_dev.c. Probe calls a function named fimc_init_global (), which completes camera allocation and clock acquisition. The prototype of this function is as follows:

Static int fimc_init_global (struct platform_device * pdev)

This platform_device is transmitted by the kernel from the platform code, which contains information related to the specific platform, which should contain the camera information.

Function implementation:

Static int fimc_init_global (struct platform_device * pdev)
{
Struct fimc_control * CTRL;
Struct initi_platform_fimc * pdata;
// This struct is used to describe a camera.
// When analyzing the platform code, you can see how it is filled.
Struct initi_platform_camera * Cam;
Struct CLK * srclk;
Int ID, I;

// Obtain the platform information
Pdata = to_fimc_plat (& pdev-> Dev );
Id = pdev-> ID; // ID may be 0, 1, 2
CTRL = get_fimc_ctrl (ID); // obtain the fimc_control struct pointer corresponding to the ID.

/* Registering external camera modules. re-arrange order to be sure */
For (I = 0; I <fimc_maxcams; I ++ ){
Cam = pdata-> camera [I]; // retrieve camera information from Platform Data
If (! Cam)
Continue; // change break to continue by YS

/* Writeback doesn' t need clock setting */
If (cam-> id = camera_wb ){
Fimc_dev-> camera [cam-> id] = Cam;
Break;
}

// Obtain the clock source information
Srclk = clk_get (& pdev-> Dev, Cam-> srclk_name );
If (is_err (srclk )){
Fimc_err ("% s: failed to get mclk source \ n", _ FUNC __);
Return-einval;
}

// Obtain the clock information of camera
/* Mclk */
Cam-> CLK = clk_get (& pdev-> Dev, Cam-> clk_name );
If (is_err (cam-> CLK )){
Fimc_err ("% s: failed to get mclk source \ n", _ FUNC __);
Return-einval;
}

If (cam-> CLK-> set_parent ){
Cam-> CLK-> parent = srclk;
Cam-> CLK-> set_parent (cam-> CLK, srclk );
}

/* Assign camera device to fimc */
Fimc_dev-> camera [cam-> id] = Cam; // assign the camera obtained from the platform to the global data structure.
// Fimc_dev
}

Fimc_dev-> initialized = 1;

Return 0;
}

We can see that this function actually retrieves camera information from the platform data and assigns it to fimc_dev. fimc_dev, which is defined in fimc. h. The type is struct fimc_global. The prototype is as follows:

/* Global */
Struct fimc_global {
Struct fimc_control CTRL [fimc_devices];
Struct initi_platform_camera * camera [fimc_maxcams];
Int initialized;
};

Now let's take a look at how the platform code describes a camera and how to transmit abstract data structures to the platform data.

The platform code corresponding to s5pc100 SOC is located:

ARCH/ARM/mach-s5pc100/mach-smdkc100.c

We describe a camera in this way:

# Ifdef config_video_ov9650
/* Add By ys for ov9650 */
Static struct initi_platform_camera camera_c = {
. ID = camera_par_a,/* fixme */
. Type = cam_type_itu,/* 2.0 m ITU */
. FMT = itu_60policycbcr422_8bit,
. Order422 = cam_order422_8bit_ycbycr,
. I2c_busnum = 1,
. Info = & camera_info [2],
. Pixelformat = v4l2_pix_fmt_yuyv,
. Srclk_name = "dout_mpll ",
. Clk_name = "sclk_cam ",
. Clk_rate = 16000000,/* 16 MHz */
. Line_length = 640,/* 640*480 */
/* Default resol for preview kind of thing */
. Width = 640,
. Size = 480,
. Window = {
. Left = 0,
. Top = 0,
. Width = 640,
. Size = 480,
},

/* Polarity */
. Inv_pclk = 1,
. Inv_vsync = 0,
. Inv_href = 0,
. Inv_hsync = 0,

. Initialized = 0,
};
# Endif

All information related to ov9650 is described here. Type indicates that the camera is an ITU interface, FMT indicates that the output format of the camera is itu_601_ycbcr422_8bit, and order422 indicates that the order of the three components of YUV is YCbCr. These are consistent with the previous descriptions. In addition, there is information about the clock source, the clock size, and the resolution of the captured image. Here we set 640x480 (VGA mode ), after debugging, it is found that the ov9650 works smoothly and clearly in VGA mode. Polarity indicates the polarity of the signal. The specific settings must be consistent with those of the camera.

I2c_busnum is the bus number of the I2C bus. Because s5pc100 has two I2C buses (0 and 1), we connect them to sda1, so i2c_busnum is 1.

Camera_c is a member of the fimc_plat struct:

/* Interface setting */
Static struct initi_platform_fimc fimc_plat = {
. Default_cam = camera_par_a,
. Camera [2] = & camera_c,
. Hw_ver = 0x40,
};

Camera_c is assigned to the third element of the Camera Array in fimc_plat. The third element is Android. This will be explained when analyzing the hardware abstraction layer of the android camera.

The structure of struct initi_platform_fimc is actually the platform data structure corresponding to fimc. In the Platform code, the following three functions are responsible for registration:

Cloud_fimc0_set_platdata (& fimc_plat );
Cloud_fimc1_set_platdata (& fimc_plat );
Cloud_fimc2_set_platdata (& fimc_plat );

As for how these functions are implemented, we will not analyze them here. If you are interested, you can read the code by yourself.

That is to say, as long as the platform code is filled with a struct of the struct initi_platform_camera type and then added to fimc_plat, The fimc driver can obtain the corresponding camera information.