Design and development of Linux player

This article is based on the development experience of Dawnlightplayer. Dawnlithtplayer is the beginning of today March, and maddrone together in the spare time development of a cross-platform, multi-threaded player, mainly under the development of Linux, the text used in the sample code is truncated from it.
Dawnlightplayer can now run on Linux and Windows systems, and has developed a GUI with VC and Python, supports most audio and video file formats and network streams, adds support for the CMMB protocol, does not support RMVB, SWF Video file formats such as those that have not yet been publicly agreed.

Directory:
I. Process of the player
1. Enter
2. Decoding
3. Output
Two. Implementation of the player
1. Input implementation
2. Decoding thread implementations
3. Output Thread Implementation
Three. Video Output library
1. SDL (Multi-platform, support hardware scaling)
2. DirectX DirectDraw (Win32 platform, hardware scaling supported)
3. OpenGL (Multi-platform, support hardware scaling)
4. X11 (Linux/unix)
5. FrameBuffer (Linux, no hardware scaling)
Four. Audio output
1. OSS (Open sound System for Linux)
2. ALSA (Advanced Linux sound Architecture)
3. DirectSound (WIN32)
Five. Audio and video synchronization
1. Sync video on audio basis
2. Sync Audio as a video reference
3. Synchronizing to an external clock
Six.
1. Save as JPEG file using Jpeglib
2. Save as PNG file using libpng
Seven. YUV RGB software Conversion
Eight. Software scaling



I. Process of the player

1. Input: From the file or network, such as reading the original data, such as X.avi, X.mov, Rtsp://xxx, the original data analysis, such as files, the first to analyze the file format, from the file to obtain audio and video encoding parameters, video length of time and other information, Then to take out the audio encoding data and video encoding data sent to the decoding section, where this is temporarily called the source data block packet.

2. Decoding: initialization, using the input from the source data obtained from the information call different decoding library initialization, and then receive the input to transmit audio and video encoded data, respectively, to decode and video decoding, video decoding the data is generally YUV or RGB data, here is called picture, Audio decoded data is sampled data, is the sound card can play the data, here is temporarily called sample. The decoded data is then sent to the output section.

3. Output: Receive the picture and sample sent from the decoding section and display it. Video display generally uses a graphics library, such as SDL, Xlib, DirectDraw, OPENGGL, framebuffer, etc., audio output is to write the sample to the system audio driver, audio driver into the sound card playback, the available audio output has ALSA, OSS, SDL , DirectSound, Waveout and so on.

Two. Implementation of the player

Recommended implementation Scenarios
A audio_packet queue, a video_packet queue, a picture queue, a sample queue
One input thread, two decode threads, two output threads, one UI control thread

1. Input implementation
The parsing of the file, first to understand the format of the file, the file format is generally referred to as the file container. Open file format, according to the format of the protocol read analysis can be, but like rmvb,swf this is still not open format of the file, it is not easy to do, but also the current general player difficulties. General file Format Parsing Libavformat Library has been done, as long as the use of it on the line, the following is shown in the sample code snippet:

Initialization
static int avin_file_init (void)
{
Avformatparameters params, *ap =? ms;
Err = Av_open_input_file (&fmtctx, Input_filename, NULL, 0, AP);
if (Err < 0)
{
Av_log (NULL, Av_log_error, "%d:init input from File error\n", __line__);
Print_error (Input_filename, err);
return-1;
}

Fmtctx->flags |= avfmt_flag_genpts;

Err = Av_find_stream_info (FMTCTX);
if (Err < 0)
{
Av_log (NULL, Av_log_error, "%d:init input from File error\n", __line__);
Print_error (Input_filename, err);
return-1;
}

if (FMTCTX-&GT;PB) fmtctx->pb->eof_reached = 0;
Dump_format (fmtctx, 0, input_filename, 0);

....
}
Read packet:
while (1)
{
Avpacket *pkt = NULL;
PKT = Av_malloc (sizeof (avpacket));
ret = Av_read_frame (Fmtctx, PKT);

Send the packet to the decoding section:
Can be memcpy, or treated with a linklist structure, such as:
Push_to_video_packet_queue (PKT);
}

If it is your own private input, such as the video input of mobile TV, the code is as follows, part of the pseudo-code:
while (1)
{
Your_parse_code ();
Size = Your_get_video_data (BUF);

PKT = Av_mallocz (sizeof (avpacket));
x = Av_new_packet (PKT, Vret);
memcpy (Pkt->data, buf, size);
pkt->pts = Your_time;

Push_to_video_packet_queue (PKT);
}

2. Decoding thread implementations
Decoding is an algorithm, most of which can only use existing decoding libraries, such as LIBAVCODEC, the following sample code:
while (1)
{
Avpicture *picture;
Avpacket *pkt = Pop_from_video_packet_queue ();
Avframe *frame = Avcodec_alloc_frame ();
Avcodec_decode_video (VIDEO_CTXP, Frame, &got_picture, Pkt->data, pkt->size);
if (got_picture)
{
Convert_frame_to_picture (picture, frame);
Picture->pts = pkt->pts;
Push_to_picture_queue (picture);
}
}
Audio similarity

3. Output Thread Implementation

Video output to control FPS, such as 25 frames per second video, then the display time of each frame if 1/25 seconds, but a frame of RGB data written to the memory can not be used for 1/25 seconds time, then control, not Allows 25 frames of data to be displayed in 0.1 or 0.2 seconds, the simplest implementation is after each display of a frame of data, sleep (1/fps-display time).

Audio and video synchronization this important work should also be done in the output thread. Synchronizing video on an audio basis, synchronizing audio as a video reference, or synchronizing with an external clock is a viable approach, but synchronizing video with audio as a benchmark is the simplest and most effective method. Audio driver as long as the sample rate is set, sample size and channels, the write data will play at this constant speed, if the driver output buffer full, write can wait.

Video:
while (1)
{
Picture = Pop_from_picture_queue ();
Picture_shot (picture); /* */
Vo->display (picture);
Video_pts = picture->pts;
Sync_with_audio (); /* Sync */
Control_fps (); /* FPS */
}
Audio:
while (1)
{
Sample = Pop_from_sample_queue ();
Ao->play (sample);
Now_pts = sample->pts;
}

Three. Video Output library

1. SDL (Multi-platform, support hardware scaling)

SDL (Simple DirectMedia Layer) are a Cross-platform multimedia library designed to provide low level access to audio, Keyboa Rd, Mouse, joystick, 3D hardware via OpenGL, and 2D video framebuffer.

In fact, SDL is a middleware, it encapsulates the underlying OpenGL, FrameBuffer, X11, DirectX and so on to provide a unified API interface to the upper layer, the advantage of using SDL is that we do not have to make a video output program for X11 or DirectX separately.

SDL can directly display YUV data and RGB data, generally decoded picture is yuv420p format, do not have to do YUV2RGB conversion can be directly displayed, the main code is as follows:

static int vo_sdl_init (void)
{
....
Screen = Sdl_setvideomode (WW, WH, 0, flags);
Overlay = Sdl_createyuvoverlay (DW, DH, sdl_yv12_overlay, screen);
....
}

static void Vo_sdl_display (Avpicture *pict)
{
Sdl_rect Rect;
Avpicture p;

Sdl_lockyuvoverlay (overlay);
P.data[0] = overlay->pixels[0];
P.DATA[1] = overlay->pixels[2];
P.DATA[2] = overlay->pixels[1];
P.linesize[0] = overlay->pitches[0];
P.LINESIZE[1] = overlay->pitches[2];
P.LINESIZE[2] = overlay->pitches[1];
Vo_sdl_sws (&p, PICT); /* Only do memcpy */
Sdl_unlockyuvoverlay (overlay);

rect.x = DX;
Rect.y = dy;
RECT.W = DW;
rect.h = DH;
Sdl_displayyuvoverlay (overlay, &rect);
}

2. DirectX DirectDraw (Win32 platform, hardware scaling supported)

DirectX is an output that is used more on Windows, and also supports direct YUV or RGB display, sample code:

static int vo_dx_init (void)
{
Dxcreatewindow ();
Dxinitdirectdraw ();
Dxcreateprimarysurface ();
Dxcreateoverlay ();
Detectimgformat ();
}

static void Vo_dx_display (Avpicture *pic)
{
Vfmt2rgb (My_pic, pic);
memcpy (G_image, my_pic->data[0], my_pic->linesize[0] * height);
Flip_page ();
}

3. OpenGL (Multi-platform, support hardware scaling)

OpenGL is a 3D game library, cross-platform, high efficiency, support the majority of display acceleration, display 2D RGB data as long as the use of the Gldrawpixels function is sufficient, while disabling some OpenGL pipeline operation more efficient, such as:

Gldisable (gl_scissor_test);
Gldisable (gl_alpha_test);
Gldisable (gl_depth_test);
Gldisable (Gl_dither);

4. X11 (Linux/unix)

X11 is the basic graphical interface library on the Unix/linux system platform, which is mainly based on X11, such as the common GTK,QT. But X11 API interface is too many, complex, very bad for development, the basic GUI program will generally use GTK,QT, etc., will not directly invoke X11 API, here just for efficiency. Mplyaer's LIBVO has X11 full-use code, including full-screen functionality.

static void Vo_x11_display (avpicture* pic)
{
Vfmt2rgb (My_pic, pic);
Ximg->data = my_pic->data[0];
Xputimage (Xdisplay, Xvowin, XVOGC, Ximg,
0, 0, 0, 0, DW, DH);
Xsync (Xdisplay, False);
Xsync (Xdisplay, False);
}

5. FrameBuffer (Linux, no hardware scaling)

Framebuffer is part of the Linux kernel, providing a map to the memory's access address, but without any accelerated use.

static void Vo_fb_display (Avpicture *pic)
{
int i;
uint8_t *src, *dst = fbctxp->mem;

Vfmt2rgb (My_pic, pic);
src = my_pic->data[0];

for (i = 0; i < fbctxp->dh; i++)
{
memcpy (DST, SRC, FBCTXP-&GT;DW * (FBCTXP-&GT;VARINFO.BITS_PER_PIXEL/8));
DST + = fbctxp->fixinfo.line_length;
src + = my_pic->linesize[0];
}
}

Four. Audio output

1. OSS (Open sound System for Linux)

The OSS is the simplest audio output under Linux and can be directly write.

static int ao_oss_init (void)
{
int i;
DSP = open (Dsp_dev, o_wronly);
if (DSP < 0)
{
Av_log (NULL, Av_log_error, "Open oss:%s\n", Strerror (errno));
return-1;
}
i = sample_rate;
IOCTL (DSP, Sndctl_dsp_speed, &i);
i = Format2oss (SAMPLE_FMT);
IOCTL (DSP, SNDCTL_DSP_SETFMT, &i);
i = channels;
if (i > 2) i = 2;
IOCTL (DSP, Sndctl_dsp_channels, &i);

return 0;
}

static void Ao_oss_play (Avsample *s)
{
Write (DSP, S->data, s->size);
}

2. ALSA (Advanced Linux sound Architecture)

Alsa does the comparison of failed, long function names.

static void Ao_alsa_play (Avsample *s)
{
int num_frames = s->size/bytes_per_sample;
snd_pcm_sframes_t res = 0;
uint8_t *data = s->data;

if (!alsa_handle)
return;

if (Num_frames = = 0)
return;

Rewrite
res = Snd_pcm_writei (alsa_handle, data, num_frames);
if (res = =-eintr)
Goto rewrite;
if (Res < 0)
{
Snd_pcm_prepare (Alsa_handle);
Goto rewrite;
}
if (Res < num_frames)
{
Data + = res * BYTES_PER_SAMPLE;
Num_frames-= res;
Goto rewrite;
}
}

3. DirectSound (WIN32)

The disadvantage of MS DirectX is that it is not as good as OSS or Alsa in Linux, and when there is no sample writing, it is automatically silent,directsound during playback, when no sample data is fed into the output thread. It always replays the last 0.2 or 0.5 seconds of data. Because only recently ported Dawnlightplayer used Windows, the mechanism was not well understood.

static void Dsound_play (Avsample *s)
{
int Wlen, ret, len = s->size;
uint8_t *data = s->data;

while (Len > 0)
{
Wlen = Dsound_getspace ();
if (Wlen > len) wlen = len;
RET = Write_buffer (data, Wlen);
Data + = RET;
Len-= ret;
Usleep (10*1000);
}
}

Five. Audio and video synchronization

1. Sync video on audio basis

The video output thread is handled as follows:
Start_time = Now ();
....
Vo->display (picture);
Last_video_pts = picture->pts;
End_time = Now ();
Rest_time = End_time-start_time;
Av_diff = last_audio_pts-last_video_pts;
if (Av_diff > 0.2)
{
if (Av_diff < 0.5) Rest_time-= REST_TIME/4;
else rest_time-= REST_TIME/2;
}
else if (Av_diff <-0.2)
{
if (Av_diff > -0.5) rest_time + = REST_TIME/4;
else rest_time + = REST_TIME/2;
}
if (Rest_time > 0)
Usleep (Rest_time);

2. Sync Audio as a video reference


3. Synchronizing to an external clock



Six.

It can be done in the decoding thread, or in the output thread, as seen in the previous output thread section. Just save the picture before display. Generally add some code, such as save in PNG or JPEG format.

1. Save as JPEG file using Jpeglib

static void Draw_jpeg (Avpicture *pic)
{
Char fname[128];
struct Jpeg_compress_struct cinfo;
struct Jpeg_error_mgr jerr;
Jsamprow Row_pointer[1];
int row_stride;
uint8_t *buffer;

if (!po_status)
return;

Vfmt2rgb24 (My_pic, pic);
Buffer = my_pic->data[0];

#ifdef __mingw32__
sprintf (fname, "%s\\dlpshot-%d.jpg", Get_save_path (), framenum++);
#else
sprintf (fname, "%s/dlpshot-%d.jpg", Get_save_path (), framenum++);
#endif
fp = fopen (fname, "WB");
if (fp = = NULL)
{
Av_log (NULL, Av_log_error, "fopen%s error\n", fname);
Return
}
Cinfo.err = Jpeg_std_error (&jerr);
Jpeg_create_compress (&cinfo);
Jpeg_stdio_dest (&cinfo, FP);

Cinfo.image_width = width;
Cinfo.image_height = height;
Cinfo.input_components = 3;
Cinfo.in_color_space = Jcs_rgb;

Jpeg_set_defaults (&cinfo);
Cinfo.write_jfif_header = TRUE;
Cinfo. Jfif_major_version = 1;
Cinfo. Jfif_minor_version = 2;
Cinfo.density_unit = 1;
Cinfo. x_density = jpeg_dpi * width/width;
Cinfo. y_density = jpeg_dpi * height/height;
Cinfo.write_adobe_marker = TRUE;

Jpeg_set_quality (&cinfo, jpeg_quality, jpeg_baseline);
cinfo.optimize_coding = jpeg_optimize;
Cinfo.smoothing_factor = Jpeg_smooth;
if (Jpeg_progressive_mode)
{
Jpeg_simple_progression (&cinfo);
}
Jpeg_start_compress (&cinfo, TRUE);

Row_stride = width * 3;
while (Cinfo.next_scanline < height)
{
ROW_POINTER[0] = &buffer[cinfo.next_scanline * Row_stride];
(void) Jpeg_write_scanlines (&cinfo, Row_pointer, 1);
}

Jpeg_finish_compress (&cinfo);
Fclose (FP);
Jpeg_destroy_compress (&cinfo);

return;
}

2. Save as PNG file using libpng

static void Draw_png (Avpicture *pic)
{
int k;
Png_byte *row_pointers[height]; /* GCC C99 */

if (init_png () < 0)
{
Av_log (NULL, Av_log_error, "Draw_png:init png error\n");
return;
}

Vfmt2rgb24 (My_pic, pic);

for (k = 0; k < height; k++)
ROW_POINTERS[K] = my_pic->data[0] + my_pic->linesize[0] * k;

Png_write_image (Png.png_ptr, row_pointers);

Destroy_png ();
}


Seven. YUV RGB Conversion

YUV and RGB conversion and scaling, generally on the low-end devices, to have hardware acceleration to do, otherwise the CPU is too much. on today's high-end PCs, software can be used to do this, and the Libswscale library is coming. Libswscale has been optimized for X86 CPUs, such as the use of MMX, SSE, 3DNOW and other CPU-related multimedia instructions.

static int Vfmt2rgb (Avpicture *dst, Avpicture *src)
{
static struct Swscontext *img_convert_ctx;

Img_convert_ctx = Sws_getcachedcontext (Img_convert_ctx,
width, height, src_pic_fmt,
width, height, my_pic_fmt, sws_x, NULL, NULL, NULL);

Sws_scale (Img_convert_ctx, Src->data, Src->linesize,
0, Width, dst->data, dst->linesize);

return 0;
}

For example, the conversion from yuv420p to RGB24, as long as setting

SRC_PIC_FMT = pix_fmt_yuv420p;
MY_PIC_FMT = Pix_fmt_rgb24;


Eight. Software scaling

Software scaling can use the above Libswscale library, the calling code is basically the same, just change the width and height of the target picture, such as magnification twice times:

static int zoom_2 (Avpicture *dst, Avpicture *src)
{
static struct Swscontext *img_convert_ctx;

Img_convert_ctx = Sws_getcachedcontext (Img_convert_ctx,
width, height, src_pic_fmt,
Width*2, Height*2, my_pic_fmt, sws_x, NULL, NULL, NULL);

Sws_scale (Img_convert_ctx, Src->data, Src->linesize,
0, Width*2, Dst->data, dst->linesize);

return 0;
}

Design and development of Linux player