[Translation] emiplib 1.1.1 doxygen

Note: This is the original address. In addition, I keep the original key words]

Hasselt University-expertise centre for Digital Media

1. Introduction

This is the instruction manual of emiplib. emiplib is EDM media over IP libray. This library was developed at the expertise centre for digital media (http://www.uhasselt.be) Institute at Hasselt University (http://www.edm.uhasselt.be. As shown in the name of this library, the goal of this library is to make various media better transmitted over the network, including but not limited to audio and video.

1.1 license

[Note]

1.2 Contact Information

The home address of this library can be found here. Emiplib has an email list. You can send an email to [email protected] to subscribe to it.

2. Design Philosophy

To provide a flexible framework, this library is designed to provide a large number of small components. Each component is responsible for specific tasks, such as audio sampling, writing audio to audio files, or transmitting audio packets. These components can be distributed in one chain that allows different components to exchange messages. In this way, we can create more complex applications by creating links between different components.

In addition to the above core library, emiplib also strives to provide a series of Wrapper Classes used to combine component. This facilitates emiplib users: Apart from manually linking different components, we can also create a session object for VoIP. If you need more flexible usage, you can easily manually combine different components.

3. core content of the database

In this section, we will detail the core content of the database. We will explain from a simple perspective:

3.1 Basics

For example"Design Philosophy"As described in this section, the core library consists of three parts:

• Components
• Component chains
• Messages

The Child class of mipcomponent is component and can be embedded in the component chain (that is, mipcomponentchain ). When the chain is activated, the message is transmitted through the link of the component chain. Such messages are inherited from the mipmessage class.
Suppose we have two components. One sound card inputs component 'sndin 'and the other one outputs component 'sndoout '. We put these components in a chain called 'loopchain' and start it:

loopchain.setChainStart(&sndin);loopchain.addConnection(&sndin,&sndout);loopchain.start();

In this way, we can tell the chain that the first component is 'sndin ', and then put it in a link from 'sndin' to 'sndoout. When the chain is started, a background thread starts to execute the following tasks:

• Mipsystemmessage_type_waittimeThe type of mipsystemmessage is created and sent from the start end of the chain. In this example, the start end of the chain is the sound card input component, and the component (such as the microphone) waits until a certain number of samples are received.
• This Chain transmits data through an existing link. In this case, there is only one link from the sound card input to the sound card output component. Therefore, the data is sampled from the sound card input component and transmitted to the sound card output component. In this example, the message contains the original audio sample. When the sound card outputs component to receive the message, the speaker will play the sound. As you can see, this example describes an echo effect application.
• When all messages are distributed out, the thread continues to loop and sends a message of mipsystemmessage_type_waittime to the first component. When the mipcomponentchain: Stop method is called or an error occurs, the background process ends.

Therefore, the first component of the chain is timer (timer ). In this case, the sound card inputs component itself.

3.2 details

First, we will describe how this chain is established, and then describe how messages are transmitted. Finally, the feedback mechanism is described.

3.2.1 create a chain

The connection of component can be specified by the mipcomponentchain: addconnection method. In this way, you can simply add a specific connection to the connection list and store it in the chain. We can use the mipcomponentchain: setchainstart method to set the start component. This component will accept the first message:Mipsystemmessage_type_waittimeChild-type mipsystemmessage.
When calling the mipcomponentchain: Start functionConnection listWill be re-ordered based on the correct message distribution queue. Is a chain starting with timing component. Assume that the link is created in the order of the following numbers:

 

Placing component on different layers is the result of algorithm repeat connection. The first layer is the result of the mipcomponentchain: setchainstart method. The algorithm iterates all connections to find the connection starting with the timer component. Each connection is removed from the old connection list and added to the new reordered connection list. The end of the connection is added to the next layer. When the first layer is processed, the algorithm iteratively processes the component of the next layer. The final processing result is as follows: 4, 2, 1, 6, 3, 5.

If you do not specify the start component or the algorithm to find some connections, an error is returned.

3.2.2 distribute message

If the chain can start, the background process will generate a large number of messages and distribute messages between different components. Chain itself createsMipsystemmessage_type_waittimeChild-type mipsystemmessage instance, and then sends itself to the first component of the chain, using the mipcomponent: Push method. During the call, the chain uses the mipcomponent: Lock and mipcomponent: Unlock methods to lock the component.

This message indicates that the first component enters the waiting state until other messages can be distributed. When the push method of this component is returned, the background thread starts to iterate the sorted connection list. The header and tail component are locked for each connection in the list. Through the mipcomponent: Pull interface, the message is extracted from the connection header component, and the end component of the connection is input through the mipcomponent: Push interface. For more information about the push and pull methods, see the mipcomponent document.

If the push or pull method returns false, the thread will call the mipcomponentchain: onthreadexit member method and then exit. The name and description of the component that produces the error will be passed to the parameter of this function.

3.2.3 feedback chain

The message distribution system only supports one-way transmission. For many reasons, we may want to reverse transmit information. This can be done by using the third parameter of the mipcomponentchain: addconnection member function to specify whether to provide feedback.

Before starting the background process, the feedback chain is established based on the connection list information. For example, if connection 1 and connection 3 are marked as feedback connections, the feedback chain will be composed of RTP component, RTP Audio Encoder, and sound card input.

When the background thread is running, the feedback message is also distributed after the message is distributed. For each feedback chain, a mipfeedback message is created and transmitted through each feedback component of the mipcomponent: processfeedback member function in the chain. By implementing this function, a component can obtain or modify feedback information.

Note that many feedback chains can exist at the same time, and they can even have the same component at the end of the chain. However, a component cannot process the feedback of two chains at the same time.

3.2.4 branch and merge

The example in the "Create chain" section shows how to create and connect to many different component (such as message dumper and sound card input component) from the specified component (such as timer ), this can even merge subchains. In the following example, two different audio file component is used to read the data of each audio file:

The data of each file is re-sampled and the sampling rate is unified. A mixer combines the audio of two files into an output stream, and then outputs the verified encoding method to the sound card.

Now let's assume that one of the audio files has an appropriate sampling rate. In this way, it is unnecessary to pass its data to the reseller. You may think it is a good path:

The connection sequence is as follows:

                   Timer -> Soundfile input 2                   Timer -> Soundfile input 1       Soundfile input 2 -> Mixer       Soundfile input 1 -> Sampling rate converter                   Mixer -> Sample encoder Sampling rate converter -> Mixer          Sample encoder -> Soundcard output

In this case, when one track is still from the reseller to the mixer, the other is from the mixer to the sampling encoder. In good cases, there will be time difference between two channels and the mixed speaker; in poor cases, the later audio tracks may be discarded.

Therefore, when branch and merge are used, you may prefer to allocate the same number of component layers for different branches to avoid this problem-you can use the mipcomponentalias class to help solve the problem.

4. Tutorial

[Note]

5. Available component

The simplest way to view component is to look at the directory structure.

Warning:

In principle, a specified component instance can be used on multiple chains, and each chain is locked when it is used. However, in fact, only a few components can be used by multiple chains. Most component may cause synchronization problems.

6. Wrapper available

Currently, two wrapper instances are available:

• Mipaudiosession: this is a simple VoIP session. It can use multiple audio compression codecs.
• Mipvideosession: this is a simple VoIP session. It can use the H.263 + codecs of libavcodec.