IOS and Android audio Development Summary

IOS and Android audio Development Summary

Recently, because the project needs to change the sound, it is not difficult to learn and Process audio. However, it is still tedious (more than expected)

While my mind is still excited, I will summarize my thoughts and record them.

This article mainly introduces three parts:

1. How to change VOICE 2, Android voice 3, ios voice

1.

It's easy to write a function or library for your own voice change, so SoundTouch is widely used.

This library allows you to change the speed, beat, and tone of a voice. (the most important thing is to increase or lower the tone of the voice to turn it into a male girl. Refer to Tom)

The idea is to place the entire library on the underlying layer of different platforms. You only need to include the header file soundtouch. h.

The SoundTouch class provides many methods, the most important of which is the setPitch and setRate methods for adjusting sound parameters. You can set parameters when using them.

However, before using the function, you must set the following parameters:

                mSoundTouchInstance->setSetting(SETTING_USE_QUICKSEEK, 0);                mSoundTouchInstance->setSetting(SETTING_USE_AA_FILTER, !(0));                mSoundTouchInstance->setSetting(SETTING_AA_FILTER_LENGTH, 32);                mSoundTouchInstance->setSetting(SETTING_SEQUENCE_MS, 40);                mSoundTouchInstance->setSetting(SETTING_SEEKWINDOW_MS, 16);                mSoundTouchInstance->setSetting(SETTING_OVERLAP_MS, 8);

Then set the required parameters.

                mSoundTouchInstance->setChannels(2);                mSoundTouchInstance->setSampleRate(8000);                mSoundTouchInstance->setPitch(2);

It is important to explain several audio processing parameters here.

Channel: channals, which can be single channel or dual channel, corresponding to 1, 2

Sampling Rate: SampleRate 8000-44100, which is usually a few common values. In Android, 44100 is supported by all devices, so it is safer to set it to 44100.

Number of BITs per channel: bitsPerChannel is generally set to 16

The number of channels for each frame ChannelsPerFrame for pcm Data, this is 1

There are several other parameters, which may not be the same for Android and ios. The above parameters are all used and important and must be mastered.

 

2. implement voice changes in Android

Because the project requires real-time playback of recordings, you need to read the audio data stream (in PCM format) for playback. The APIs used are AudioRecorder and AudioTrack.

There are many materials related to the specific usage, and the official documents are also detailed. The general idea is to initialize first:

        //initilize
　　　　 trbusize=AudioTrack.getMinBufferSize(RECORDER_SAMPLERATE,AudioFormat.CHANNEL_OUT_STEREO,                AudioFormat.ENCODING_PCM_16BIT);        mAudioTrack = new AudioTrack(AudioManager.STREAM_MUSIC,RECORDER_SAMPLERATE,                AudioFormat.CHANNEL_IN_STEREO,                AudioFormat.ENCODING_PCM_16BIT,                trbusize,                AudioTrack.MODE_STREAM);        rebusize = AudioRecord.getMinBufferSize(RECORDER_SAMPLERATE, AudioFormat.CHANNEL_IN_STEREO,AudioFormat.ENCODING_PCM_16BIT);        mAudioRecord= new AudioRecord(MediaRecorder.AudioSource.MIC,RECORDER_SAMPLERATE,AudioFormat.CHANNEL_IN_STEREO, AudioFormat.ENCODING_PCM_16BIT, rebusize);

Because the parameters supported by different devices may be different, you can write a loop to try all the possible parameters.

It is followed by recording and playback, which can be put into two threads respectively. Generally, the recording data is stored in a file and then played, which can meet the normal recording requirements. However, the disadvantage is that, after a long recording time, the file will be very large. If you play the video in real time on the network, it will definitely not work. The solution is to transmit the recording data to a buffer zone, and then take the data directly from the buffer zone during playback. This buffer can be implemented using a cyclic queue or a consumer list in java.

Then there is the voice change part. If you want to use the c ++ library in Android, you can only implement it through jni. You can write several functions.

                while(isInstancePlaying){                    if(l<21){                        byte[] mbyte=new byte[64];                        mAudioRecord.read(mbyte,0,64);                        SoundTouch.getSoundTouch().putSamples(mbyte,0,INPUT_LENGTH);                        SoundTouch.getSoundTouch().setPitchSemiTones(pitchTone);                        SoundTouch.getSoundTouch().receiveSamples(mbyte,INPUT_LENGTH);                        byteArray.add(mbyte);                        l=byteArray.size();                    }                    else{                            mAudioTrack.write(byteArray.getFirst(),0,64);                            byteArray.removeFirst();                            l=byteArray.size();                    }

The Code contains three functions: putSamples, setPitchSemiTones, and esesamples. all three are native methods. In the SoundTouch library, the corresponding functions provided by the SoundTouch class are implemented, which is relatively simple. Through these functions, the sound changes can be implemented.

L variable is the length of the bytes list (byteArray in the Code). When it is less than 20, it is added to the end of byteArray. At the same time, AudioTrack constantly reads the first element in the array to play back and then deletes the element.

Remember to release mAudioTrack and mAudioRecorder after the final playback. Implemented through the stop and release methods.

 

3. IOS voice changes

 

Because I have never been familiar with ios before, I encountered many problems and finally solved them.

Audio Processing in ios is a little more troublesome than Android. The core api used is AudioQueue. You must understand its principles before using it, unlike Android, this api is used for ios playback and recording. It is equivalent to implementing the AudioRecorder and AudioTrack functions in Android, but the internal process changes during playing and recording.

Core Ideas:

There is a built-in queue in Audio. First, you can create several buffers (about 3-6) to load Audio data, after playing or recording in the self-built queue, use the user-defined callback function for processing, so that the buffer can be reused and some user-defined functions can be implemented in the callback function, such as voice changes and file writing. The official team gave a detailed description.

The first is the recording flowchart:

 

 

Then, the playback flowchart:

How to change the voice?

Ios voice changes do not require Android's jni, because the oc language can be mixed with c ++, so this is relatively easier. The process is as follows:

First, instantiate a SoundTouch class in your program, and set its parameters during initialization (setSetting ), then, in the callback function described above, you can process the data stream produced by the recording and save it to a file or play it directly. The idea is like this, but the parameters in the function are relatively cumbersome. If the previous principle is not understood, it will be difficult to do so.

Real-time playback?

Similar to Android, you can write a loop queue to cache audio data, and then play the audio while uploading data to it, unlike Android, these operations need to be implemented in the corresponding callback function. A simple method is to directly play pcm Data in the callback function of the recording. Because the data comes in one piece, each time the buffer is used, a callback function is called, which can be played directly in the callback function.

 

The above is a brief introduction to recording and real-time playback on the two platforms. There are a lot of things here, which deserves further research.