IOS audio playback (5): transcoding from AudioQueue

IOS audio playback (5): transcoding from AudioQueue

 

Preface

 

This article describes how to useAudioStreamFileAndAudioFileParses audio data formats and detaches audio frames. The next step is to use the separated audio frame for playing. This section will explain how to use it.AudioQueuePlay audio data.

Introduction to AudioQueue

AudioQueueYesAudioToolBox.frameworkOne of them, which is described by Apple in the official documentationAudioQueueOf:

Audio Queue Services provides a straightforward, low overhead way to record and play audio in iOS and Mac OS X. It is the recommended technology to use for adding basic recording or playback features to your iOS or Mac OS X application.

In this document, Apple recommends developers to useAudioQueueIn the app. Here we will introduce the playback function.

For supported data formats, Apple says:

Audio Queue Services lets you record and play audio in any of the following formats:* Linear PCM.* Any compressed format supported natively on the Apple platform you are developing for.* Any other format for which a user has an installed codec.

It supportsPCMData, compression formats supported by the iOS/MacOSX platform (MP3, AAC, etc.), other users can provide their own decoder audio data (for this one, in my understanding, the audio format is decoded into PCM data and then played back to AudioQueue ).

Working Mode of AudioQueue

In useAudioQueueWe must first understand its working mode. It is named as this because it has a Buffer Queue mechanism. InAudioQueueAfter startup, you must passAudioQueueAllocateBufferGenerate severalAudioQueueBufferRefStructure, These buffers will be used to store the audio data to be played, and these buffers are generated by theirAudioQueueFor instance management, the memory space has been allocated (according to the Allocate method parameters). WhenAudioQueueWhen the Buffer is Dispose, the Buffer is also destroyed.

When audio data needs to be played, memcpy must firstAudioQueueBufferRef(The Memory pointed to by mAudioData has been allocated.AudioQueueAllocateBufferAnd assign the value of the input data size to the mAudioDataByteSize field. CallAudioQueueEnqueueBufferInsert Buffer with audio dataAudioQueueBuilt-in Buffer Queue. When a Buffer exists in the buffer QueueAudioQueueStart.AudioQueueThe Buffer in the buffer queue is played one by one in the order of Enqueue, each time a Buffer is used, it is removed from the Buffer Queue and a callback is triggered on the user-specified RunLoop to notify the user of a certainAudioQueueBufferRefThe object has been used. You can reuse the object to store the subsequent audio data. In this way, the cyclic audio data will be played one by one until the end.

The official document provides a diagram to describe the process:

In my understanding, callback refers to an audio data loading method, which can be triggered by the callback after data usage mentioned earlier.

AudioQueue playback

According toAudioQueueThe working principle can be understood as follows:

1. CreateAudioQueue, Create a buffer array BufferArray;
2. UseAudioQueueAllocateBufferCreate severalAudioQueueBufferRef(Generally 2-3 requests), put it in BufferArray;
3. When there is data, a buffer is taken from BufferArray. After memcpy data is usedAudioQueueEnqueueBufferMethod To insert bufferAudioQueueMedium;
4. AudioQueueAfter Buffer exists in, callAudioQueueStartPlay. (You can control the number of buffer to be filled in before playing the video, as long as the video can be played continuously );
5. AudioQueueA buffer is consumed after playing the music. The buffer is called back and sent out in another thread, and the buffer is put back into the BufferArray for the next use;
6. Return to Step 3 and continue until the playback ends.

   As shown in the preceding steps,AudioQueueThe playing process is actually a typical producer and consumer problem. Producer isAudioFileStreamOrAudioFileAudio Data frames in the production areaAudioQueueIn the buffer queue, wait until the buffer is filled up for consumption by the consumer;AudioQueueAs a consumer, the data in the buffer queue is consumed, and the data in the callback of another thread is notified to the consumer that the data has been consumed by the producer to continue production. ThereforeAudioQueueIn the process of playing audio, it is inevitable that you will be exposed to several problems such as multi-thread synchronization, semaphore usage, and deadlock avoidance.

   After learning about the workflow, let's look back.AudioQueueMethods, most of which are very easy to understand, and some need to be explained.

   Create AudioQueue

   Use the following methods to generateAudioQueueInstance

   12345678910111213

   OSStatus AudioQueueNewOutput (const AudioStreamBasicDescription * inFormat, AudioQueueOutputCallback inCallbackProc, void * inUserData, CFRunLoopRef inCallbackRunLoop, CFStringRef inCallbackRunLoopMode, UInt32 inFlags, AudioQueueRef * outAQ); OSStatus AudioQueueNewOutputWithDispatchQueue(AudioQueueRef * outAQ, const AudioStreamBasicDescription * inFormat, UInt32 inFlags, dispatch_queue_t inCallbackDispatchQueue, AudioQueueOutputCallbackBlock inCallbackBlock);

   First, let's look at the first method:

   The first parameter indicates the audio data format type to be played, which isAudioStreamBasicDescriptionObject, is to useAudioFileStreamOrAudioFileThe parsed data format information;

   Second ParameterAudioQueueOutputCallbackIs a Buffer.After being used;

   The third parameter is the context object;

   The fourth parameter inCallbackRunLoop isAudioQueueOutputCallbackThe RunLoop to be called back. If NULL is input, the callback will be sent again.AudioQueueThe internal RunLoop is called back, so it is generally possible to pass NULL;

   The fifth parameter, inCallbackRunLoopMode, is set to RunLoop mode. If NULL is inputkCFRunLoopCommonModesAnd pass NULL;

   The sixth parameter, inFlags, is a reserved field. It does not work currently and is passed to 0;

   The seventh parameter returns the generatedAudioQueueInstance;

   The return value is used to determine whether the creation is successful (OSStatus = noErr ).

   The second method is to replace RunLoop with a dispatch queue, and the other parameters are the same.

   Buffer-related method 1. Create a Buffer

   12345678

   OSStatus AudioQueueAllocateBuffer(AudioQueueRef inAQ, UInt32 inBufferByteSize, AudioQueueBufferRef * outBuffer); OSStatus AudioQueueAllocateBufferWithPacketDescriptions(AudioQueueRef inAQ, UInt32 inBufferByteSize, UInt32 inNumberPacketDescriptions, AudioQueueBufferRef * outBuffer);

   The first method is passed inAudioQueueThe size of the instance and Buffer, and the size of the Buffer;

   The second method can specify the number of PacketDescriptions in the generated Buffer;

   2. Destroy Buffer

   1	OSStatus AudioQueueFreeBuffer(AudioQueueRef inAQ,AudioQueueBufferRef inBuffer);

   Note that this method is generally used only when a specific buffer needs to be destroyed (because the dispose method will automatically destroy all buffers), and this methodOnly when AudioQueue is not processing dataCan be used. Therefore, this method is generally unavailable.

   3. Insert Buffer

   1234	OSStatus AudioQueueEnqueueBuffer(AudioQueueRef inAQ, AudioQueueBufferRef inBuffer, UInt32 inNumPacketDescs, const AudioStreamPacketDescription * inPacketDescs);

   There are two Enqueue methods. The first method and the second method are given above.AudioQueueEnqueueBufferWithParametersYou can perform more operations on the buffer of Enqueue. I have not studied the second method in detail. Generally, the first method can meet the requirements, here I will only describe the first method:

   This Enqueue method needs to be passed inAudioQueueThe instance and the Buffer that requires Enqueue. For inNumPacketDescs and inPacketDescs, You need to select the input parameters as needed. The document says that these two parameters are mainly used for VBR data playback, however, we have mentioned that even the CBR data AudioFileStream or AudioFile will provide PacketDescription, so this cannot be generalized. Simply put, if PacketDescription is passed, NULL is given. You don't have to worry about whether it is a VBR.

   Playback Control 1. Start Playing

   1	OSStatus AudioQueueStart(AudioQueueRef inAQ,const AudioTimeStamp * inStartTime);

   The second parameter can be used to control the start time of playing. Generally, you can directly start playing and pass in NULL.

   2. Decode data

   123	OSStatus AudioQueuePrime(AudioQueueRef inAQ, UInt32 inNumberOfFramesToPrepare, UInt32 * outNumberOfFramesPrepared);

   This method is not commonly used because it is called directly.AudioQueueStartDecoding starts automatically (if needed ). The parameter is used to specify the number of frames to be decoded and the number of frames actually decoded;

   3. pause playback

   1	OSStatus AudioQueuePause(AudioQueueRef inAQ);

   It should be noted that this method will be immediately paused once it is called, which meansAudioQueueOutputCallbackThe callback will also be paused. In this case, pay special attention to the thread scheduling to prevent the thread from waiting infinitely.

   4. Stop playing

   1	OSStatus AudioQueueStop(AudioQueueRef inAQ, Boolean inImmediate);

   If the second parameter is set to true, the playing (synchronization) will be stopped immediately. If the parameter is set to falseAudioQueueIt will play all the buffer of the Enqueue and then stop (asynchronous ). When using this function, you must specify the appropriate parameters as needed.

   5. Flush

   12	OSStatusAudioQueueFlush(AudioQueueRef inAQ);

   After the call, the decoder status will be reset after playing all the buffer of Enqueu to prevent the current decoder status from affecting the decoding of the next audio segment (such as switching the Playing Song ). If andAudioQueueStop(AQ,false)It does not take effect because the false parameter of the Stop method does the same thing.

   6. Reset

   1	OSStatus AudioQueueReset(AudioQueueRef inAQ);

   ResetAudioQueueWill clear all buffer with Enqueue and triggerAudioQueueOutputCallback, CallAudioQueueStopThis method is also triggered. The direct call of this method is generally used in seek to clear the residual buffer.AudioQueueStop).

   7. Obtain the playback time

   1234	OSStatus AudioQueueGetCurrentTime(AudioQueueRef inAQ, AudioQueueTimelineRef inTimeline, AudioTimeStamp * outTimeStamp, Boolean * outTimelineDiscontinuity);

   Among the input parameters, the First and Fourth parameters areAudioQueueTimelineWe didn't use it here. We passed in NULL. Returned results after the callAudioTimeStampYou can obtain the playback time from the timestap structure. The calculation method is as follows:

   12	AudioTimeStamp time =...; // obtain NSTimeInterval playedTime = time. mSampleTime/_ format. mSampleRate using the AudioQueueGetCurrentTime method;

   Note the following when using this time to obtain the method:

   1. In the first case, the playback time refersActual playback timeIt is different from playing progress in general. For example, after you start playing the video for 8 seconds, you can operate the slider to play the progress seek to 20th seconds and then play the video for 3 seconds. In this case, the playing time is usually 23 seconds, the playback progress.GetCurrentTimeThe actual playback time is 11 seconds. Therefore, the timingOffset of seek must be saved for each seek:

   12345

   AudioTimeStamp time = ...; // obtain NSTimeInterval playedTime = time by using the AudioQueueGetCurrentTime method. mSampleTime/_ format. mSampleRate; // The playback time of seek. NSTimeInterval seekTime = ...; // time at which seek is required NSTimeInterval timingOffset = seekTime-playedTime;

   The playback progress after seek needs to be calculated based on timingOffset and playedTime:

   1	NSTimeInterval progress = timingOffset + playedTime;

   2. Note thatGetCurrentTimeThe method sometimes fails, so it is best to save the last obtained playback time. If the call fails, the last saved result is returned.

   Destroy AudioQueue

   1	AudioQueueDispose(AudioQueueRef inAQ, Boolean inImmediate);

   All the buffers are cleared when the second parameter is destroyed. The meaning and usage of the second parameter are as follows:AudioQueueStopThe method is the same.

   When using this method, you must note that whenAudioQueueStartAfter the callAudioQueueIn fact, there is a short gap between them. IfAudioQueueStartAfter the callAudioQueueCalled during the period before the actual start of OperationAudioQueueDisposeThe program will be stuck. This problem was discovered when I used AudioStreamer. It must be found in iOS 6 (I did not test iOS 7, but iOS 7 was not released when I found the problem ), the reason is that AudioStreamer enters the Cleanup link when the audio EOF is enabled. The Cleanup link flush all the data and then calls the Dispose function. Therefore, when there is very little data in the audio file, it may occur.AudioQueueStartAt the time of the call, the EOF enters Cleanup, and the above problem occurs.

   To avoid this problem, the first method is to schedule the thread and ensure that the call of the Dispose method must be after each RunLoop (that is, at least after a buffer is successfully played ). The second method is to listenkAudioQueueProperty_IsRunningAttribute.AudioQueueAfter the Start method is called, it is changed to 1 and 0 after it is stopped. Therefore, make sure that after the Start method is calledIsRunningCan be called only when the value is 1.

   Attributes and Parameters

   And otherAudioToolBoxClass,AudioToolBoxMany parameters and attributes can be set, retrieved, and listened. The following are related methods, so we will not repeat them here:

   123456789101112

   // Parameter-related method AudioQueueGetParameterAudioQueueSetParameter // Attribute-related method listener // Method for listener attribute change AudioQueueAddPropertyListenerAudioQueueRemovePropertyListener

   Attribute and parameter list:

   12345678910111213141516171819202122232425262728293031323334

   // Attribute list enum {// typedef UInt32 AudioQueuePropertyID kAudioQueueProperty_IsRunning = 'aqrn ', // value is UInt32 success = 'aqsr', // value is Float64 success = 'aqdc ', // value is UInt32 kAudioQueueProperty_CurrentDevice = 'aqcd', // value is CFStringRef kAudioQueueProperty_MagicCookie = 'aqmc ', // value is void * Signature = 'xops ', // value is UInt32 kAudioQueueProperty_StreamDescription = 'aqft ', // value is AudioStreamBasicDescription kAudioQueueProperty_ChannelLayout = 'aqcl', // value is AudioChannelLayout layout = 'aqme ', // value is UInt32 kAudioQueueProperty_CurrentLevelMeter = 'aqm', // value is array of AudioQueueLevelMeterState, 1 per channel Limit = 'aqm', // value is array of AudioQueueLevelMeterState, 1 per channel identifier = 'dcb', // value is UInt32 kAudioQueueProperty_ConverterError = 'qcve', // value is UInt32 kAudioQueueProperty_EnableTimePitch = 'q _ tp ', // value is UInt32, 0/1 kAudioQueueProperty_TimePitchAlgorithm = 'qtpa ', // value is UInt32. See values below. signature = 'qtpe', // value is UInt32, 1 = bypassed}; // parameter list enum // typedef UInt32 AudioQueueParameterID; {Signature = 1, kAudioQueueParam_PlayRate = 2, kAudioQueueParam_Pitch = 3, kAudioQueueParam_VolumeRampTime = 4, kAudioQueueParam_Pan = 13 };

   Among them, valuable attributes include:

   • kAudioQueueProperty_IsRunningListen to it to know the currentAudioQueueWhether it is running. The function of this parameter is described.AudioQueueDispose.
   • kAudioQueueProperty_MagicCookieMagicCookie must be set for some audio formats.AudioFileStreamAndAudioFile.

     Valuable parameters include:

     • kAudioQueueParam_VolumeIt can be used to adjustAudioQueueThe playback volume. Note that the volume isAudioQueueThe internal playback volume and system volume are independently set and the final superposition takes effect.
     • kAudioQueueParam_VolumeRampTimeParameters andVolumeParameters can be used in combination to enable fade-in and fade-out of audio playback;
     • kAudioQueueParam_PlayRateThe parameter can be used to adjust the playback speed; Postscript

       This articleAudioQueueThe topic is over. The method mentioned above can meet most of the playback requirements,AudioQueueMore than that,AudioQueueTimeline,Offline Rendering,AudioQueueProcessingTapI have not covered and researched other functions yet. In the future, more new functions may be added, so there is no end to learning.

       In addition, becauseAudioQueueThe related content cannot be presented separately in Demo, so I will give the Demo of the last content (a simple local audio player) here for your convenience.AudioQueue. If you find it hard to understand a part of the content mentioned above, please leave a message below or share it with me on Weibo, you can also read official documents (I always think that official documents are the best way to learn );

       Sample Code

       Both AudioStreamer and FreeStreamer use AudioQueue.

       Next announcement

       The next article describes how to useAudioSession,AudioFileStreamAndAudioQueueImplement a simple local file player.