1 Audio system - function
1.1 Overview
...
The input acts as the standard audio interface, where the sub applications is delivering their audio samples. After the samples are hitting the audio output, then optional processing is applied to the audio signal. Then a volume control, is used to set the audio level for the input. As the audio output can have one or more audio outputs, grouped in audio busses, then the audio signal can be assigned to one or more busses. As the audio input channel count might not match up with the output channel count, then a render layout, is used to define, how to route the input signal to the audio output.
1.1 Audio input
...
The audio input provides an sample rate converter (SRC) for the cases, where the source sample rate don't match the audio output sample rate. The audio input also provides a buffer, as the audio samples is delivered in chunks. The Av Delay provides the possibility to align the audio and video sync. The Fx chain provides optional audio processing, for eg speech channels. The Fx chain settings can be done preset vice, by the content type. Eg. then movies don’t needs any processing. The input volume is for adjusting the volume for the specific audio input, before the audio signal is mixed to the audio bus.
1.2 Bus selection
...
Each input can be enabled to each bus. As the input audio channel count might not match the Bus audio channel count, then a Audio render layout – is used to define, how the audio channels should be aligned. One case can be a 5.1 input connected to a stereo bus. If the 5.1 source signal ain't properly down mixed to stereo, then the only the front L and R speaker informations will be played – and essential informations from eg the center and LFE channels is not played.
1.3 Audio render layouts
...
When the inputs needs to be routed to a audio bus, then the audio render layouts define, how the signal should be routed to the audio bus. When eg. The Audio Settings window is opened via the toolbar Config > Audio Settings
Sources
...
Every audio capable Layer will show within the sources menu, where the volume and mute settings can be configured. These settings are also available within the Playback tab of the Layer.
...
Layer types which can contain multiple sublayers, such as Compositions or Clip Banks, will have an Audio Bus, providing independent audio controls for the Bus and its sources.
...
Channel Mappings
When audio sources are routed to the output device, the channel mappings dictate how the channels will be mapped. By default,
A 5.1 signal is routed to a 5.1 bus
...
, as a 1:1
...
mapping
A 5.1 signal is routed to a stereo bus,
...
1.4 Audio render processing blocks
...
The audio render processing blocks, shares the same optional Fx chain as the inputs – just to keep it simple. In some situations, then some parts of the Fx chain can be useful – eg in cases, where a stereo signal needs to be unwrapped to 5.1. In such a situation, then it is useful, to get the low end from the stereo signal, played by the subwoofer. Each processing block also got it's own volume control.
1.5 Output master groups
...
The physical audio outputs can be grupped, in discrete channels, to Master channel groups. Hereby eg. 8 channel audio output card, can have two default configurations – one 5.1 master output, and one stereo output.
For other needs, then the Master groups can be assigned to the number of discrete audio outputs – depending on the needs.
When creating a grouping of audio output's, then a separate mix bus is created, for the audio output group.
Each group have it's own master volume.
2 Implementation of the audio system
The audio system relies within the ofx add-on ofxIglooAvUtils. This add-on includes both the audio system classes, and the FFmpeg AvLibs. This add-on is the only place, where the AvLibs get’s updated!.
2.1 ofApp implementation
When you need to implement the audio interface within a app, then you first need to include:
#include <ofxIglooAvUtils/src/audioOutput.h>
Next, you need to create a pointer for the audioOutput:
audioOutput* pAppAudio;
Within your ofApp::setup() add:
pAppAudio = new audioOutput();
pAppAudio->setup();
The audio output class is loading and applying the audio configuration during the setup() call, and is ready to rock after this single call.
Please note: The audioOutput class is creating an internal audioOut() callback function, so you don’t need to add extra code, within the ofApp::audioOut().
2.2 How to pass audio to the audioOutput?
When you need to pass audio to the audio output, then you first need to get a pointer to a audioInput. This is done, by calling the audioOutput->setupSource() function. This function will return a pointer to your audioInput object. Each time the audioOutput->setupSource() is called, then a new audio input will be created. So for instance, if you got an app, where you need to playback audio from eg. 10 different classes, then each class needs each own audioInput. The audioOutput class will then deal with all the concurrent 10 audio input’s - and eg mix them all together.
There exist no limitation in the number of audio channels you can use, with an audioInput(). The channel count can be all from 0 audio channels, to infinite.
On the sample rate of things, then you simply go tell the audioInput() what sample rate your audio samples is in - then the audioInput() is performing the sample rate translation, to match the audioOuput() sample rate on it’s own.
Depending on the application, then you can deliver the audio samples to the audioInput object either once you have the samples, or eg for movie playback, you can deliver the samples to the audioInput, using a callback function. The callback function is handy, for situations, where you eg need to pull your audio samples.
When you are done, using the audioInput, then you need to release the object from the audioOutput. This is done by calling the audioOutput->closeInput() - with a pointer to the audioInput object.
2.2.1 Creating the audioInput
When you need to create a audioInput, for delivering audio samples to the audio output, then in your .h file you first need to include:
#include <ofxIglooAvUtils/src/audioOutput.h>
#include <ofxIglooAvUtils/src/audioInput/audioInput.h>
Next you need to declare a pointer for the audioInput object:
shared_ptr<audioInput> pAudioBuffer;
If you need to use a callback function, then you need to define this function as:
void pullAudio();
Next step is then to setup the audio input, and get the pointer:
audioOutput->setupSource(bind(&::pullAudio, this), audioChannels, sampleRate, "Label", 0);
In cases, where you dont need a callback function - then simply pass a NULL instead of the callback function. audioChannels, is the number of audio channels you need to playback. sampleRate is the samplerate to use - the samplerate can easily be changed later on. The “label” is simply a text label, that can be presented on the GUI - for letting the user know, what’s on the input.
2.2.2 Delivering of the audio samples
When you want to deliver audio samples to the audio output, then this is done by calling:
pAudioBuffer->addSamplesToBuffer(float* pAudioSamples, int samples, int64_t pts)
pAudioSamples is a float* to your buffer, and samples is the number of samples you want to deliver. pts is currently not implemented.
Several addSamplesToBuffer() overloads exist. Eg if you are having pointers to different streams, each holding a number of audio channels, then you can use the overloaded function:
addSamplesToBuffer(float* pAudioSamples, int samples, int channelOffSett, int sourceChannels, int64_t pts)
Here you are simply telling the audioInput(), to place the audio samples for a number of audio channels, with an offsett defined by channelOffSett.
2.2.3 Audio callback function
The audio callback function is called each time, our audio output needs some more samples - from it’s output buffer. As the audio output can use a different sample rate and/or a different buffer size, then the audio callback function don’t necessarily reflects, that your audioInput() buffer needs to be filled, with more audio samples. So when using the audio callback function - then your application need to detect, if it needs to deliver some more audio samples or not.
2.2.3 Audio samples formats
A key thing, when working with audio samples, if how this samples are placed within the data buffer.
The standard OF format is to use interlanced samples, and this is the default format for the audioOutput class. The constellation of the audio samples, in the interlaced format, shown here on a stereo stream:
LRLRLRLR
The interlaced format is simply one sample for the Left channel, one sample for the Right channel - and so on.
Some applications, is using a different approach, for storing the audio samples:
LLLLRRRR
This is a non interlaced audio stream. Within one buffer of audio samples, then the audio samples for each audio channel is placed, channel after channel. For such applications, then you can use the function:
addNonInterleavedSamplesToBuffer(float* pAudioSamples, int samples, int64_t pts)
And the audio sub system will take care of this for you.
2.2.4 Buffer management
In some cases, then you don’t want to buffer all your audio samples on your audioInput() class. For instance, then the AvPlayer is buffering the audio samples from the movie. But, with the AvPlayer, then we need to use the audio output for steering the playback speed. If we didn’t do so, then we would either have audible ticks and clicks, or simply lack the audio and video alignment. So in this case, then we don’t want to use a lot of buffering on our audioInput() - and do the buffering within the AvPlayer.
So in this case, we will keep the audioInput() buffering low as possible.
The audioInput() class got some functions, to help managing the current buffer levels.
audioInput->getBufferSize() returns the current audioInput() buffer level
audioInput->getFramesPrBuffer() returns the audioOutput() audio buffer size
So in this case, where our primary interest is to keep the audioInput() buffering low as possible, then it’s simply a matter of:
if(audioInput->getBufferSize() > audioInput->getFramesPrBuffer()) return;
The audioInput->getFramesPrBuffer() reflects the number of samples about to be pulled by our audio output. As long as our input buffer is at least holding this number of samples - then we are just fine.
...
by downmixing to stereo.
Output Device
...
ICE supports WASPI and ASIO audio devices. Select the output device from the list.