Improved Headphone Sound Stage Background to ENCM 515

Improved Headphone Sound Stage Background to ENCM 515 Labs on DSO Audio Channel Modeling Based on concepts in Bessinger Thesis Various copyrights M. Smith, E. Bessinger and L. E. Turner M. Smith, Electrical and Computer Engineering, University of Calgary, Canada smithmr @ ucalgary. ca

To be tackled today n n Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines n n Sound colouration through FIR filters n n Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab. 2 -- Implementation in “C” (with and without pointers) Lab. 3 -- Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Lab. 4 -- In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling n Lab. 5 -- Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project? 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 2

Main reference material E. C. Bessinger, Localization of Sound Using Headphones M. Sc. thesis, Electrical and Computer Engineering, January 1994. Supervisor Dr. Turner Translated for DSP processors by M. Smith 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 3

Improved Head Sound Stage Actual Mono Sound Source Perceived Repositioned Sound Source 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 4

To be tackled today n n Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines n n Sound colouration through FIR filters n n Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab. 2 -- Implementation in “C” (with and without pointers) Lab. 3 -- Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Lab. 4 -- In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling n Lab. 5 -- Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project? 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 5

Concept behind Labs. n OFF-LINE initial development using Visual. DSP++ simulator n n n Sound Source -- fixed input data array Process Sound -- KNOWN ALGORITHM Sound Sink -- fixed output data array Testing -- plotting and comparing to theory Possible algorithmic time constraints investigated ON-LINE (REAL-TIME) testing using SHARC board n n Sound Source -- background task (or CODEC) Process Sound -- SAME KNOWN ALGORITHM Sound Sink -- CODEC background task Testing -- using aurally using GOOD earphones and quantified using known sound source 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 6

Labs. 2 and 3 -- delay line -- Concept DELAY 0 DELAY_LEFT DELAY_RIGHT n n No relative delay modelled into the audio channel -- then sound perceived in centre of head Modelling a relative delay into the right ear audio channel. Sound arrival will shift sound to left as “sound” seems to get to left ear first 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 7

Delay Necessary -- Calculation Distance to Sound Time Delay = -------------Speed of Sound Algorithm Delay = Time Delay * Sampling Rate Relative Distance Speed of sound Sampling Rate Delay needed in array Sound resolution 12/2/2020 = = = 0. 3 m 300 m/s 44 k. Hz ? ? ? ? Reasonable or NOT? ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 8

GLOSSARY n FIFO n n LIFO n n n Delay Line -- Implementation using FIFO Last In -- First Out stack C-stack -- Implementation using LIFO FIR IIR FT (CFT) DFT n First In -- First Out stack FFT 12/2/2020 Finite duration Impulse Response Infinite duration Impulse Response continuous Fourier transform discrete Fourier transform -- fast Fourier transform which is an implementation of DFT ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 9

To be tackled today n n Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines n n Sound colouration through FIR filters n n Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab. 2 -- Implementation in “C” (with and without pointers) Lab. 3 -- Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Lab. 4 -- In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling n Lab. 5 -- Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project? 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 10

Real-time N-pt Delay code concept void Process. Sound(int channel_one, int channel_two, int *left_channel, int *right_channel { if ((sound_source & LEFT_DELAY) == LEFT_DELAY) { if ((sound_source & MOVEDELAY) == MOVEDELAY) Memory. Move_Left. Delay(process_var 1, &channel_one); -------------------------- void Memory. Move_Left. Delay(int N, int *channel_one) { float left_delayline[MAXDELAY + 1] = {0}; WHAT’S WRONG AND MISSING? // Insert new value into the back of the FIFO delay line left_delayline[0 + N] = (float) *channel_one; // N delays implimented // Grab delayed value from the front of the FIFO delay line *channel_one = (int) left_delayline[0]; // Update the FIFO delay line using inefficient mass memory moves for (count = 0; count < N; count++) <--- Or N + 1 or N - 1? left_delayline[count] = left_delayline[count - 1]; WHAT’S WRONG } 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 11

Basic Algorithm -- Handle Delay <----- LEFT DELAY -----> Sample into audio channel <-----DELAY -----> Requires 2 * (LD + RD - 2) memory operations Sample from channel 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 12

To be tackled today n n Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines n n Sound colouration through FIR filters n n Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab. 2 -- Implementation in “C” (with and without pointers) Lab. 3 -- Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Lab. 4 -- In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling n Lab. 5 -- Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project? 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 13

Labs. 4 -- delay line -- Concept FILTER 30 LEFT FILTER 330 RIGHT FILTER 30 RIGHT n n Get “ambience” by taking into account constructive and destructive interference around the face. This implies knowing characteristics of “audio channel” and modelling using an FIR filter -- 2 FIR filters per speaker -- processing requirement increasing 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 14

Delay is not only thing to take into account Four audio channels from 2 sources in front 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 15

Lab. 4 -- Transfer function h 30(f) 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 16

Lab. 4 -- FIR transfer -- h 330(f) 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 17

Why different Audio Channels? n n Delay is not only positioning element -- take into account actual direction Transfer function from loudspeaker to ear modelled by FIR filter. S Enough time to perform calculation between samples? 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 18

FIR schematic diagram 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 19

FIR = Complex Delay Line 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 20

Real-time FIR Filter void Process. Sound(int channel_one, int channel_two, int *left_channel, int *right_channel if ((sound_source & FIRFilter) FIRFilter(&channel_one, &channel 2); float fircoeffs_30[], fircoeffs[330]; void FIRFilter(int *channel_one, int *channel_two) { // Insert new value into FIFO delay line left_delayline[0 + N] = (float) *channel_one; right_delayline[0 + N] = (float) *channel_two; channel_one_30 = channel_one 330 = 0; Need equivalent of following loop for EACH sound source for (count = 0, count < FIRlength - 1, count++) { channel_one_30 = channel_one_30 + arrayleft[count] * fir_30(count); channel_one_330 = channel_one_330 + arrayright[count]* fir_330[count]; } *channel_one = (int) channel_one_30 + scale_factor * channel_one_30 ditto 2 Update Left Channel delay line; Update Right Channel Delay line ENCM 515 -- Background to Audio Channel Modelling } 12/2/2020 21 Copyright smithmr@ucalgary. ca

To be tackled today n n Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines n n Sound colouration through FIR filters n n Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab. 2 -- Implementation in “C” (with and without pointers) Lab. 3 -- Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Lab. 4 -- In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling n Lab. 5 -- Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project – Try on other processor architectures – try with video? 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 22

Lab. 5 -- Room Effects n Need to have multiple sources n n more FIR operations Need to have reflections off the wall -- possibilities n IIR filters, Comb, FFT (frequency domain) 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 23

Standard IIR filter example -- biquad SMALL DELAYS n n Normally in multi-stages Make parallel BETWEEN rather than IN stages -- see Smith (Micro Magazine article, 1992) 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 24

Real Time standard IIR code finalleftear = h 0 * lefttemp 1 + h 1 * lefttemp 2 + h 3 * lefttemp 3; finalrightear = h 0 * rightemp 1 + h 1 * righttemp 2 + h 3 * righttemp 3; newlefttemp 1 = gain * leftear + h 4 * lefttemp 2 + h 5 * lefttemp 3; newrighttemp 1 = gain * rightear + h 4 * rightemp 2 + h 5 * righttemp 3; lefttemp 3 = lefttemp 2; lefttemp 2 = lefttemp 1; lefttemp 1 = newlefttemp 1; righttemp 3 = righttemp 2; righttemp 2 = righttemp 1; righttemp 1 = newrighttemp 1; // Parallel ear calculations // Parallel stages // Optimization -- final exam question? // Optimization // Need other delay and FIR filters for audio channel characteristics 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 25

Lab. 5 -- IIR -- comb filter -- different Long delay line 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 26

Full Lab 5 -- FIR and IIR Time available if multi-task? 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 27

Lab. 5 requires 7 comb filters BIG DELAYS (Use PACK instructions or decimation? ) 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 28

Lab. 5 Possibilities -- FFT? n n Implement filters in frequency domain rather than in time domain Frequency domain algorithms can be considerably faster than time domain algorithms especially when performing several filtering operations on one set of data n Require more memory to process BLOCKS OF DATA 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 29

Real Time FIR Convolution Filter via multiplication in frequency domain FFT is faster that FIR and IIR if memory available (2 N log N + 4 N) rather than 3 * N operations for 512 points // Signal Conditioning -- Windowing wleftear[n] = leftear[n] * window[n]; FFT(wleftear, leftfrequency, forward FFT) for 512 points // Signal convolution with FIR frequency domain response new_leftfrequency 1[n] = leftfrequency[n] * filter 1[n] + leftfrequency[n] * filter 2[n] + leftfrequency[n] * filter 3[n]; FFT(new_leftfrequency, newleftear, inverse FFT) 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 30

DSP 4 -- FFT -- Decimation Require special architecture for speed 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 31

Major FFT implementation Reference n n Major reference on signal conditional and filter implementation in frequency domain FFT -- friscy Fourier Transforms? Smith (Microprocessors and Microsystems) 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 32

DIF -- FFT code -- in Fortran pseudo code DO 11 I = 1, 15 -- allows FFT size up to 2^15 M = I; N 2 = 2 **I; IF (N. EQ. N 2) go to 21 11 CONTINUE RETURN 21 N 2 = N DO 30 I = 1, M N 1 = N 2; N 2 = N 2/2; I 1 = 1; I 2 = N/N 1; -- indexes DO 30 J = 1, N 2 --- Note that N 2 gets smaller C= cosarray(I 1); S = sinarray(I 1); I 1 = I 1 + I 2 -- Note indexing format DO 30 K = J, N step N 1 L = K + N 2 tempr = xreal(K) - xreal(L); xreal(K) = xreal(K) + xreal(L) tempi = ximag(K) - ximag(L); ximag(K) = ximag(K) + ximag(L) xreal(L) = C * tempr + S * tempi; ximag(L) = C * tempi + S * tempr; 30 CONTINUE Bit. Reverse. Array. Manipulation(N, xreal, ximag) 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 33

To be tackled today n n Audio channel modelling concepts as detailed in Bessinger’s thesis on improved sound stage Sound re-positioning through delay lines n n Sound colouration through FIR filters n n Post Lab. 1 Quiz – Determination of delay effects using known input audio signals (simulated and real-time) Lab. 2 -- Implementation in “C” (with and without pointers) Lab. 3 -- Implementation in “assembly”, with and without pointers, with specialized SHARC architecture (hardware circular buffers) Lab. 4 -- In “C/C++”, assembly, custom assembly (hardware loops) and VERY custom assembly (highly parallel algorithm) Additional Audio Channel Modeling n Lab. 5 -- Multi-tasking environment -- SHARC RTOS -- Room colouration through IIR filters -- Student project – Try on other processor architectures – try with video? 12/2/2020 ENCM 515 -- Background to Audio Channel Modelling Copyright smithmr@ucalgary. ca 34