The Chinese University of Hong Kong Department of

The Chinese University of Hong Kong Department of Computer Science and Engineering Lyu 0202 Advanced Audio Information Retrieval System

Network-based Adv. AIR System l Consists of client side and server side l Client Side ¡ Consists of 2 parts Advanced Part • Audio Data Mining • Audio Data Retrieval and Indexing l Basic Part • Audio Streaming from Server side l l Server Side ¡ For Audio Streaming, Searching on Server

Advanced Part of Adv. AIR system l Audio Data Mining ¡ Segmentation ¡ Recognition Engine ¡ Segmentation with Speaker Recognition l Audio Retrieval and Indexing ¡ Query by Humming ¡ Pattern Matching ¡ Search on Server

Audio Data Mining – Recognition Engine l Consists of Three functions: ¡ ¡ ¡ l Speaker Recognition engine ¡ l Open-set system with 10 models and 1 general model Language Recognition engine ¡ l Speaker Recognition Language Recognition Gender Recognition Close-set system with 3 models (Cantonese, English, Mandarin) Gender Recognition engine ¡ Close-set system with 2 models (Male and Female)

Audio Data Mining - Segmentation Group 1 Group 2 Group 3

Audio Data Mining - Segmentation l Bayesian Information Criterion is used for determining the acoustic change point of the input Mpeg file l First, input an Mpeg file l Next, extract the features l Use BIC criterion to calculate the change point l Finally, have a list of segments which is cut according to acoustic change point

Audio Data Mining – Recognition Engine Trained Model Input Mpeg Extract feature Calculate a score For each model Select the most suitable model

Audio Data Mining – Recognition Engine l Use Gaussian Mixture Model ¡ text independent, robust, computationally efficient l 256 mixture for each models l Need pre-processing (Training) l First, input Mpeg file l Next, extract the features l Calculate a score for each models and select the model with the best score

Audio Data Mining – Segmentation with Speaker Recognition l Automatic speaker recognition engine l First, do segmentation l Next, each segmentation is sent to the speaker recognition engine l Finally, we get list of segments in which the speakers of each segment will be known

Group 1 Group 2 Group 3 Speaker identification Process Speaker 1 Speaker 2 Speaker 3 Speaker 2 Speaker 1 Speaker 2

Audio Retrieval and Indexing - Query by Humming l First Step: ¡ ¡ ¡ l Second Step: ¡ l Do Pitch Tracking using time domain autocorrelation function, ACF for the input audio clips Track the trend of input audio clips, in the manner of “UP”, “Down” or “Same” Intermediate output: a file consists of a list of “Up”, “Down”, “Same” Do largest substring matching for each of the intermediate output of audio clips in the database and the intermediate output of the input audio clip and calculate a score Last Step: ¡ List the audio clips in database according to the score

Intermediate representation Hummed Song Pitch tracker Intermediate Database Largest Substring matching

Pitch tracker l Tack the pitch of hummed voice, convert into representation of relative change of voice l E. g. Do Me Fa So Fa Re Me • U U U D D U

Audio Retrieval and Indexing – Direct Audio Search l First Step: ¡ covariance matrix is calculated from the feature vectors of the cue-audio and a clip in database l Second Step: ¡ AHS (arithmetic harmonic sphericity) distance measurement to calculate a score l Last Step: ¡ List the audio clips in database according to the score

Target Clips with Same size AHU Comparison Source Clip

Audio Retrieval and Indexing – Search on Server l Direct Audio Search on Server l Server Side has a database l Client connect to server l Client select a cue-audio and upload to the server l Server will do the direct audio search and send back the result l Client can use the audio streaming to get the result file

Basic Part - Audio Streaming Adv. AIR is N-to-N system, allow N server and N client l Client and Server can be added at any time l It’s Fault Tolerant l

Basic Part – Server Side l Have two parts: ¡ ¡ For Audio Streaming For Searching on Server (Direct Search on server) Separate it because Searching on Server use a lot of resource l A server can’t process for too many users at the same time l Only privileged users allow to use the searching on server function l

Basic Part – Client Side l Client request for download, an audio clips is divided into many small parts l Each server send a small parts to client simultaneously to speed up the download speed l Client combine all the small parts to form the whole file

The End