Pitch Tracking in Time Domain J S Roger

Pitch Tracking in Time Domain J. -S. Roger Jang (張智星) MIR Lab, CSIE Dept. National Taiwan University jang@mirlab. org, http: //mirlab. org/jang 2021/3/7

Outline Pitch tracking PDF (periodicity detection function) in time domain � ACF: Autocorrelation function NSDF: Normalized squared difference function � AMDF: Average magnitude difference function 2/32

Acoustic Features in Time Domain Basic acoustic features in time domain � Volume � Pitch � Timbre frame. Disp 4 fea 01. m 3/32

Acoustic Features in Frequency Domain Basic acoustic features in time domain � Volume (or energy) � Pitch (or FF) frame. Disp 4 fea 02. m � Timbre 4/32

Fundamental Frequency and Pitch Fundamental frequency (FF) � No. of fundamental periods in a sec � Unit: Hertz or Hz Not to be confused with sample rate! Pitch � Computed from FF via a log-based formula: � Unit: semitone or MIDI number Piano roll via HTML 5 5/32

Pitch Tracking (音高追蹤) Pitch tracking (PT) � The process of computing the pitch vector of a given audio segment (對整段音訊求取音高) Applications � Query by singing/humming (哼唱選歌) Quiz! � Tone recognition for Mandarin (華語的音調辨識) � Intonation scoring for English (英語的音調評分) � Stress detection in English word (英語單字的重音偵測) � Text-to-speech synthesis (語音合成) � Pitch/duration modification (音高與音長改變) � Motor RPM detection via sound (由聲音決定馬達轉速) � Heart rate calculation via ECG (由心電圖計算心跳率) �… 6/32

Frame Blocking wave 2 frame. m 7/32

More Terms on Frame Blocking Quiz! Overlap Frame Sample rate = 16 k. Hz Frame size = 512 samples Frame duration = 512/16000 = 0. 032 s = 32 ms Overlap = 192 samples Hop size = frame size – overlap = 512 -192 = 320 samples Frame rate = 16000/320 = 50 frames/sec frame size = hop size + overlap hop size overlap 8/32

Typical Steps for Pitch Tracking Pre-processing (segment based) � Filtering � Excitation extraction Main processing (frame by frame) � PDF (periodicity detection function) computation � Pitch candidates via max picking over PDF � Pitch refinement via parabolic interpolation (optional) Post-processing (segment based) � Unreliable pitch removal via volume/clarity thresholding � Pitch smoothing via median filters, etc. 9/32

PDF: Periodicity Detection Functions Goal of PDF: to detect the pitch of a frame Two types of PDF � Time domain (時域) ACF (Autocorrelation function) AMDF (Average magnitude difference function) … � Frequency domain (頻域) Harmonic product spectrum Cepstrum … 10/32

ACF: Auto-correlation Function A frame is represented by s(t), t=0~n-1 1 128 Original frame s(t): Shifted frame s(t-t): t=30 acf(30) = inner product of the overlap part Quiz! Period Quiz! To play safe, the frame size needs to cover at least two fundamental periods! 11/32

Facts about ACF Some facts about ACF � It is a function of t, the time delay. Quiz! � We need to perform mean justification (subtract the mean from each element in the frame) before ACF. � ACF’s value is getting smaller due to smaller overlap for inner product. � We need to have a better criterion (to be detailed) for picking the right maximum. 12/32

ACF: Formula 1 A frame is represented by s(t), t=0~n-1 s(t) Shift to right s(t): t s(t-t): t ACF formula 1: s(t-t) Quiz! 13/32

ACF: Formula 2 A frame is represented by s(t), t=0~n-1 s(t): t Shift to left s(t+t): t s(t+t) ACF formula 2: This formula is the same as the previous one! Quiz! 14/32

ACF Example Sunday. wav frame 2 acf 01. m � Sample rate = 16 k. Hz � Frame size = 512 (starting from point 9000) Fundamental frequency � Max of ACF occurs at index 130 � FF = 16000/(130 -0) = 123. 077 Hz Index 0 Index 130 Here we use 0 -based indexing. 15/32

Locate the Right Max in ACF If human’s FF range is [40, 1000], then the interval to locate fundamental period (FP) is: frame 2 acf. Pitch. Point 01. m Sample rate Index: 0 Index: FP Quiz! 16/32

What Could Go Wrong? Human pitch range could go wrong � Pitch too high Vitas (local short clip) Whistling � Pitch too low Requires a big frame size to cover at least two fundamental periods Quiz! 17/32

Example of ACF-based PT (1/2) Specs � Sample rate = 11025 Hz � Frame size = 353 points = 32 ms � Overlap = 0 � Frame rate = 31. 25 f/s wave 2 pitch. By. Acf 01. m Playback � Original singing � Pitch by ACF 18/32

Example of ACF-based PT (2/2) Specs � Same as previous example � By calling pitch. Track. Basic. m directly pt. By. Acf 01. m Playback � Play audio (left channel) and pitch (right channel) at the same time 19/32

Demo of ACF-based PT Real-time display of ACF for pitch tracking (SAP toolbox) � go. Pt. By. Acf. mdl Pre-recorded audio file � go. Pt. By. Acf 2. mdl Live microphone input 20/32

ACF Variants to Avoid Tapering Normalized version Frame 2 acf 02. m Half-frame shifting Frame 2 acf 03. m 21/32

NSDF: ACF Variant with Normalized Range NSDF: Normalized squared difference function � Formula: � A variant of ACF within the range [-1, 1], based on the inequality: 22/32

NSDF Example Clarity: height of the pitch point frame 2 nsdf 01. m 23/32

AMDF: Average Magnitude Difference Function A frame is represented by s(t), t=0~n-1 1 128 Original frame s(i): Shifted frame s(i-t): t=30 Quiz! amdf(30) = sum of abs. difference of the overlap part Period 24/32

Comparison between ACF & AMDF Formulas � ACF: � AMDF: Quiz! Two major advantages of AMDF over ACF � AMDF requires less computing power Subtraction instead of multiplication No need to have mean justification in advance � AMDF is less likely to run into the risk of overflow 25/32

Example of AMDF suday. wav frame 2 acf 01. m � Sample rate = 16 k. Hz � Frame size = 512 (starting from point 9000) Fundamental frequency � Pitch point occurs at index 130, which is harder to determine Index 0 Index 130 26/32

Example of AMDF to Pitch sunday. wav frame 2 amdf 4 pt 01. m � Sample rate = 16 k. Hz � Frame size = 512 (starting from point 9000) Fundamental frequency � Pitch point occurs at index 130, which is determined correctly � FF = 16000/(130 -0) = 123. 077 Hz Index 0 Index 130 27/32

Example of AMDF based PT Specs � Sample pt. By. Amdf 01. m rate = 11025 Hz � Frame size = 353 points = 32 ms � Overlap = 0 � Frame rate = 31. 25 f/s Playback � Original singing � Pitch by AMDF 28/32

AMDF: Variations to Avoid Tapering Normalized version Frame 2 amdf 02. m Half-frame shifting Frame 2 amdf 03. m 29/32

How to Enhance ACF’s Peaks? Divide ACF by AMDF to enhance peaks Frame ACF AMDF ACF/AMDF Enhanced peak 30/32

Fundamental Frequency to Semitone Conversion Semitone: A music scale based on A 4 of 440 Hz Reasonable pitch range of humans � E 2 (82 Hz) Quiz! – C 6 (1047 Hz) Reference � 超簡單樂理 Lesson 1 by嘎老師 31/32

References 如何執行相關程式碼 � 下載相關 具箱（下載後，需使用addpath指令將 具箱 加入搜尋路徑） Utility Toolbox SAP Toolbox Machine Learning Toolbox � 下載程式包 example. zip 大部分的程式範例及相關說明 � Audio Signal Processing 32/32