Developing a DSP Algorithm using a TTD process

Developing a DSP Algorithm using a TTD process Application FIR filter

TTD and XPI and TTD n n XPI – e. Xtreme Programming inspired life cycle TTD – test driven development n n n Main tool Automated testing framework Reference: - A More Agile Approach to Embedded System Development, Smith et al. , IEEE Software magazine, March, 2009. 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 2 / 33

XP inspired development process n Stage 1: Envisioning embedded product n This stage produces requirements. The customers or end users (domain-knowledgeable personnel) should be able to specify their ideas as a set of acceptance tests (in conjunction with standard user stories) n n n in a straightforward manner. User stories – Wish list of what you would like the product to do Now prioritize the wish list for the first part that customer identifies as most important. Plan to develop, deliver and demonstrate a prototype. 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 3 / 33

XPI: Stage 2: Prototyping n n Domain-knowledgeable technical personnel produce an initial prototype to analyze the proposed embedded solution. The solution would revolve around numerical algorithms implemented in a mathematically oriented scripting language such as Matlab. Developing additional acceptance test cases. Using TDD supported by a testing framework designed for Matlab development. Stage 1’s acceptance tests provide the initial skeleton for the prototype, Stage 2 adds “flesh” onto this outline. Testing FIR filters 1/26/2022 Copyright smithmr@ucalgary. ca 4 / 33

Stage 3: Initial XPI Production System n n We must rewrite the system in a high-level compiled language such as C or C++. The biggest issues here surround the refactoring necessary to support moving Matlab floating-point calculations onto high speed, low-cost embedded processors that can handle integer operations. Stage 3 basically repeats the pattern from Stage 2, but in a new programming language using a new unit-testing framework. However, because of the existence of the previous stages, we’ve already constructed numerous tests. 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 5 / 33

Stage 4: Full Production System n n By the fourth stage, we’re fully involved with the target embedded environment itself. The system, although repeating earlier patterns, now needs to adapt to its new hardware environment with limited memory resources and must support interactions with device-specific peripherals on the target system. The system undergoes further modification to tune for the required performance by n rewriting inefficient programming constructs, n using signal-processing-specific #pragma state- to direct architecturally aware compilers, and n rewriting critical components in low-level, highly customized machine code. 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 6 / 33

Assign. 1 n From individual assignment 1 you have n n C++ un-optimized averaging filter Compiler optimized C++ averaging filter ASM version – software loop – averaging filter Tests for averaging filter (function, timing and theory comparison) 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 7 / 33

Steps in optimizing n n y[m] = Sum x[m – n] where n = 0 to N-1 Data stored oldest to newest x[m-n+1], x[m-n+2] … x[1], x[0] FIFO[0] ………………. . . FIFO[n-1] so that possible use of hardware circular buffers which go forward We code as sum = 0; for count = 0 to N-1 sum = sum + FIFO[n-1 -count]; This “array” code requires specific DAG memory adds – slow because difficult to combine with other instructions (not enough instruction bits) 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 8 / 40

Steps in optimizing – Step 2 n n n We code as sum = 0; for count = 0 to N-1 sum = sum + FIFO[n-1 -count]; This “array” code requires specific DAG memory adds – slow because difficult to combine with other instructions (not enough instruction bits) Change to pointer operation using post decrement FIFOpt = & FIFO[n-1 -count]; sum = 0; for count = 0 to N-1 sum = sum + * FIFOpt--; Change to averaging FIR filter which access coefficients FIRcoeffs[N ] = {1, 1, 1, … 1, 1}; FIRcoeffpt = &FIRco eff[0] FIFOpt = & FIFO[n-1 -count]; sum = 0; for count = 0 to N-1 sum = sum + (* FIFOpt--) * (* FIRcoeffpt++); Note the two meaning of the C++ operator * n n Multiplication of two variables (integer, float (single and extended), complex int are possible) Contents of memory location pointed to by a pointer 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 9 / 40

Steps in optimizing – Step 3 n n Change to averaging FIR filter which access coefficients FIRcoeffs[N ] = {1, 1, 1, … 1, 1}; FIRcoeffpt = &FIRco eff[0] FIFOpt = & FIFO[n-1 -count]; sum = 0; for count = 0 to N-1 sum = sum + (* FIFOpt--) * (* FIRcoeffpt++); Refactor to take advantage of processor DAG characteristics n n Most processors are faster at incrementing forward – and the instruction works in parallel with more instructions FIRcoeffs[N ] = {1, 1, 1, … 1, 1}; FIRcoeffpt = &FIRco eff[0] FIFOpt = & FIFO[0]; sum = 0; for count = 0 to N-1 sum = sum + (* FIFOpt++) * (* FIRcoeffpt++); Change away from {1 …. . 1} and you have normal FIR filter code 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 10 / 40

Lab. 1 Code conversion: Averaging to FIR n n Average Filter -- Do Buffer update loop then do fetch a new data memory value with add loop FIR -- Modify second loop to bring in second data value (FIR coeffs) and perform multiply and add n n C++ un-optimized FIR -- data and coeffs from dm Compiler optimized C++ FIR -- data and coeffs from dm ASM version – software loop – FIR filter -- data and coeffs from dm Test -- (function, timing and theory comparison) n Averaging filter has same functional results as FIR if you test with coefficients (1/N, 1/N …. 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 11 / 33

Lab. 1 Code conversion: New FIR n n Add new FIR specific tests – discussed today Modify one loop to bring in second data value (FIR coeffs) and perform multiply and add n n n Compiler optimized C++ FIR – data from dm and coeffs from pm ASM version – software loop – FIR filter Other optimizations will become part of Lab. 2 n – data from dm and coeffs from pm 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 12 / 33

Lab. 1 – How to test a totally new DSP algorithm n n For any new algorithm We could write the code in Matlab n n We then translate the Mat. Lab code into C n n Then write tests in Matlab to show works Then translate the Mat. Lab tests into C and use them. In Lab. 1, we are going to take a short cut as we know what should be happening with FIR – straight to C and Testing 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 13 / 33

Rough idea of function n Insert a new value into the “new” location of a FIFO array Update FIFO filter to throw away oldest value n Perform FIR filter operation n For (I = 0; I < length. FIR) sum = sum + FIFO(length. FIR – I) * FIRcoeffs(I) n n Return sum Write code in a way that the code is testable 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 14 / 33

Write for Testable means what? n n n Ability to easily test with different length of FIR filters Ability to easily test with different FIR filter coefficients Ability to test with a variety of input values non-Functional test – ability to time execution time Ability to be able to test n n Outside u. TTCOS – automated tests on ‘mock’ data Inside u. TTCOS – human acceptance test using real audio data – Should be smooth audio performance – not hick-cups caused by lost signals 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 15 / 33

What to worry about n n FIR filter of length ? Actual values of FIR coefficient are ? ? n n Unimportant when not inside u. TTCOS as we are not expecting ‘acceptable’ audio signals, just correct results Very important when used in human testing – inside u. TTCOS n n Must be stable – otherwise will blow your ears out Must be known so you can use your ears to validate that the correct sort of filtering is occurring 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 16 / 33

Story 1 – Basic ‘does anything work’ test n n FIR filter of length 4 “Fake” known FIR coefficients – A 0, A 1, A 2 and A 3 n n A 0 = 2. 0 (never use 1. 0 since 1. 0 * 1. 0 is the same as 1. 0 * 1. 0 and 1. 0 – meaning if there is a mistake in coding you would never know A 1 = A 2 = A 3 = 0. 0 (at the moment) Process the “fake” audio array Input[ ] = 0, 0, 0, 3, 0, 0, 0, 0 n Note that the impulses of amplitude “ 3” are further apart than FIRlength 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 17 / 33

Story 1 – Basic ‘does anything work’ test n Input[ ] = 0, 0, 0, 3, 0, 0, 0, 0 n n n Note that the impulses of amplitude “ 3” are further apart than FIRlength Values read out of the array and placed in Leftchannel 1 each time we apply the filter Used as In 1 = 0; In 2 = 0; In 3 = 0; In 4 = 0; In 5 = 0 etc 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 18 / 33

What is expected result? n Write a test, Watch it Fail when only a stub CPP or ASM function is present n n Write the code to satisfy the test n n the test of the test. Must fail when no code is present otherwise is useless test The test of the code Remember – the test writing is expected to be easy for the C++ code n So generate sensible written tests for the C++ code and modify them for use when we optimize the assembly code 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 19 / 33

What is the expected result? n Fake FIR coefficients 2. 0, 0, 0, 0 DATA FILTER COEFFS ARRANGED IN REVERSED ORDER as A 0 is multiplied by newest value n n Therefore result (two sets of impulse responses) would be filtered. Input[ ] = 0, 0, 0, 6, 0, 0, 0, 8, 0, 0, 0 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 20 / 33

n Go to. xls file 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 21 / 40

FIR in imaging a row n FIR filter of size 40 on image of size 256 n n Get initial and final transients Image might look like this B 1, B 1, I, I, B 2, B 2 Where B is background (or more realistically – valid image but not of great interest So we prime the filter to avoid transients by assuming image repeats in space B 2, B 1, B 1, I, I, B 2, B 1, B 1 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 22 / 33

What is the expected result? n n FIR coefficients 5. 0, 0, 0, input[ ] = 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0 Therefore result (two sets of scaled impulse response) would be filtered. Input[ ] = 0, 0, 0, 15, 0, 0, 0, 20, 0, 0, 0 n 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 23 / 33

Story n n Want to be able to evaluate what happens (speed issues with compiler) when you write the code with fixed-length external arrays FIR[32] rather than FIR[N] where N can be any number, and array outside function (global) Want to be able to evaluate what happens (speed issues with compiler) when you write the code with passing array parameters into the filter 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 24 / 33

Write the Test – FIR with parameter passing TEST(Story 1_A 1) { float input[ ] = 0, 0, 0, 3. 3, 0, 0, 0, 4. 4, 0, 0, 0; float FIRcoeffs[ ] = 2. 0, 0, 0, 0; (4 FIR coeffs) float expected. Result[ ] = 0, 0, 0, 6. 6, 0, 0, 0, 8. 8, 0, 0, 0; float actual. Result[ ] = 0; for (I = 0; I < length(input)) actual. Result[I] = FIRparam. Passing(input[I], FIRcoeffs, length(FIRcoeffs)); CHECK_ARRAY_CLOSE(expected. Result, actual. Result, length(expected. Result), threshold); Testing FIR filters } 1/26/2022 Copyright smithmr@ucalgary. ca 25 / 33

Write the Test – FIR with parameter passing V 2 TEST(Story 1_A 2) { float input[ ] = 0, 0, 0, 3. 3, 0, 0, 0, 4. 4, 0, 0, 0; float FIRcoeffs[ ] = 2. 0, 0, 0, 0; (6 coefficients) float expected. Result[ ] = 0, 0, 0, 6. 6, 0, 0, 0, 8. 8, 0, 0, 0; float actual. Result[ ] = 0; for (I = 0; I < length(input)) actual. Result[I] = FIRparam. Passing(input[I], FIRcoeffs, length(FIRcoeffs)); CHECK_ARRAY_CLOSE(expected. Result, actual. Result, length(expected. Result), threshold); Testing FIR filters } 1/26/2022 Copyright smithmr@ucalgary. ca 26 / 33

Write the Test – FIR with static array float static. Array[ ] = { 0. } // The C++ optimizer may find FIR filters accessing external arrays might be easier to optimize (fixed location) FIRTEST(Story 1_B) { float input[ ] = 0, 0, 0, 3, 0, 0, 0, 0; float FIRcoeffs[ ] = 2. 0, 0, 0, 0; float expected. Result[ ] = 0, 0, 0, 6, 0, 0, 0, 0; float actual. Result[ ] = 0; Copy(FIRcoffs, static. FIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) in. Value = input[I]; // Global variable FIRstatic. Num 4( ); // Read global input, coefficients and produces global output actual. Result[I] = out. Value; } CHECK_ARRAY_CLOSE(expected. Result, actual. Result, length(expected. Result), threshold); } Testing FIR filters 1/26/2022 Copyright smithmr@ucalgary. ca 27 / 40

Write the Test – FIR with static array V 2 TEST(Story 1_B) { float input[ ] = 0, 0, 0, 3, 0, 0, 0, 0; float FIRcoeffs[ ] = 2. 0, 0, 0, 0; float expected. Result[ ] = 0, 0, 0, 6, 0, 0, 0, 0; float actual. Result[ ] = 0; Copy(FIRcoffs, static. FIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) in. Value = input[I]; FIRstatic. Num 6( ); // Using different FIR filter routine for actual. Result[I] = out. Value; // Each filter length might be faster } CHECK_ARRAY_CLOSE(expected. Result, actual. Result, length(expected. Result), threshold); } 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 28 / 33

Write the stubs – expect to fail File 1 float FIRparam. Passing. Index(float in. Value, float FIRcoeffs[ ], int num. Coeffs( ) { return -3. 1459; } n File 2 float FIRparam. Passing. Pointer(float in. Value, float *FIRcoeffs, int num. Coeffs( ) { return -5. 1459; } n 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 29 / 33

Write the stubs – expect to fail File 3 float outvalue. Num 4 Index = 5. 6789; float FIRcoeffs. Num 4[ ] = { 0 }; void FIRstatic. Num 4 Index(void) { // Implement via Index outvalue. Num 4 Index = -8. 1459; } n File 4 float outvalue. Num 6 Index = 7. 6789; float FIRcoeffs. Num 6[ ] = { 0 }; void FIRstatic. Num 6 Index(void) {// Implement via Index outvalue. Num 6 Index = -9. 1459; } n 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 30 / 33

Write the Test – FIR with static array – V 3 TEST(Story 1_C) { float input[ ] = 0, 0, 0, 3, 0, 0, 0, 0; float FIRcoeffs[ ] = 2. 0, 3. 0, 4. 0, 5. 0; float expected. Result[ ] = 0, 0, 0, 6, 9. 0, 12. 0, 15. 0, 0, 0, 0; float actual. Result[ ] = 0; Copy(FIRcoffs, static. FIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) in. Value = input[I]; FIRstatic. Num 4( ); actual. Result[I] = out. Value; } CHECK_ARRAY_CLOSE(expected. Result, actual. Result, length(expected. Result), threshold); } 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 31 / 33

Write the Test – FIR with static array – V 4 TEST(Story 1_C) { Each input produces an “impulse response” float input[ ] = 0, 0, 0, 3, 0, 0; float FIRcoeffs[ ] = 2. 0, 3. 0, 4. 0, 5. 0; float expected. Result[ ] = 0, 0, 0, 6, 9, 12, 15 + 6, 0 + 9, 12 + 6, 15+ 9, 12, 15, 0; float actual. Result[ ] = 0; Copy(FIRcoffs, static. FIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) in. Value = input[I]; FIRstatic. Num 4( ); actual. Result[I] = out. Value; } CHECK_ARRAY_CLOSE(expected. Result, actual. Result, length(expected. Result), threshold); } 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 32 / 33

When do you have enough tests? n n It is a standard question Salary tax function n n If If salary < 10000 then no tax 10000 <= salary < 20000 then 10% tax 20000 <= salary < 40000 then 20% tax 40000 <= salary then 30% tax Partition of space n n Salary = 5000, 15000, 25000, 40000 – typical Salary = -10000, 0 -- special cases Salary = 10000, 9999, 10001 – boundary test Etc. 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 33 / 33

How do you test FIR algorithm? void FIRNum 4(void) { FIFO[3] = invalue; float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } out. Value = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 34 / 33

Introduce deliberate mistakes n You have a Personal Software Process n n n You know what sort of errors you typically make E. g. Too many times around the loop Forgetting important constants Go in a deliberately make the mistake you commonly make See what sort of tests spot that mistake Always include those sort of tests 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 35 / 33

Test should spot the following error void FIRNum 4(void) { FIFO[3] = invalue; float sum; // float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } out. Value = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 36 / 33

Exam question n n Why might this incorrect algorithm pass the first test? Give me example code that would probably cause this incorrect algorithm always pass tests void FIRNum 4(void) { FIFO[3] = invalue; float sum; // float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } etc 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 37 / 33

Code causing test to pass TEST(Story 1_C) { float input[ ] = 0, 0, 0, 3, 0, 0; float FIRcoeffs[ ] = 2. 0, 3. 0, 4. 0, 5. 0; float expected. Result[ ] = 0, 0, 0, 6, 9, 12, 15 + 6, 0 + 9, 12 + 6, 15+ 9, 12, 15, 0; float actual. Result[ ] = 0; Copy(FIRcoffs, static. FIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) in. Value = input[I]; Do. Garbage Function( ); FIRstatic. Num 4( ); actual. Result[I] = out. Value; } CHECK_ARRAY_CLOSE(expected. Result, actual. Result, length(expected. Result), threshold); } Testing FIR filters 1/26/2022 Copyright smithmr@ucalgary. ca 38 / 33

Example – why does Do. Garbage( ) make FIRNum 4( ) work? void FIRNum 4(void) { FIFO[3] = invalue; float sum; // float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } etc void Do. Garbage(void) { int garbage = 0; } 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 39 / 33

POSSIBLE solution to incorrect Test results n Have a couple of tests that are similar TEST(Story 1_C) { float input[ ] = 0, 0, 0, 3, 0, 0, 0, 0; float FIRcoeffs[ ] = 2. 0, 3. 0, 4. 0, 5. 0; float expected. Result[ ] = …. . } TEST(Story 1_D) { float input[ ] = 3, 0, 0, 0, 0; float FIRcoeffs[ ] = 2. 0, 3. 0, 9. 0, -5. 0; float expected. Result[ ] = …. . } n Use the automated framework to run the list of tests both forward and backwards 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 40 / 33

Mututation testing n n Have a tool that deliberately generates a new version of the file with an inserted mistake E. g. n n n change < to <= or > Change + to – Change * to + (note 2 * 2 = 2 + 2) 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 41 / 33

How do you test FIR algorithm? void FIRNum 4(void) { FIFO[3] = invalue; float sum = 0; for (int I = 0; I <= 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } out. Value = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 42 / 33

How do you test FIR algorithm? void FIRNum 4(void) { FIFO[3] = invalue; float sum = 0; for (int I = 0; I < 3; i++) { sum = sum - FIFO[I] * FIRcoeffs[I]; } out. Value = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 43 / 33

Test should spot the following error void FIRNum 4(void) { FIFO[3] = invalue; float sum; // float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } out. Value = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 1/26/2022 Testing FIR filters Copyright smithmr@ucalgary. ca 44 / 33