Chair for Computer Aided Medical Procedures Augmented Reality

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Automatic Feature Generation for Endoscopic Image Classification Ulrich Klank 17 September 2020 Chair for Computer Aided Medical Procedures & Augmented Reality Department of Computer Science | Technische Universität München

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Endoscopic Image Classification Synchronization of surgeries by phases is essential for Workflow Project 1 Analyzing endoscopic images supports the classification of the current state into a phase. 1 A. Ahmadi, T. Sielhorst, R. Stauder, M. Horn, H. Feussner, N. Navab -Recovery of surgical workflow without explicit models CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 2

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Task How can these images be classified into surgical phases? Or if they can’t be classified well enough, which features can be added to the rest of the information? CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 3

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Definition of a Feature Gradient Histogram Feature as representation of discriminative information Numerical vector calculated out of the image Reduction of the dimension Features Basic: Gradient/Color/Motion Histogram Integral image 1 Phase concruency 2 1 Viola, Jones- Robust Real-time Object Detection 2 Kovesi - Image Features from Phase Congruency CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 4

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Feature Space With a limit to algorithms with an output smaller or equal than the input we get a number of injective function about: (cd) cd ( cd ¡ 1) 2 depending on the dimension of the input vector d and the number of different values c which are possible in the input. Exploration of this space with Genetic Programming 1 A method to explore a high dimensional space Fitness function influences the resulting features 1 Feature Extraction from Multiple Data Sources Using Genetic Programming - John J. Szymanski, * Steven P. Brumby, Paul Pope, Damian Eads, Diana Esch-Mosher, Mark Galassi, Neal R. Harvey, Hersey D. W. Mc. Culloch, Simon J. Perkins, Reid Porter, James Theiler, A. Cody Young, Jeffrey J. Bloch and Nancy David CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 5

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Feature Generation by Genetic Programming 12 Code Evaluation Mutation 1 Wolfgang Banzhaf, Peter Nordin, Robert E. Keller, Frank D. Francone - Genetic Programming: An Introduction 2 Koza - Genetic Programming CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 6

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Implementation Steps Definition of a programming language (more) Designed for recombination and mutation Designing a Framework to run programs Translation of basic features in the new language Evaluation of the fitness of programs Recombination and Mutation CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 7

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu An Example as Proof of Concept Implementation of a color histogram in the new language Test on an easy sample, 6 images(10*10*RGB, jpeg) in 3 classes: Dimension of input here (cd): 76. 800 CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 8

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Evaluation of the Example The Fitness Function is the core of the system CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 9

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Evaluation of the Example Comparison of several Programs Pi P 1 CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 P 2 … 10

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Evaluation of the Example P 1 P 2 … A random selection of the labeled input Ik several from of each class I 1 I 2 CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 I 3 11

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Evaluation of the Example P 1 I 1 P 2 I 2 … I 3 Execution of the program with every selected input Ik As result there are output vectors: o 11 ; o 12 ¢¢¢o 23 CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 12

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Evaluation of the Example P 1 I 1 P 2 I 2 … I 3 o 11 ; o 12 ¢¢¢o 23 Compare each pair of output vectors of one Program, by the distance dik ; l = P l en ( oi k [m ]¡ oi l [m ]) 2 m = 0 ( j o [m ]j + j o [m ]j ) 2 ik il CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 13

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Evaluation of the Example P 1 I 1 P 2 I 2 … I 3 o 11 ; o 12 ¢¢¢o 23 i = dk; l P l en (oi k [m ]¡ oi l [m ]) 2 m = 0 (jo [m ]j+ jo [m ]j) 2 ik il Select the maximum of these distances, only considering pairs of different classes: ar gmax k l (dik l jk: class = 6 l: class) Now we have two classes k, l which our feature separates CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 14

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Evaluation of the example P 1 I 1 P 2 I 2 … I 3 o 11 ; o 12 ¢¢¢o 23 i = dk; l P l en (oi k [m ]¡ oi l [m ]) 2 m = 0 (jo [m ]j+ jo [m ]j) 2 ik il Subtract the distances in images of the classes k or l f i = dik l ¡ d ki k + d il l 2 ar gmax k l (dik l jk: class = 6 l: class) CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 15

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu 3. 1 Results in this Example < 64 > < 72 > < -16 > New Program < 72 > < -16 > CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 16

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Fitness in the Example: P 1 Input Program was calculating a Color Histogram (Output length 512 in ~2000 steps), resulting fitness: ~ 1. 2 % P 2 The new generated program calculated one number (in 30 steps) and got a fitness of: ~ 98. 8 % CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 17

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Evolution Technique used in the Example Selection of the two best Program and generation several children by these two methods: One parental program: Mutation Adding of random commands Removing a single commands 1 Duplicating an existing command Switching commands P 1 P 2 P 11 P 12 Two parental programs: Breeding Mixing both, preferring blocks of a specific size Adding random commands in the gaps between CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 18

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Conclusion The concept seems to work Adaptation to workflow project’s data can start CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 19

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Steps to adapt it to the Workflow project Adapt fitness function (more) Change Genetic rules (more) Optimize the framework Generate C code out of the intermediate language CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 20

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Any Questions? CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 21

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Appendix CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 22

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Planned Improvements in the Genetic rules Probability to survive should depend on the fitness of all other existing programs Even a program with a low fitness should have a small change to survive Also the reproduction should not be restricted to the best program Back CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 23

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Fitness Actual fitness expresses: Linear separation of two classes Further improvements: Static Analysis of Code 1 Semantic Checks to prevent running in infinite loops to prevent the execution of programs not using the input … Other Classifiers to evaluate the output Back 1 Colin G. Johnson-Deriving genetic programming fitness properties by static analysis CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 24

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Specification of the language Assembler style with some high level functions 1 Opcode and 3 Arguments for one command High level examples: Reading of the input and its dimensions For loops Adding of more functions possible Back CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 25

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Scheme for the programming language OPCode 1. Par ameter 010 0 For From counter to Constant stack 123 -1 2. 3. Parameter er 100 65 To End of the Constant loop Constant 20 … CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 26

Chair for Computer Aided Medical Procedures & Augmented Reality | wwwnavab. cs. tum. edu Generated Program (without nops) INPUTDIMENSION 0 0 4595026 MULTIPLY 0 0 0 SAVE 0 -1 0 LOAD 3 0 0 INPUTDIMENSION 1 0 0 PLUS 0 12604 0 SAVE 23367 -1 0 FOR 0 -1 37 LOAD 4 0 0 PLUS 0 6457 0 PLUS 31190 0 0 LOAD 1 0 0 JUMP 55 0 0 SAVE 5 -1 0 DIVIDE 0 0 0 LOAD 5 0 0 DIVIDE 0 0 0 POP 0 0 -1 PUSH 32 0 0 READINPUT -1 -1 -1 LOAD 1 -1 0 LOAD 5 0 -1 POP 0 0 -1 PUSH 64 0 -1 LOAD 2 0 0 RETURN 0 0 0 SAVE 3 -1 0 JUMP 0 0 0 PUSH 8 0 0 FOR 8 520 58 CAMP | Department of Computer Science | Technische Universität München | 17 September 2020 27